Running validation

The evaluate subcommand is the validation harness for any change you propose to the system. This page is the operational guide on how to use it — when to run a single-paper iteration, when to run a full sweep, and how to inscribe results.

For the methodology behind validation, see how we measure quality. For the canonical worked example, see iteration discipline (kammer prompt iteration).

The single-paper loop

For changes targeting a known failure mode on a specific paper:

# 1. Establish baseline on the target paper
elife-extract evaluate \
  --reference-dir ~/Projects/mainenlab/elife-claim-trees/claims \
  --work-dir /tmp/eval-baseline \
  --paper kammer-2026-foveal-feedback \
  --review-mode external

# 2. Make the change (prompt revision, model swap, ...)
# ... edit prompts/external-reviewer.md ...

# 3. Re-run on the target paper with the change
elife-extract evaluate \
  --reference-dir ~/Projects/mainenlab/elife-claim-trees/claims \
  --work-dir /tmp/eval-iter1 \
  --paper kammer-2026-foveal-feedback \
  --review-mode external

# 4. Diff the per-paper scorecards
diff /tmp/eval-baseline/kammer-*/scorecard.json /tmp/eval-iter1/kammer-*/scorecard.json

Or read the per-paper scorecards directly. Each scorecard.json has the full match list; you can spot-check which specific role mismatches got fixed.

Cost: ~$10 per iteration. Time: ~15 min per iteration.

The full-sweep validation

When the single-paper iteration shows a meaningful change, validate against the corpus:

elife-extract evaluate \
  --reference-dir ~/Projects/mainenlab/elife-claim-trees/claims \
  --work-dir /tmp/eval-sweep-iter1 \
  --all \
  --review-mode external

Cost: ~$100. Time: ~85 min sequential.

The aggregate scorecard (/tmp/eval-sweep-iter1/aggregate-scorecard.md) gives you mean and median per metric across the 10 papers. Diff against your baseline:

diff /tmp/elife-eval-baseline/aggregate-scorecard.md /tmp/eval-sweep-iter1/aggregate-scorecard.md

A net-win pattern looks like: recovery flat or up, role up, panel flat or up. A regression pattern: any of those metrics drops materially.

The 10-paper sweep is also where you check per-paper deltas — a change might lift the average but tank one paper. Open each paper’s scorecard.json, compare to the baseline, and confirm no individual paper regressed by more than ~5 points.

Selecting which papers to run

evaluate accepts three forms of paper selection:

• --paper <slug> — single paper
• --papers <slug1>,<slug2>,... — comma-separated list
• --all — every paper-dir under --reference-dir

Use --paper for targeted iteration. Use --papers for a representative subset (e.g., one paper from each domain category). Use --all for the canonical full sweep.

--skip-existing for resumable sweeps

The full sweep takes ~85 minutes. Network failures, OOM, or accidental kills can interrupt it. The harness writes per-paper scorecards as they complete; --skip-existing reuses those scorecards on retry:

# First attempt (interrupted at paper 6)
elife-extract evaluate --reference-dir ... --work-dir /tmp/eval --all --review-mode external

# Resume — picks up from paper 7
elife-extract evaluate --reference-dir ... --work-dir /tmp/eval --all --review-mode external --skip-existing

Failed papers (those whose scorecard.json contains an error: field) are also skipped on retry. To re-run a failed paper, delete its scorecard:

rm /tmp/eval/kammer-2026-foveal-feedback/scorecard.json
elife-extract evaluate ... --skip-existing  # re-runs only kammer

What the scorecards contain

Per-paper scorecard.json:

{
  "paper_slug": "kammer-2026-foveal-feedback",
  "paper_doi": "10.7554/eLife.<id>",
  "n_ref": 23,
  "n_cli": 47,
  "n_recovered": 21,
  "n_exact": 16,
  "n_partial": 5,
  "n_panel_match": 11,
  "n_role_match": 12,
  "matches": [
    {
      "ref_slug": "...",
      "cli_slug": "...",
      "match_quality": "exact|partial|none",
      "panel_match": true,
      "role_match": false,
      "notes": "..."
    },
    ...
  ],
  "review_mode": "external",
  "cli_dir": "/tmp/eval/kammer-.../kmmer-2025-...",
  "error": null
}

The matches array is the key debugging surface. Each entry tells you which CLI claim the matcher paired with each reference claim, the match quality, and whether the panel / role agreed. To diagnose role failures, filter for role_match: false and read the notes to understand why.

The aggregate scorecard (aggregate-scorecard.md) is human-readable markdown with mean/median per metric, per-paper detail rows, and a list of failures.

Cost-conscious workflows

Validation is expensive. Three patterns for keeping cost in check:

1. Iterate on the cheapest target first. The kammer iteration was $10 per cycle; it took 1 cycle to validate a 14-point lift. Compare to running a full $100 sweep per iteration — you’d burn $1000 to validate the same change at the corpus level.

2. Cache extracts; re-run only the changed step. When iterating on the reviewer prompt only, you don’t need to re-extract — the reconciled draft is unchanged. Run extract once, then write --review-mode external --draft <prior-draft> for each iteration. Saves ~$5 per iteration.

(Note: evaluate currently doesn’t support this — it always re-extracts. A future enhancement could add a --reuse-draft flag.)

3. Use evaluate --papers with a subset. A 3-paper sample (representative of your domain mix) gives you ~$30 cost and most of the per-paper-failure-mode signal. Run the full sweep only when the subset shows the change is a net win.

Inscribing results

After every accepted iteration, inscribe in the worklog at home/collabs/elife/claim-trees/jobs/extract-cli.md:

- 2026-MM-DD [user:session] **Prompt iteration: <name>.** Hypothesis: <what failure mode this addresses>. Change: <which file edited, what added>. Validation: <single-paper or full sweep, on which paper(s)>. Per-claim deltas: <which specific failures got fixed>. Headline metrics: <recovery before -> after, panel before -> after, role before -> after>. Cost: ~$<X>. Decision: ship / revert / further iteration. Rationale: <one sentence>.

This creates an empirical record of the system’s evolution. Future iterations can read what was tried, what worked, what didn’t.

Failures and what to do

Failure mode	Diagnostic
prepare failed: [Errno 8] nodename nor servname	DNS / network. Wait, retry. Often transient.
extract failed: 429	Rate limit. Reduce parallelism. Retry.
external review failed: <validation error>	Reviewer emitted JSON that fails Pydantic schema. Inspect the raw output (saved at /tmp/elife-extract-debug-...). Likely a prompt-variant issue.
scoring failed: <matcher error>	Matcher Opus call failed. Usually transient; retry.
Recovery dropped on a specific paper	Diagnose: did the extraction agents change? The reviewer? Look at per-claim matches with match_quality: none.
Role agreement dropped	Diagnose: which roles are misclassified? Cluster by role pattern. May be over-correction from a prior iteration.

Implementation references

• elife_extract/evaluate.py — the harness
• elife_extract/cli.py:cmd_evaluate() — the orchestration
• home/collabs/elife/claim-trees/jobs/extract-cli.md — the worklog where iteration deltas live

Next steps

• How we measure quality — the round-trip methodology in detail
• Iteration discipline — kammer worked example
• Adding a prompt variant — the most common change to validate