Iteration discipline

This page documents the workflow for changing the system in any way that affects extraction quality — a prompt revision, a model substitution, a new review mode, a different reconciliation strategy. Every change goes through evaluate before deployment. This is the discipline that makes prompt engineering a hypothesis test instead of a guess.

The canonical worked example is the kammer prompt iteration: 57% role agreement to 71% on a single targeted prompt change, validated by the same harness used for the 10-paper sweep.

The general workflow

1. Establish baseline    elife-extract evaluate --all → aggregate-scorecard-baseline.md
2. Make the change       (prompt revision, model swap, new step)
3. Re-run evaluate       elife-extract evaluate --all → aggregate-scorecard-iter1.md
4. Diff aggregates       compare metrics; per-paper deltas
5. Decide                ship if net win; revert if regression
6. Document              inscribe the change + measured delta in the worklog

Steps 1, 3, and 4 cost ~$100 each (the full 10-paper sweep). Step 2 is free. Step 5 is the judgment call; step 6 is the discipline.

For changes that target a specific failure mode on a specific paper, you can iterate cheaper:

1'. Single-paper baseline    elife-extract evaluate --paper <slug> → before.md
2'. Make the change
3'. Re-run on that paper     elife-extract evaluate --paper <slug> → after.md
4'. Diff

Single-paper iteration costs ~$10 each (one extract + one matcher run). When the change is targeted at a known failure pattern, this is sufficient to validate the pattern is fixed before committing to the full sweep cost.

The kammer worked example

The problem. Initial 10-paper sweep showed kammer at 57% role agreement — the worst paper in the corpus. Diagnostic on the per-claim matches identified 9 specific role mismatches, clustered into patterns:

• 3 cases of CLI=empirical vs ref=control (the curator marks empirical results as control when they primarily rule out an alternative)
• 2 cases of CLI=prediction vs ref=hypothesis (the reviewer over-shifted to predictions, even where the paper had explicit hypothesis claims)
• 1 case each of CLI=prediction vs ref=literature-context, CLI=empirical vs ref=synthesis, CLI=control vs ref=empirical (judgment calls), CLI=control vs ref=methodological (subtle distinction)

The hypothesis. Three new bias rules in the reviewer prompt would address the dominant patterns:

• Bias 5 (hypothesis preservation): “Don’t reclassify the paper’s organising bets as predictions just because you are also surfacing predictions elsewhere.”
• Bias 6 (control recognition): “Mark control when the empirical claim’s primary work is to rule out an alternative explanation.”
• Bias 7 (literature-context recognition): “Claims grounded in cited prior work — even when the citation is implicit — are literature-context, not interpretation or prediction.”

The change. Added the three bias sections to prompts/external-reviewer.md. ~80 lines of new prompt text, no other changes to the pipeline.

The validation. Single-paper re-run on kammer:

elife-extract evaluate \
  --reference-dir ~/Projects/mainenlab/elife-claim-trees/claims \
  --work-dir /tmp/elife-eval \
  --paper kammer-2026-foveal-feedback \
  --review-mode external

The result. Per-paper deltas:

Metric	Before	After	Delta
Recovery	91%	91%	unchanged (as expected)
Panel	52%	48%	-4 (within stochastic noise)
Role	57%	71%	+14

Three specific fixes confirmed:

Reference claim	Old role	New role
foveal-feedback-below-direct-stimulation	✗ (empirical vs control)	✓ (now control)
hypothesis-shared-representational-format	✗ (prediction vs hypothesis)	✓ (now hypothesis)
target-excluded-fovea-in-99pct-saccades	✗ (control vs methodological)	✓ (now methodological)

Six remaining failures are subtler patterns the iteration didn’t catch — implicit controls without explicit “rules out” language, hypothesis-vs-prediction at the boundary, 1-to-1 matcher limitations.

The cost. Iteration was ~$10 (one full pipeline run on kammer + matcher). Compare to a full 10-paper sweep cost of $100; this iteration paid back its cost in the diagnostic information alone.

The decision. Ship the prompt change. Re-run the full 10-paper sweep with the iterated prompt to confirm the change is a net win across the corpus before declaring the new baseline.

What the worked example illustrates

Targeted iteration is cheap. Single-paper validation at ~$10 lets you test prompt changes without committing to the full sweep cost. When the change targets a known failure pattern on a known paper, this is the right level.

Iteration has diminishing returns. Three out of nine failures fixed in one iteration. The remaining six are subtler — some are at the architectural limits (1-to-1 matcher), some are genuinely judgment calls. Further iteration cycles on kammer would likely fix 1-2 more, with each iteration buying less than the last.

Fixes can introduce new failures. The Phase F prompt iteration that added the prediction-surfacing rule (Bias 1) helped Headley’s role agreement from 65% to 96%, but on kammer it overcorrected — some hypothesis claims got reclassified as predictions, requiring Bias 5 to fix. Cross-paper validation matters precisely because a change that helps Paper A may hurt Paper B.

The discipline matters more than the result. The 14-point gain on kammer is meaningful, but the more important outcome is the demonstration that prompt iteration can be measured. Future prompt changes have a clear path: change → run evaluate → measure → ship or revert.

Corpus-wide confirmation (v2 sweep)

After the kammer single-paper iteration validated the prompt change, a full 10-paper sweep (evaluate --all --review-mode external with the iterated prompt) confirmed the change is a net win across the corpus.

Metric	v1 baseline (4-bias reviewer)	v2 (7-bias reviewer)	Δ
Recovery (mean / median)	95.3% / 98.0%	96.9% / 100.0%	+1.6 / +2.0
Role (mean / median)	78.1% / 76.4%	83.1% / 84.3%	+5.0 / +7.9
Panel (mean / median)	54.7% / 53.1%	60.9% / 58.6%	+6.2 / +5.5

Material per-paper deltas:

• kammer: role 57% → 82% (+25), panel 52% → 91% (+39) — the targeted iteration delivered, beyond expectations
• headley: panel 54% → 81% (+27) — the bias-5/6/7 rules unexpectedly helped panel here
• rozak: role 65% → 55% (-10) — the only material regression; methods paper, the new control-recognition rule over-applies
• kolb: role 75% → 70% (-5) — small regression, similar pattern

The pattern is clear: targeted iteration on a known failure mode generalizes across the corpus, with predictable trade-offs (a paper structurally similar to the failure mode helps; a paper structurally different may regress slightly). This is the iteration discipline working as designed — every change validated empirically before deployment, deltas inscribed for the worklog.

Cost of the v2 confirmation sweep: ~$100. The v1 → v2 transition is recommended as the new default; rozak and kolb regressions are documented for future targeted iteration via --prompt-variant.

When to iterate vs accept

The validation result is a signal, not a verdict. Three reasonable responses to a metric below threshold:

1. Iterate. Targeted prompt revision with measurable delta. Worth the effort if the failure pattern is concrete and the iteration is likely to address it.

2. Accept and document. Some failure modes are residual — judgment calls between near-equivalent role assignments, 1-to-1 matcher limitations, paper structures the prompt can’t be calibrated for without sacrificing other papers. Ship the current baseline; document the limitation; let the analyst’s Step 5 review handle these cases on a per-paper basis.

3. Architectural change. If the failure pattern is structural rather than promptable (e.g., multi-panel claims would need a multi-panel matcher, not a multi-panel prompt), the right response is a roadmap item, not a prompt iteration.

The kammer iteration was a (1) — concrete pattern, prompt change addresses it, measurable delta. The remaining 6 kammer failures are mostly (2) — analyst review handles them. Architectural improvements (concurrent extraction, anthropic batch API) are (3).

Documenting iterations

Every accepted iteration goes in the worklog at home/collabs/elife/claim-trees/jobs/extract-cli.md with:

• The hypothesis (what failure mode the change addresses)
• The change (which file modified, what added/removed)
• The validation method (single-paper re-run on which paper, full sweep, or both)
• The measured delta (before/after metrics, per-paper if relevant)
• The cost (~$10 single-paper, ~$100 full sweep)
• The decision (ship / revert / further iteration)

This creates an empirical record of the system’s evolution. Future iterations can read what was tried, what worked, what didn’t.

Implementation references

• elife_extract/evaluate.py — the harness that produces scorecards
• prompts/external-reviewer.md — the file most prompt iterations target
• home/collabs/elife/claim-trees/jobs/extract-cli.md — the worklog where iteration deltas are inscribed

Next steps

• The 10-paper sweep results — the baseline you’d compare iterations against
• Adding a prompt variant — for non-default-target iterations
• Known limitations — what’s worth iterating vs accepting