The 10-paper sweep results
This page reports the system’s empirical performance on the 10 public eLife papers in the curated reference corpus. Two full sweeps have run; v2 (with the iterated reviewer prompt incorporating Biases 5-7) is the current baseline.
For the methodology behind these numbers, see how we measure quality. For the iteration that produced v2, see iteration discipline.
Aggregate metrics — current baseline (v2)
Section titled “Aggregate metrics — current baseline (v2)”Sweep ran 2026-05-11 with Sonnet 4.6 × 3 extraction agents, Opus 4.6 reconciler, Opus 4.6 external reviewer with the iterated 7-bias prompt, CrossRef DOI verification.
| Metric | Mean | Median | Threshold | Status |
|---|---|---|---|---|
| Claim recovery | 96.9% | 100.0% | ≥ 80% | ✅ PASS |
| Role classification | 83.1% | 84.3% | ≥ 75% | ✅ PASS |
| Panel assignment | 60.9% | 58.6% | ≥ 90% | ❌ FAIL |
Recovery is the load-bearing metric and it passes by a wide margin. Median recovery is 100% — every reference claim has a corresponding CLI extraction on half the corpus, and the lowest-recovery paper (rozak) still hits 87%.
Role classification clears its threshold by 8 points on the mean and 9 points on the median. The external reviewer’s structural-inference pass converts what would be a failure mode in auto-approve mode (~65% role agreement) into a strongly passing result.
Panel assignment improved from v1 (54.7% → 60.9%) but still fails the 90% threshold. The improvement came from the iterated reviewer’s better handling of multi-panel claims; the residual failure is the documented multi-panel collapse problem.
Delta from v1 baseline
Section titled “Delta from v1 baseline”The v2 iteration added Biases 5-7 to the external reviewer prompt (hypothesis preservation, control role recognition, literature-context recognition). Headline deltas across the corpus:
| Metric | v1 | v2 | Δ |
|---|---|---|---|
| Recovery (mean / median) | 95.3% / 98.0% | 96.9% / 100.0% | +1.6 / +2.0 |
| Role (mean / median) | 78.1% / 76.4% | 83.1% / 84.3% | +5.0 / +7.9 |
| Panel (mean / median) | 54.7% / 53.1% | 60.9% / 58.6% | +6.2 / +5.5 |
The iterated prompt is a measurable net win on every aggregate metric. See iteration discipline for how v2 was developed and validated.
Per-paper detail (v2)
Section titled “Per-paper detail (v2)”| Paper | n_ref | n_cli | Recovery | Panel | Role |
|---|---|---|---|---|---|
| artiushin (spider atlas) | 17 | 79 | 100% | 59% | 94% |
| bouyeure (fear RSA) | 30 | 66 | 100% | 47% | 90% |
| ejdrup (dopamine) | 25 | 83 | 96% | 58% | 83% |
| gadeke (guilt insula) | 27 | 65 | 100% | 48% | 85% |
| headley (inhibitory rhythms) | 26 | 89 | 100% | 81% | 92% |
| kammer (foveal feedback) | 23 | 43 | 96% | 91% | 82% |
| kolb (iGABASnFR2) | 20 | 70 | 100% | 75% | 70% |
| rozak (neurovascular DL) | 23 | 72 | 87% | 60% | 55% |
| scheller (self-prioritization) | 23 | 59 | 100% | 48% | 83% |
| wengert (KCNC1) | 31 | 91 | 90% | 43% | 96% |
Most dramatic per-paper deltas vs v1:
- kammer role 57% → 82% (+25 points), panel 52% → 91% (+39 points)
- headley panel 54% → 81% (+27 points)
- artiushin recovery 88% → 100% (+12 points), panel 60% → 59% (within noise)
Material regressions vs v1 (only two, both small):
- rozak role 65% → 55% (-10 points). Methods paper; the iterated reviewer’s stricter control-recognition rule may over-apply on a paper that doesn’t have the typical hypothesis-test structure.
- kolb role 75% → 70% (-5 points). Similar pattern, less material.
These are the residual failures the iteration trades for the corpus-wide gains. Targeted prompt iteration (per iteration discipline) on either paper could close these without regressing the rest of the corpus.
Recovery distribution: 87-100%, with 5 of 10 papers at 100%. The 87% (rozak) and 88% (artiushin) are the lowest; both are unusual paper structures (rozak is a deep-learning methods paper; artiushin is an observational atlas).
Role distribution: 57-96%. Three papers below the 75% threshold individually (kammer 57%, rozak 65%, scheller 70%). Mean and median both clear the threshold; the failures are clustered in papers with structural shapes the reviewer prompt was less calibrated for.
Panel distribution: 36-74%. Consistently below the 90% target. Wengert’s 36% is the worst; multi-panel reference claims dominate that paper.
What the numbers tell us
Section titled “What the numbers tell us”The Headley v3 result generalizes. Headley alone (100% recovery, 96% role with external review) was a single-paper data point that could have been Headley-specific. The 10-paper sweep confirms the pattern holds across diverse papers — different domains (atlas, fMRI, computational, channelopathy), different argument structures (hypothesis-driven, observational, methods-focused), different scales (17 to 31 reference claims).
The external reviewer is doing real work. Without it (auto-approve mode on the same papers), role agreement averages ~65%. With it, 83% (mean, v2). The 18-point average lift comes from the structural-inference work the prose-level extraction systematically misses.
The iteration discipline works. The Phase F kammer iteration (single-paper test, +14 points on role) generalized to a +5 point average across the corpus, with the largest individual gain (+25 points) on the targeted paper itself. This is the canonical worked example of the iteration discipline.
Recovery is robust. Even on the lowest-recovery paper (rozak, 87%), the CLI surfaces 20 of 23 reference claims. Misses cluster in papers where the curated reference includes claims requiring deep methodological audit beyond what the Structure-reader agent attempts.
Panel agreement is the next iteration target. v2 lifted panel from 55% to 61%; the threshold is 90%. The Caption-reader prompt could be revised to more aggressively preserve multi-panel lists; the reconciler prompt could be revised to consolidate single-panel CLI claims when they jointly map to a multi-panel reference. Either change would lift the panel score further; both untested empirically.
Where each paper lags (v2)
Section titled “Where each paper lags (v2)”After the iteration, only two papers fall below the 75% role threshold individually:
rozak (55% role) — neurovascular deep-learning paper. Methods-heavy with unusual claim structure (computational pipeline as the contribution, not biological findings). The iterated prompt’s stricter control-recognition rule may over-apply here. The default prompts are calibrated for biological-finding papers; methods papers may need a --prompt-variant.
kolb (70% role) — sensor engineering paper. Borderline case; small drop from v1’s 75%. Likely the same over-application pattern as rozak but smaller.
The 8 papers at 75% or above include the previously-lagging kammer (now 82%) and scheller (now 83%) — both moved into compliance with the iterated prompt.
v1 baseline (for reference)
Section titled “v1 baseline (for reference)”The original 10-paper sweep (no iterated prompt; 4-bias reviewer) produced:
| Metric | Mean | Median |
|---|---|---|
| Recovery | 95.3% | 98.0% |
| Role | 78.1% | 76.4% |
| Panel | 54.7% | 53.1% |
Per-paper v1 numbers preserved at tests/aggregate-scorecard-10paper.md in the repo for historical comparison.
Cost of the sweep
Section titled “Cost of the sweep”| Phase | Cost |
|---|---|
| 10 × extract (~$5 each) | ~$50 |
| 10 × external review (~$2 each) | ~$20 |
| 10 × matcher (~$3 each) | ~$30 |
| Total | ~$100 |
Wall time: ~85 minutes sequential. Could run in 30-40 minutes with 3-way shell-level parallelism.
What this validates and what it doesn’t
Section titled “What this validates and what it doesn’t”Validates:
- The pipeline produces curator-quality claim coverage on diverse paper types
- The external reviewer’s role-classification lift generalizes beyond Headley
- The system’s per-paper cost (~$7 with reviewer) is a meaningful improvement over manual extraction (~4 hours of analyst time per paper)
- The validation harness itself works at scale and produces actionable per-paper diagnostics
Doesn’t validate (or doesn’t validate yet):
- Performance on non-eLife papers (the prompts are calibrated for eLife structure)
- Performance on papers without typical experimental structure (preprints, methods papers, observational surveys may need prompt variants)
- Long-term stability (the prompts are version-controlled; future model changes may shift behavior)
- Cross-paper edge resolution (out of scope for round-trip scoring)
Implementation references
Section titled “Implementation references”tests/aggregate-scorecard-10paper.md— the raw scorecard from this sweep- Per-paper scorecards:
<work-dir>/<paper-slug>/scorecard.json elife_extract/evaluate.py— the harness that produced these numbers
Next steps
Section titled “Next steps”- Known limitations — the panel issue, the role-lag papers, what’s outside scope
- Iteration discipline — kammer prompt iteration as a worked example
- Cost and performance — projecting these per-paper numbers to corpus scale