The external reviewer (Step 4.5)

The methodology document — docs/method.md § 3.3 Step 5 — frames Step 5 as a hard human-review gate: “Nothing is written to disk until the table is approved.” This was correct when the pipeline was a curator-and-agents loop. It breaks down when the CLI runs in environments without a curator: batch jobs, integration handoffs, scaling to a journal-sized corpus.

Step 4.5 is the system’s substitute for the curator at the review gate. It is a single Opus call that takes the reconciled draft plus the paper’s prose and produces a revised draft addressing the four (then seven) systematic biases that prose-level extraction exhibits.

This page explains why the step exists, what it does, what the empirical result is, and the failure modes it doesn’t fix.

Why this step exists

The three Sonnet extraction agents read the prose and return what is in the prose. That sounds tautological, but the problem is precisely that the prose often does not contain the paper’s argument structure as discrete propositions.

Consider the Headley 2026 paper. Its 26 curated reference claims include:

One organising hypothesis: “Perisomatic and distal dendritic inhibition serve distinct computational roles in regulating neuronal output.”
Six predictions: each phrased as “if [hypothesis], then [observable].”
Twelve empirical claims: the panel-grounded measurements that test the predictions.
Three controls: empirical claims whose primary work is to rule out alternative explanations.
Plus: scope / methodological / synthesis / interpretation / literature-context claims.

Now read Headley’s prose. The hypothesis is implicit — the abstract says “we sought to understand how rhythmic perisomatic and distal dendritic inhibition impacted integration.” There is no explicit “if X, then Y” prediction phrasing in the results section. The paper jumps from “we sought to understand” to “we found.” The deductive structure exists in the modelling decisions — pairing distal inhibition with a beta frequency sweep is what tests the prediction that beta is optimal for distal — but the prediction is never stated as a discrete proposition.

A prose-grounded extraction agent reading this paper will surface the empirical findings (what was measured) and miss the predictions (what was expected). Empirically, on the first round-trip test, the auto-approve pipeline recovered all 26 reference claims (100% recovery) but classified 0 of the 6 prediction-* claims with role: prediction — they all came out as empirical or synthesis. Role agreement landed at 65%, well below the 75% threshold.

This is not a bug in the extraction agents. They are doing the right thing — extracting what the prose says. The deductive structure the curator inferred from the modelling decisions is not in the prose to be extracted.

What Step 4.5 does

The reviewer prompt — at prompts/external-reviewer.md — names seven systematic biases the reviewer is responsible for fixing:

Bias	Example	Fix
1. Prediction role under-coverage	All 6 Headley `prediction-*` claims came out as `empirical`	Surface predictions as separate claims when the empirical structure implies a deductive expectation
2. Hypothesis role under-coverage	Headley’s organising hypothesis came out as `empirical`	Surface the paper’s organising bet as `role: hypothesis`
3. Multi-panel claim collapse	`panel: fig4, fig5` references matched to single-panel CLI claims	Expand to comma-separated panel lists when the prose anchors a claim across panels
4. Synthesis vs interpretation confusion	The CLI labels both as `synthesis`	Use `interpretation` when the claim invokes broader theory; `synthesis` when it integrates within the paper’s own evidence
5. Hypothesis-to-prediction over-shifting (added)	After Bias 1’s fix, some hypotheses got reclassified as predictions on papers with explicit hypothesis layers	Preserve the paper’s organising bet as `hypothesis` even when adding predictions elsewhere
6. Control role under-recognition (added)	Empirical claims whose work is “rules out alternative X” came out as `role: empirical`	Mark `role: control` when the claim’s primary function is eliminative
7. Literature-context under-recognition (added)	Cited prior work paraphrased in the paper came out as `interpretation`	Mark `role: literature-context` when the claim depends on a specific cited prior paper, even when the citation is implicit

The reviewer takes the reconciled draft and:

Changes the role / claim-type / panel of existing claims that were misclassified
Adds new claims the extraction agents missed (typically hypothesis and prediction claims that formalize the implicit deductive structure)
Annotates revisions with notes: "[reviewer] role: empirical -> prediction. The paper frames this as testing the prediction that gamma is optimal..." for audit
Preserves the original draft alongside the revised one (<draft>.json and <draft>.reviewed.json) so any change is recoverable

The reviewer is not allowed to delete claims (downgrade to notes if you think they should go), invent quantitative values, or merge two distinct claims (split-then-merge would lose information).

Why Opus

The extraction agents run on Sonnet 4.6 because their task is straightforward: read a slice of prose, surface the propositions, return JSON. The reviewer’s task is structural inference — “given this empirical sequence, identify the implicit hypothesis and the predictions it entails.” That is meaningfully harder, and Opus is meaningfully better at it.

We tested both hypotheses empirically. The Phase F prompt iteration (without external reviewer; revised the Sonnet extraction prompts to push harder for prediction surfacing) moved Headley’s role agreement from 65% to 65.4% — flat. The external reviewer pass (using Opus on the reconciled draft) moved it from 65.4% to 96.2% — a 31-point gain. The capability difference between Sonnet and Opus on structural-inference tasks is the explanation; the prompt revision alone could not close the gap.

The cost trade is small: one additional Opus call per paper, ~$1-2, ~3 minutes. The total per-paper cost rises from ~$5 to ~$7. For an eLife-scale corpus of hundreds to thousands of papers, this is well within the budget the methodology’s hand-extraction would have required.

Empirical results — Headley v3

The same paper, three pipeline configurations, scored against the 26-claim curated reference:

Metric	v1 (auto-approve)	v2 (revised Sonnet prompts)	v3 (external Opus reviewer)
Claim recovery	100%	100%	100%
Panel agreement	53.8%	50.0%	50.0%
Role agreement	65.4%	65.4%	96.2%
Match quality (exact / partial)	10 / 16	8 / 16	13 / 13

v3’s role agreement exceeds the 75% threshold by a wide margin. The 30-point gain is concentrated in the prediction (6/6 surfaced and correctly classified) and hypothesis (2/2 surfaced) categories — exactly the failure modes the prompt targets.

The 10-paper sweep across the public eLife corpus confirms the result generalizes — see validation results for per-paper detail.

What the reviewer doesn’t fix

Panel agreement stayed flat across v1 / v2 / v3 (~50%). The reviewer prompt mentions the multi-panel collapse problem (Bias 3) but doesn’t aggressively expand single-panel claims to multi-panel lists. The Caption-reader is the right place to fix this; a dedicated prompt iteration on the Caption-reader’s panel-list output would close the gap.

Subtle role-classification calls persist. On the kammer paper (a fovea-feedback fMRI study with rich control role usage), the reviewer iteration moved role from 57% to 71% — meaningful but still below the 75% threshold. The remaining failures are cases where the role is genuinely ambiguous (a non-significant result that functions as a control without explicit “rules out” language; an empirical observation the curator integrates as synthesis). These are judgment calls that the methodology’s Step 5 review gate is designed to handle; the reviewer is doing structural inference, not full curator-replacement.

One-to-one matcher limitations — when the CLI splits a single reference claim into two CLI claims (statistical-test claim + interpretation claim), the matcher’s 1-to-1 alignment can only score one of them as the match. The other shows up as an “unmatched CLI claim” in the scorecard, even when together they correctly capture the reference. This depresses headline metrics in a way that overstates the failure rate.

When to skip the external reviewer

Not always. Three cases where --review-mode auto-approve (no reviewer) is the right choice:

You already plan to manually review the resulting .md files. The auto-approve output is a valid draft; the analyst can adjust roles and claims at the file level after the write step. Skipping the reviewer saves ~$2/paper and ~3 minutes when human review is happening anyway.
You’re testing the extraction layer, not the reviewer layer. When iterating on the Sonnet extraction prompts (prompt variant A/B testing), running the reviewer on each variant adds noise — it’s a non-deterministic transformation that may mask the extraction-level signal you’re measuring.
The paper has no implicit deductive structure to recover. Atlas papers, methods papers, and observational surveys often don’t have the hypothesis-prediction-test arc the reviewer is calibrated for. The reviewer doesn’t add nonsense in these cases (we verified this on Artiushin, the spider-atlas paper — the reviewer preserved the observational structure rather than inventing hypotheses), but it also doesn’t add value.

Where to go from here

The full reviewer prompt is at the prompts reference
The 10-paper sweep results shows how the reviewer performs across the corpus
The iteration discipline walks through the kammer prompt iteration that added Biases 5, 6, and 7
Review modes — when to use each compares the four review options