The review gate

Step 5 of the methodology says: “Nothing is written to disk until the table is approved.” This is the gate between the system’s draft and the final claim files. The CLI implements four review modes, three of which honor the gate’s spirit and one (auto-approve) which deliberately skips it for tests and demos.

This page explains the gate’s design and the four modes’ tradeoffs.

The gate’s design

The methodology positions Step 5 as the place where the analyst makes load-bearing judgment calls:

Role assignment — the methodology says “Step 5 is where the schema’s role labels are first assigned definitively, because role-assignment requires the analyst’s judgment about what kind of work each claim is doing in the paper’s argument.”
Claim consolidation — when the three agents have over-split a claim into multiple narrower claims, the analyst merges. When they’ve under-split, the analyst splits.
Edge mapping — Step 6 (currently scaffolded) is meant to happen at review time when the analyst can see the full claim graph.
Spurious claim removal — the agents may surface claims that aren’t really propositions about the paper (methodological setups, schematic diagrams). The analyst removes these.
Missing claim addition — claims the agents missed entirely.

In its design, Step 5 is a curator-and-agents loop: the analyst sees the draft, makes judgment calls, and the agents’ work feeds into curator-quality output. The CLI honors this in --review-mode interactive.

The four review modes

`interactive`

Opens the draft as a YAML file in your $EDITOR. You revise it. On save+exit, the edited YAML is re-parsed against the DraftClaimTable schema. If the schema validates, the write step proceeds with your edits. If it fails, your edits are preserved at a .rescue.yaml for retry.

elife-extract write \
  --draft /tmp/out/draft-headley.json \
  --corpus-dir /tmp/corpus \
  --review-mode interactive

The YAML file you see is human-readable — claims are listed with all fields visible, the per-agent evidence quotes preserved as inline blocks. You can:

Edit claim sentences for clarity
Change role / claim-type assignments
Add new claims (just append a new entry following the schema)
Delete claims (remove the entry)
Adjust confidence
Add notes documenting your changes

This is the methodology’s intended operation. Cost: ~$5 in API + 15-30 min of analyst time per paper.

`external`

Runs an Opus pass that substitutes for the analyst at the review gate. The reviewer reads the paper plus the draft and produces a revised draft addressing the seven systematic biases prose-level extraction exhibits (see the external reviewer for the full design).

elife-extract write \
  --draft /tmp/out/draft-headley.json \
  --corpus-dir /tmp/corpus \
  --review-mode external

The revised draft is saved to <draft>.reviewed.json alongside the original — both are preserved for audit. Then the write step proceeds with the revised draft, no human in the loop.

This mode exists because the methodology’s curator-in-the-loop assumption breaks down for batch operation. With external, the CLI produces ~96% role-correct claims on average (vs ~65% in auto-approve) without requiring per-paper analyst time. Cost: ~$7 in API per paper, no analyst time.

`auto-approve`

Skips the review entirely. The reconciled draft is written directly as claim files.

elife-extract write \
  --draft /tmp/out/draft-headley.json \
  --corpus-dir /tmp/corpus \
  --review-mode auto-approve

This mode deliberately does not honor the methodology’s gate. Use it when:

You’re testing the extraction layer and don’t want the reviewer adding noise
You’ll manually review the resulting .md files after the write step
You’re iterating on the extraction prompts and want a fast feedback loop
You’re running the demo for someone and the rough output is fine

Don’t use it for production output unless you’re following with manual review. Quality: 100% recovery, ~65% role agreement, ~50% panel agreement on the Headley round-trip baseline. Cost: ~$5 per paper, no analyst time.

`dry-run`

Prints the draft to stdout without writing claim files. Useful for inspection.

elife-extract write \
  --draft /tmp/out/draft-headley.json \
  --corpus-dir /tmp/corpus \
  --review-mode dry-run

No file emission, no Opus reviewer call, no analyst interaction. Just shows you what would have been written. Use to verify a draft’s contents before committing to a write or before deciding which review mode to use for real.

Comparison

Mode	Cost / paper	Time / paper	Quality (role agreement, n=10 papers)	Use case
`interactive`	$5 + analyst	~5 min CLI + 15-30 min analyst	~95% (analyst-quality)	Production single-paper, full corpus quality
`external`	$7	~10-15 min	78% (n=10 mean)	Bulk operation, journal use, batch validation
`auto-approve`	$5	~5 min	65%	Testing, demos, when manual review of `.md` follows
`dry-run`	$5 (extract only)	~5 min	n/a (no write)	Inspection before committing

Operational guidance

The methodology’s interactive mode is the gold standard for corpus quality. The system’s external mode is the gold standard for unattended operation at the methodology’s quality bar minus the residual judgment calls that need a curator. The other two modes are for specific operational shapes; both are honest about their tradeoffs.

For most journal-augmentation use cases — surfacing claim structure for peer reviewers, anti-hallucination checks on cited references, batch extraction of recently-published papers — external is the right default. Reserve interactive for papers entering a permanent curated corpus where the analyst time is well-spent.

Implementation references

elife_extract/review.py — the interactive mode ($EDITOR integration, YAML round-tripping, rescue-file handling)
elife_extract/external_review.py — the external Opus pass
elife_extract/cli.py — cmd_write() dispatches based on --review-mode

Next steps

The external reviewer (Step 4.5) — full design of what external mode does
Review modes — when to use each — operational decision tree from the CLI’s perspective
Validation results — empirical performance of each mode across the 10-paper public corpus