Review modes — when to use each

The write subcommand’s --review-mode flag determines what happens at Step 5 of the methodology. Four modes, each appropriate for a different operational shape. This page is the decision tree.

For the architectural rationale (why a review gate exists at all), see the review gate.

The four modes at a glance

Mode	What happens	Cost / paper	Time / paper	Role agreement (n=10 papers)
interactive	Opens draft in $EDITOR; analyst edits	$5 + analyst time	~5 min CLI + 15-30 min analyst	~95% (analyst-quality)
external	Opus pass revises draft; no human in loop	$7	~10-15 min	78% (mean)
auto-approve	Skips review; writes draft as-is	$5	~5 min	65%
dry-run	Prints draft; no write	$5 (extract only)	~5 min	n/a

Operational decision tree

Are you adding a paper to a curated corpus where quality matters more than throughput?
├── YES → --review-mode interactive
│         (analyst review, ~30 min/paper, curator-quality output)
│
└── NO  → Are you running on many papers without per-paper analyst time?
          ├── YES → Will the output be reviewed at the .md file level later?
          │         ├── YES → --review-mode auto-approve
          │         │         (skip review, manual review of files after)
          │         │
          │         └── NO  → --review-mode external
          │                   (Opus reviewer; recommended for unattended operation)
          │
          └── NO  → Just inspecting an extraction's quality?
                    └── --review-mode dry-run
                        (extract + print, no claim files written)

Worked examples

Operational shape 1 — Single paper, full corpus quality

You have one new eLife paper that should join your curated claim-graph corpus. The analyst is available to review the draft.

elife-extract extract --doi 10.7554/eLife.<id> --corpus-dir ./claims
elife-extract write \
  --draft ./out/draft-<slug>.json \
  --corpus-dir ./claims \
  --review-mode interactive

The extract step produces the reconciled draft in ~5 min. The write step opens the draft as YAML in your $EDITOR. You spend 15-30 minutes:

• Adjusting role classifications (the system gets ~95% right; you fix the rest)
• Splitting/merging claims where the agents over- or under-decomposed
• Adding concepts: lists from your controlled vocabulary
• Setting epistemic: per claim based on argumentative weight
• Authoring displayClaim: and shortClaim: for high-traffic claims

Save and exit. The claim files are written to ./claims/<slug>/.

Total cost: ~$5 + your time. Output: curator-quality.

Operational shape 2 — Bulk corpus build for journal augmentation

You have N papers (could be 10, could be 1000) that you want to extract for surfacing claim structure to peer reviewers. Per-paper analyst time isn’t available.

for doi in 10.7554/eLife.95562 10.7554/eLife.<other> ...; do
  elife-extract extract --doi $doi --corpus-dir ./claims
done

for draft in ./out/draft-*.json; do
  elife-extract write --draft $draft --corpus-dir ./claims --review-mode external
done

The extracts can run in parallel at the shell level (subject to your Vertex rate quota). The writes run sequentially (each one is a single Opus call, easy to parallelize if needed).

Cost: ~$7 per paper × N. Time: ~15 min per paper sequential (or fewer with parallelism).

For 100 papers: ~$700, ~25 hours sequential. For 1000 papers: ~$7K, ~10 days sequential or ~3 days with 3-way parallelism.

Output: ~78% role agreement on average (matches the 10-paper sweep). For peer-review augmentation, this quality is sufficient — reviewers see structured claims, citations are anti-hallucination-checked, the schema is consistent.

Operational shape 3 — Test, demo, or batch where you’ll review the .md files manually

You want to extract a paper quickly for a demo, or you’ll do all the review at the file level after the write step.

elife-extract run --doi 10.7554/eLife.<id> --corpus-dir ./claims

The run subcommand chains extract + write (auto-approve) + verify-refs. You get claim files in ~5-7 minutes for ~$5.

Then either:

• For a demo: open a few claim files, show the schema, point at the panel anchors and evidence quotes. Done.
• For real usage: open each claim file, manually adjust roles / claim text / panel as needed. This is essentially interactive mode but at the file level instead of the YAML draft level.

For most demos, auto-approve quality is fine. For actual corpus integration, this path takes longer than interactive because the file-level review lacks the YAML draft’s overview affordances.

Operational shape 4 — Inspecting before committing

You’re not sure if the extraction came out clean. You want to see the draft before deciding which review mode to use for real.

elife-extract write \
  --draft ./out/draft-<slug>.json \
  --corpus-dir ./claims \
  --review-mode dry-run

The CLI prints the draft to stdout. No Opus reviewer call, no file writes. You read it; if it looks good, re-run with --review-mode external (or interactive) to actually write.

Cost: $0 (the extract was already done; dry-run adds nothing). Time: <1s.

When external is wrong

Three cases where the Opus reviewer pass is the wrong choice:

1. You want to test the extraction layer specifically. When iterating on the Sonnet extraction prompts (e.g., A/B testing prompt variants), running the reviewer on each variant adds noise — it’s a non-deterministic transformation that may mask the extraction-level signal. Use auto-approve for these comparisons, then test the reviewer separately.

2. The paper has no implicit deductive structure to recover. Atlas papers, methods papers, observational surveys often don’t have the hypothesis-prediction-test arc the reviewer is calibrated for. Empirically (Artiushin spider atlas, n=1) the reviewer doesn’t add nonsense in these cases — it preserves the observational structure rather than inventing hypotheses — but it doesn’t add value either. Save the $2.

3. You already plan to manually review the resulting .md files. Running the reviewer pass adds $2 and 3 minutes per paper for a quality lift the analyst would otherwise capture in their file-level review. Skip the reviewer; let the analyst do that work directly.

When interactive is overkill

If you’re processing 10 papers and only spot-checking a sample, external is the right default. If you’re processing 100 papers and you’ve already manually reviewed enough samples to trust the system, external is the right default. interactive is for cases where each paper’s curator-quality output matters per-paper, not just on average.

Implementation references

• elife_extract/review.py — interactive mode ($EDITOR integration)
• elife_extract/external_review.py — external Opus pass
• elife_extract/cli.py cmd_write() — dispatch logic

Next steps

• Batch operation — running on many papers efficiently
• Cost and performance — per-step / per-model breakdown
• The external reviewer — full design of the Opus pass