Skip to content

The 8-step process

The eLife claim-trees methodology — described in docs/method.md § 3 of the corpus repo — is an eight-step process for translating a paper from prose into a claim graph. The system implements all eight steps, plus an extension at Step 4.5 (the external reviewer pass) that the methodology document doesn’t anticipate but that the empirical results justify.

This page situates the system in the methodology’s framework. For the canonical methodology document, read docs/method.md § 3 directly — it is the authority. For the system’s implementation of each step, follow the per-step links to the architecture and CLI sections.

The eight steps

Section titled “The eight steps”
#NameWhat happensAuthority
1PrepareLocate and fetch the paper, code repository, data deposit. Map figure structure (number of figures, panels per figure).docs/method.md § 3.3 Step 1 / prepare.py
2Abstract scanIdentify 2-4 top-level claims from the abstract — the paper’s main bets. These will become the synthesis or interpretation nodes at the top of the dependency graph.§ 3.3 Step 2 / folded into Step 3 in current implementation
3Three independent extractionsThree agents read the paper independently from partitioned slices: a Results-reader sees abstract + results prose; a Caption-reader sees figure captions panel-by-panel; a Structure-reader sees methods + supplements + code. No agent sees another’s output before submitting.§ 3.3 Step 3 / agents.py
4ReconciliationCompare the three extraction lists; mark each claim as high (all three agree), contested (two agree, one differs), or single-source (only one surfaced). The reconciled list is what goes to the review gate.§ 3.3 Step 4 / reconcile.py
4.5External reviewer ★An Opus pass that recovers structural claims (hypotheses, predictions) the prose-level extraction systematically misses. Substitutes for the curator at Step 5 when the CLI runs without a curator.(not in methodology — system extension) / external_review.py
5Review (the gate)The analyst reviews the draft claim table. Corrects claim sentences, reclassifies roles and types, adds missing claims, removes spurious ones. Nothing is written to disk until the table is approved.§ 3.3 Step 5 / review.py
6Dependency mappingFor each claim, identify which other claims it structurally requires, supports, refutes, etc. 14 typed edge types.§ 3.3 Step 6 / write.py (currently scaffolded — emits empty edge sections)
7Write claim filesGenerate UUIDs, write each claim as <corpus>/<paper-slug>/<claim-slug>.md per the schema. Write the paper’s index.md.§ 3.3 Step 7 / write.py
8VerifyFor each claim where data and code are available, run the analysis and compare the output to published numerics. Update the reproductions: block.§ 3.3 Step 8 / per-paper verification/<slug>/verify.py (out of CLI scope)

What the system does to the methodology

Section titled “What the system does to the methodology”

The system is faithful to all eight steps. It also extends the methodology in two ways:

Step 4.5 is new. The methodology was designed for a curator-and-agents loop — Step 5 review is a hard human gate. The system runs in environments without a curator (batch operation, integration handoffs, eLife-scale corpus extraction). The external reviewer pass substitutes for the curator at the review gate, producing curator-quality role classifications without requiring per-paper analyst time. See the external reviewer for the design rationale and empirical justification.

Step 6 is scaffolded but not full. The methodology calls for the analyst to map dependency edges between claims at write time. The CLI emits claim files with empty edge sections; the analyst populates them. A future LLM-suggestion pass for edge inference is plausible but unimplemented.

What the methodology does that the system doesn’t

Section titled “What the methodology does that the system doesn’t”

Steps 1 through 7 are the system’s scope. Step 8 — verification — is per-paper Python that re-runs the paper’s deposited analysis. The CLI’s claim files emit with empty reproductions: blocks for the per-paper verify.py scripts to populate. See ~/Projects/mainenlab/elife-claim-trees/verification/ for the existing per-paper verifiers.

Cross-paper graph resolution — when the corpus grows enough that citations resolve to specific claims in other papers, the entity-resolution layer that turns “Author (Year)” into “claim entity X” is separate from the CLI. verify-refs resolves citations to DOIs (anti-hallucination check); turning DOIs into claim references is downstream work.

Where the methodology and the implementation diverge

Section titled “Where the methodology and the implementation diverge”

A few places where the implementation makes choices the methodology doc doesn’t anticipate:

  • claim-type vocabulary. The methodology lists 5 values (empirical, interpretive, existence, synthesis, assessment); the curated corpus actually uses 9 (additionally hypothesis, prediction, scope, methodological). The schema reflects the methodology’s 5-value vocabulary; corpus claims that use the extended values may need adjustment at the review step.
  • Reconciliation strategy. The methodology describes the high/contested/single-source confidence taxonomy. The CLI implements --reconcile-strategy confidence-tagged as the default; alternative strategies (union, intersection-only, majority-vote) are exposed via the flag for empirical comparison.
  • Auto-approve mode. The methodology says nothing should be written before review; the CLI’s auto-approve mode skips the gate. Use this mode only when you intend to manually review the resulting .md files; otherwise the methodology’s quality bar isn’t met.

Where to read more

Section titled “Where to read more”