What this system is

The eLife claim-trees project asks: what would scientific publishing look like if every paper’s claims were extracted as discrete, machine-readable propositions, linked into a dependency graph, and verifiable against deposited code and data? The methodology doc — docs/method.md § 3 in the corpus repo — describes the eight-step process by which a paper is translated into a claim graph. The corpus at /papers/ shows the result: 12 papers, 310 claim files, 939 verified references.

This system is what makes that methodology executable at scale. It is a Python CLI that takes a paper DOI as input and produces a populated <paper-slug>/ directory of claim files as output. The methodology was carried out by hand for the curated 12-paper corpus; the system carries it out automatically, with the same schema, the same review-gate discipline, and (when run with the external reviewer) comparable role-classification accuracy.

The problem

Manual claim extraction is the bottleneck of the methodology. The 12-paper curated corpus took weeks of curator time. Scaling to a journal-sized corpus — 100s of papers per year, eventually thousands — requires automation that doesn’t sacrifice the schema’s structural integrity or the methodology’s quality bar.

The naive approach — give a single LLM the paper, ask for claims — fails in characteristic ways:

• Quantitative hallucination. A single agent reading the whole paper invents numbers when prose summarizes (“a large fraction”) rather than quoting the figure caption (“63 of 100 cells”).
• Wrong panel assignment. A single agent routinely attaches a claim to the wrong panel because it hasn’t read the panel-by-panel caption text carefully.
• Discussion contamination. A single agent extracts speculative interpretations from the discussion as if they were results, blurring the empirical / synthesis / interpretation distinction.
• Missed deductive structure. A single agent reads the prose and surfaces empirical findings, missing the implicit hypothesis-prediction-test scaffolding the curator infers from the modelling structure.

The bet

Three independent readers, each with a different partition of the paper, each constrained by a focused prompt, then a reconciliation step that aligns their outputs and a review pass that recovers the deductive structure none of them individually has the context to surface.

This is the three-agent partition at the heart of the system:

• Results-reader — reads the abstract and the results prose only. Captures the paper’s framing, hypotheses, predictions, synthesis, and interpretation claims as the prose presents them.
• Caption-reader — reads the figure captions, panel by panel. Captures the panel-level empirical claims with exact quantitative values and panel anchoring.
• Structure-reader — reads the methods, supplements, and code descriptions. Captures methodological capabilities, scope conditions, and structural features of the analysis.

No agent sees another’s output before submitting. They are independent witnesses; their disagreement and convergence is itself the signal.

A reconciliation step (Opus) aligns the three lists into a confidence-tagged draft. A review gate (either human via $EDITOR or an Opus structural-inference pass) catches the role-classification gaps that prose-level extraction misses. A write step emits the schema-conformant .md files. A CrossRef pass verifies cited DOIs.

The result

On the 10 public eLife papers in the curated reference corpus, the system in --review-mode external mode achieves:

Metric	Mean	Median	Threshold	Status
Claim recovery	95%	98%	≥ 80%	✅
Role agreement	78%	76%	≥ 75%	✅
Panel agreement	55%	53%	≥ 90%	❌

Recovery is the load-bearing metric — every reference claim has a corresponding extraction. Role agreement clears its threshold; the persistent panel miss is a documented limitation (multi-panel claims like panel: fig4, fig5 get scored as panel-mismatch when the CLI emits a single-panel claim with the same content). See validation results for the per-paper breakdown.

Tip

The system is built so that every prompt change can be measured. The evaluate subcommand re-runs the pipeline on the curated corpus and produces an aggregate scorecard. This makes prompt iteration a discipline, not an experiment — change → re-run → diff aggregate → decide. The kammer iteration (57% → 71% role on a single targeted prompt change) is the canonical worked example.

What it is not

• Not a replacement for analyst judgment. Step 5 of the methodology is a hard review gate. With --review-mode interactive, the human is in the loop. With --review-mode external, an Opus pass substitutes for the curator’s structural inference — empirically equivalent for most claims, but the gate is the analyst’s call.
• Not a verification system. Step 8 of the methodology — running the paper’s analysis on deposited code/data and comparing reproduced numerics to the published figures — is per-paper Python (verification/<slug>/verify.py in the corpus repo). The CLI’s claim files emit with empty reproductions: blocks for the per-paper verifier to populate.
• Not a graph-resolution layer. When the corpus grows and citations need to resolve to other papers’ claims (rather than just to DOIs), that’s a separate piece of infrastructure built on top of verify-refs output.
• Not specific to neuroscience. The prompts are calibrated against the curated neuroscience corpus, but the schema is domain-agnostic. Other domains may need a --prompt-variant rather than a fork.

Where to go next

• Quick demo — one paper end to end — see the system run on Headley 2026
• The pipeline at a glance — the architecture diagram
• Install and configure — get it running on your infrastructure
• The 10-paper sweep results — the empirical case