Evaluation
step · corpus openDeclaration 570497de676a has not been accepted by anyone. · no paper read under it
How well does each model profile recover a paper's claim tree?
Mechanical: re-runnable from its declared inputs.
zmainen/claim-graphs#24 — the decision is taken there; the propagation is taken here.
How it works
For each paper the sweep has touched, evaluate score aligns a re-extraction — produced under one model profile — to the best reference the paper has, and writes a scorecard under runs/. This layer gathers every one of them into review/evaluation.json: profile × paper × metric — recovery, precision, role, panel, edge recovery, parts — with the reference it was scored against and whether that reference was approved.
It measures and does not judge. The reference is the committed tree unless a person has approved a version (none has yet, so every row here scores against the committed tree and says so). Only the subagent profile and the two experiments run without a paid backend; the other profiles appear as not run: no backend credit, so the table names what is missing rather than omitting it.
What it found · added 2026-09-12
Of 7 profile × paper rows, 4 were scored and 3 could not run for want of a paid backend. The best recovery recorded so far is 82% of a paper's committed claims. No reference here has been approved, so every score is against a committed tree, not a tree a person has stood behind.
Rests on
- claim-tree · step
Feeds
Nothing — this is a leaf.
How it is defined
What this layer reads besides its dependencies. Each is a declared input: its content is hashed into every run, so editing one makes those runs stale.
The declaration names this path and the repository does not have it. An input that does not exist hashes to nothing, so it cannot make a run stale — the layer is declared to depend on something it is not in fact tracking.
The declaration names this path and the repository does not have it. An input that does not exist hashes to nothing, so it cannot make a run stale — the layer is declared to depend on something it is not in fact tracking.
A measurement, scored against the committed tree
Each row re-reads one paper under one model profile and scores the result against the best reference the paper has. No reference here has been approved by a person yet, so every score is against the committed tree — a comparison between two drafts, not a score against a reading someone has stood behind. The gold set (#82) is what turns the second into the first.
Three of the four profiles need a paid backend and have not run — no backend had credit
when this was built. They are shown as not run rather than omitted, so the
table names what is missing. The subagent profile and the two experiments
below it cost nothing: every model-answered step is a dumped prompt answered by a Claude
Code subagent.
Gädeke, by profile
| Profile | claims | recovery | precision | role | panel | edges | reference |
|---|---|---|---|---|---|---|---|
frontier | not run: no backend credit | ||||||
standard | not run: no backend credit | ||||||
open | not run: no backend credit | ||||||
subagent | 74 | 79% | 35% | 77% | 23% | 6% | claim-tree v1 (committed) |
fourth-reading experiment | 34 | 82% | 71% | 74% | 41% | 0% | claim-tree v1 (committed) |
frontier-tasks experiment | 26 | 76% | 88% | 84% | 40% | 0% | claim-tree v1 (committed) |
standard-tasks experiment | 44 | 76% | 52% | 80% | 20% | 0% | claim-tree v1 (committed) |
recovery is the fraction of the reference's claims the candidate found; precision the fraction of the candidate's claims that matched one; the rest are agreement on matched pairs. A high-recovery, low-precision row over-produced — it found the reference's claims and many more besides.
What each scored row is
subagent 74 claims · 79% recovery · 35% precision
The three-reader chain answered by a Claude Opus/Sonnet subagent, as the current committed tree; #96 (v3) is still open, so this scores the current committed tree against the original v1 via the v3 alignment. Precision is against 74 claims that include the parts and alternatives added by later layers.
fourth-reading 34 claims · 82% recovery · 71% precision
Experiment: one whole-paper reading in a single pass, answered by a subagent. Not a declared layer.
frontier-tasks 26 claims · 76% recovery · 88% precision
frontier lean reader tasks, subagent-answered, on the results slice
standard-tasks 44 claims · 76% recovery · 52% precision
standard reader tasks (recorded run), subagent-answered, on the results slice
The two experiments
The fourth reading. One whole-paper pass — abstract, introduction,
results, discussion, captions, tables, methods, the panel inventory and span ids in a
single call — recovered 82% of
the committed tree at 71%
precision, higher on both than the three-reader chain's current committed tree. What it
missed was specific: the alternatives the paper rules out (which the tree carries as
alt- hypotheses, raised by the stance layer, not by any single reading) and
two prediction/test pairs it collapsed into single empirical claims. The partition buys
what a paper argues against; a whole-paper reader matches it on what a paper
argues for.
Scaffolding versus model. Holding the model fixed (both answered by the same subagent), the frontier lean reader task and the standard reader task recovered the same 76% of the reference — but the standard task emitted 44 claims to the lean task's 26, with precision 52% against the lean task's 88%. The standard prompt's quantity guidance and signal-phrase tables buy over-splitting, not recall — which is the argument for the lean frontier tasks.
Inputs and outputs
- Reads, besides its dependencies
-
- scripts/evaluation_report.py · declared, and not in the repository — it hashes to nothing, so it cannot make a run stale
- extract/claim_graphs/evaluate.py · declared, and not in the repository — it hashes to nothing, so it cannot make a run stale
- Produces
-
- review/evaluation.json
- Views
-
- table — rendered above, over the 7 rows in the artifact
Running it
The command comes from the declaration, so this text and what actually runs cannot
diverge. pipeline.py run also runs the unmet dependencies first.
python3 scripts/pipeline.py run <paper> evaluation
Underneath, that runs python3 scripts/evaluation_report.py --write.