How we measure quality

The system makes specific empirical claims: 95% claim recovery, 78% role agreement, 55% panel agreement on the 10-paper public eLife corpus. This page explains how those numbers are produced. The same harness is what you’d use to validate any prompt change, model substitution, or new review mode.

The mechanism is the evaluate subcommand. The conceptual frame is round-trip testing.

The round-trip

Round-trip testing on a single paper:

1. Reference exists. A curator has hand-extracted claims for the paper. These are the gold standard — ~/Projects/mainenlab/elife-claim-trees/claims/<paper-slug>/*.md.
2. Run the pipeline. elife-extract evaluate runs the full pipeline (prepare → extract → reconcile → optional external review → write) on the paper, given the DOI from the reference’s index.md.
3. Score. An Opus matcher reads both the reference claims and the CLI’s output claims and aligns them: for each reference claim, find the best matching CLI claim (or null if no match).
4. Compute metrics. From the alignment, derive recovery (% of reference claims matched), panel agreement (% of matched claims with same panel), role agreement (% of matched claims with same role).

This generalizes to a corpus: the harness loops over papers, scores each, then aggregates per-metric mean and median.

What the matcher does

The matcher is an Opus call. The prompt is at tests/headley_roundtrip.py (the original) and elife_extract/evaluate.py (the CLI integration). The matcher receives:

• REFERENCE: the curator’s claim list (slug, claim text, panel, role)
• CLI: the system’s output claim list (same fields)

And produces a JSON object:

{
  "matches": [
    {
      "ref_slug": "distal-inhib-drops-firing-02hz",
      "cli_slug": "doubling-distal-inhibition-reduces-firing",
      "match_quality": "exact",
      "panel_match": true,
      "role_match": true,
      "notes": ""
    },
    {
      "ref_slug": "hypothesis-distinct-compartmental-roles",
      "cli_slug": "perisomatic-distal-dendritic-inhibition-serve",
      "match_quality": "partial",
      "panel_match": "n/a",
      "role_match": false,
      "notes": "CLI captures only the perisomatic half of the hypothesis"
    },
    ...
  ]
}

For each reference claim, the matcher picks the best CLI match, scores match quality (exact / partial / none), and indicates whether the CLI’s panel and role agree with the reference’s.

The metrics

Claim recovery = (# reference claims with a CLI match) / (# reference claims)

The load-bearing metric. Recovery is “did the CLI find every claim the curator extracted?” Misses indicate the extraction pipeline is missing things the curator caught.

Panel agreement = (# matched claims with panel_match: true) / (# matched claims)

For matched claims (recovery numerator), did the CLI assign the same panel as the curator? Multi-panel reference claims (panel: fig4, fig5) often score panel-mismatch when the CLI emits a single-panel claim with the same content; this is the documented panel limitation.

Role agreement = (# matched claims with role_match: true) / (# matched claims)

For matched claims, did the CLI assign the same role (hypothesis / prediction / empirical / etc.)? Role disagreements cluster in functional roles — control (where the curator sees an empirical claim’s role as ruling out an alternative), synthesis (where the curator integrates), interpretation (where the curator maps to broader theory).

Match quality breakdown

The matcher distinguishes exact and partial matches:

• exact — the CLI claim asserts the same proposition with the same direction and entities; phrasing may differ.
• partial — the CLI claim captures a strict subset or superset of the reference proposition. Often the CLI split a reference claim into multiple narrower CLI claims; the matcher picks the closest.
• none — no CLI claim captures the reference proposition.

Exact + partial = recovered. The exact/partial split is informative even within a single recovery score — high partial means the CLI is over-decomposing, high exact means the CLI’s grain matches the curator’s.

What the metrics don’t capture

CLI claims with no reference match. The CLI typically emits 2-3x more claims than the reference (76 vs 26 on Headley). Some of these are real claims the curator chose not to surface (over-extraction); some are spurious (the agents read methodological setups as claims). The metrics measure recovery, not precision — they don’t penalize over-extraction directly.

For precision, you’d want to inspect the unmatched CLI claims and adjudicate. The scorecard surfaces the first 10 unmatched CLI claims for this purpose; full triage requires manual review.

Subjective role calls. When a claim is genuinely ambiguous between empirical and control, the matcher’s role_match is just “is the CLI’s role the same as the reference’s role?” without arbitrating which is correct. A 78% mean role agreement might actually represent ~85% defensible-role-call agreement plus ~7% where reasonable curators would disagree.

Edge mapping. The CLI emits claim files with empty edge sections; the reference has populated edges. We don’t attempt to score edge agreement — Step 6 is intentionally analyst work in the methodology.

Reproductions. The CLI emits empty reproductions: blocks; the reference has populated reproduction status from per-paper verify.py. Out of scope for round-trip scoring.

Iteration discipline

The evaluate subcommand is the validation harness for any change you propose to the pipeline:

1. Establish a baseline scorecard from the current configuration: evaluate --all --review-mode external against the curated reference.
2. Make the change (revise a prompt, swap a model, add a step).
3. Re-run evaluate --all with the change.
4. Diff aggregate scorecards. Recovery and role both up (or one up and the other holds): the change is a net win. Either drops materially: the change is a regression.

The kammer iteration is the canonical worked example — see iteration discipline for the case study.

Cost of validation

Per-paper round-trip cost: ~$10 (extract ~$5 + external review ~$2 + matcher ~$3).

Full 10-paper sweep: ~$100, ~85 min sequential.

This is the cost of the discipline — every prompt change validated empirically before deployment. Without the harness, prompt changes are guesses; with it, they’re hypothesis tests with measured outcomes.

Implementation references

• elife_extract/evaluate.py — score_against_reference(), aggregate_report()
• elife_extract/evaluate.py MATCHER_PROMPT — the matcher’s instructions
• tests/headley_roundtrip.py — the original standalone scorer (now a thin wrapper around evaluate)

Next steps

• The 10-paper sweep results — the empirical case for the system
• Known limitations — what the metrics tell us is hard
• Iteration discipline — kammer prompt iteration as a worked example