Known limitations

The 10-paper sweep produced a clean picture of where the system works (recovery, role with external reviewer) and where it doesn’t (panel agreement, certain paper structures). This page documents the failure modes and what would be needed to address each.

Panel agreement at ~55%

The most consistent miss across all 10 papers. Mean panel agreement is 54.7%, well below the 90% threshold. Range is 36-74% per paper.

Why it happens. The Caption-reader extracts panel-by-panel — one candidate claim per panel. The reference often has claims that span multiple panels: panel: fig4, fig5 is the curator’s anchor. When the matcher pairs the CLI’s single-panel claim with the reference’s multi-panel claim, panel_match scores false even when the content is identical.

What it doesn’t mean. The CLI is not placing claims at the wrong panel — the per-panel extraction is correct. The mismatch is at the granularity layer: the curator consolidates across panels; the CLI doesn’t.

What would fix it. Two prompt iteration paths, both untested empirically:

Caption-reader prompt revision. Instruct the agent to preserve multi-panel anchors when the caption text references multiple panels (e.g., “Figure 4 and Figure 5 jointly demonstrate…”). The current prompt mentions this but doesn’t enforce it strongly enough.
Reconciler prompt revision. Instruct the reconciler to consolidate single-panel CLI claims that share content into multi-panel claims when the original prose paragraph spans multiple panels. This requires the reconciler to recognize content equivalence across the Caption-reader’s separate per-panel emissions.

Either change would lift the panel score. Both are testable via evaluate before committing.

For now, treat panel as a “directional” metric. A panel score of 50%+ on 10/10 papers means the panel anchoring is correct; the discount comes from granularity differences the matcher counts as failures. For most downstream uses (rendering claim graphs, surfacing claims to reviewers), the per-panel claims are usable as-is.

Role-lag on three paper structures

Three of 10 papers fell below the 75% role threshold individually:

Paper	Role agreement	What’s hard about it
kammer (foveal feedback fMRI)	57%	Heavy `control` role usage; reviewer prompt didn’t recognize implicit controls
rozak (neurovascular DL)	65%	Methods-paper structure; default prompts calibrated for biology findings
scheller (self-prioritization behavioral)	70%	`interpretation` vs `synthesis` distinction is judgment-heavy in cognitive papers

Each is a different failure mode:

kammer’s failure is reviewer-prompt under-coverage of the control role. A targeted prompt iteration moved kammer from 57% to 71% on a single change (still below threshold, but +14 points); see iteration discipline for the worked example.
rozak’s failure is structural — methods papers don’t have the hypothesis-prediction-test arc the prompts assume. A --prompt-variant methods-paper with revised role guidance would address this; not yet authored.
scheller’s failure is at the boundary of two roles where curators and the reviewer can both make defensible-but-different calls. This may not be fully closeable via prompt iteration; some judgment-call disagreements are residual.

For now, plan for analyst review on papers in domains where the system hasn’t been tuned. The validation harness tells you which papers need it (low role agreement); the analyst’s review at Step 5 closes the gap.

Step 6 (dependency mapping) is scaffolded

Claim files emit with empty belongings: and edge sections. The methodology says edge mapping is the analyst’s work; a future LLM-suggestion pass for edge inference is plausible but unimplemented.

Why it’s scaffolded. Edge mapping requires holistic argument-structure judgment — which claim requires which other, which tests which prediction, which dissociates-with which counterpart. This is harder than role classification (which is per-claim) because it’s relational. Sketches exist for an LLM-assisted edge-inference pass, but the empirical question of whether it produces analyst-quality edges hasn’t been answered.

What this means in practice. If you ship the CLI’s auto-extracted claim files directly, the dependency graph is empty. Downstream tooling that depends on edges (the synthesis pipeline, cross-paper graph traversal) doesn’t have data to work with. For most rendering use cases (claim cards in the Astro site, search by claim text) edges aren’t required.

For now, plan for analyst time on edge mapping for papers where the dependency graph matters. For corpus quality, this is the analyst’s intellectual contribution at Step 5.

Methodology fallback chain not implemented

The methodology specifies a fallback chain for paper text acquisition: PDF → GitHub README → eLife API abstract → focused web fetch. The CLI implements only the PDF path. If the PDF fetch fails, prepare() raises rather than degrading to the fallback paths.

Why it’s not implemented. The PDF path works for all 10 sweep papers. The fallback paths handle edge cases that haven’t surfaced in practice yet. Not implementing them keeps the implementation focused; a future iteration adds them when a paper actually fails the PDF path.

What would happen if a paper’s PDF doesn’t fetch. Current behavior: hard error with the HTTP status code. The user would need to manually fetch the paper text and pipe it in (no current path exists for this either; prepare.py would need a --paper-text-path flag).

Title / year / author extraction is heuristic

prepare.py derives the paper slug from PDF metadata: title from the first non-trivial line, year from the first 4-digit year, authors from a comma-separated capitalized-name line near the title. Each is best-effort:

Title may truncate. eLife titles often span two lines in the PDF; the heuristic picks the first line only. For Headley: extracted title is “Spatially targeted inhibitory rhythms” (the actual title is “Spatially targeted inhibitory rhythms differentially affect neuronal integration”).
Year may catch a citation. The first 4-digit year in the PDF is often a citation in the text, not the publication year. For Headley: extracted year was “2024” (publication year is 2026; the 2024 came from a cited reference).
Author affiliation superscripts leak. The author name extraction picks up “Drew B Headley1” with the affiliation marker 1 glued in.

What this affects. The derived slug. For Headley: headley-2024-spatially-targeted-inhibitory instead of headley-2026-inhibitory-rhythms. Functionally fine — the slug is just a directory name and a key in the corpus index; downstream rendering uses the title field from index.md (which is the truncated form too, but recoverable).

Override. Pass --paper-slug headley-2026-inhibitory-rhythms to extract or run. For batch operation, parse the slug from your input list rather than relying on the derivation.

The canonical script’s DOI field bug

The script at ~/Projects/mainenlab/elife-claim-trees/scripts/verify-references.py checks assertions[0].doi for the cited DOI on literature-context claims. The schema actually puts the cited DOI at top-level fm["doi"] for literature-context claims (top-level doi: is ~ for other roles per § 4.1).

The CLI’s verify-refs reads from the right field (top-level for literature-context). The canonical script’s check is in latent disagreement with the schema; it works by accident because the curator manually populated DOIs in both locations on the curated corpus, but newly-extracted literature-context claims would have the DOI at top-level only and the canonical script wouldn’t find them.

Fix. A small patch to ~/Projects/mainenlab/elife-claim-trees/scripts/verify-references.py to check top-level fm["doi"] first. Not in this CLI’s scope — the canonical script is in the corpus repo, not the extraction repo.

What’s out of scope

Things the system was never designed to do:

Step 8 verification. Per-paper Python (verification/<slug>/verify.py) re-runs the paper’s analysis against deposited code/data. The CLI’s claim files emit with empty reproductions: blocks for these scripts to populate.
Cross-paper graph resolution. Citations resolve to DOIs (via verify-refs), not to other papers’ claim entities. Building the cross-paper graph is a separate downstream operation.
Editorial workflow integration. The CLI is process-safe and produces stable JSON intermediates, but doesn’t ship with eLife-editorial-system adapters.
Domain portability. Prompts are calibrated for eLife neuroscience papers. Other domains (computational papers, social science, methods-only papers) need prompt variants.

Summary

Limitation	Severity	Mitigation
Panel agreement at ~55%	Medium (recovery and role both clear thresholds)	Prompt iteration on Caption-reader and reconciler
Role-lag on certain structures	Medium (3 of 10 papers individually below threshold; mean clears)	Prompt variants per paper type; analyst review for edge cases
Step 6 scaffolded	Medium (edges empty in CLI output)	Analyst review at Step 5 (or future LLM-assisted edge inference)
Fallback chain not implemented	Low (PDF path works for all sweep papers)	Add when a paper actually fails the PDF path
Heuristic title/year/author	Low (slug only; override available)	Pass `--paper-slug` or parse from input list
Canonical script DOI bug	Low (CLI’s `verify-refs` is correct)	Patch the canonical script

Next steps

Iteration discipline — the kammer worked example for prompt iteration
Cost and performance — operational tradeoffs
Adding a prompt variant — for domain or paper-type customization