What the curator does, what the system does
The methodology was designed for a curator-and-agents loop. The system was built to run with or without a curator at the loop’s center. Knowing where the curator’s judgment is genuinely required, where it’s helpful but substitutable, and where the system can do it directly determines which review mode is right for your operation.
A taxonomy of work
Section titled “A taxonomy of work”Eight categories of work happen in producing a claim graph. Some are mechanical, some are intellectual, some are inferential. The system handles the mechanical and inferential cleanly; the intellectual is where review modes differ.
| Work category | Example | Who does it (system) | Curator value |
|---|---|---|---|
| PDF fetching, text slicing | Identify the abstract / results / methods boundaries in the PDF | System (deterministic) | None — analyst doesn’t intervene here |
| Surface extraction | Pull each declarative claim from the prose | System (Sonnet × 3) | None — the agents are reliable readers of what’s in the prose |
| Quantitative grounding | Quote the paper’s exact numbers (5.5 Hz, p < 0.001) | System (Caption-reader) | None — the prompt enforces verbatim quoting |
| Panel assignment | Anchor a claim to figure 4A vs figure 5B | System (Caption-reader) | Low — system handles single-panel; multi-panel sometimes needs adjustment |
| Cross-agent reconciliation | Decide that two phrasings of the same finding are one claim | System (Opus reconciler) | Low — semantic matching is what Opus is good at |
| Role classification (surface roles) | Decide if a claim is empirical or interpretive | System | Low to medium — system handles most cases; ~76% role agreement on the 10-paper sweep |
| Role classification (functional roles) | Decide if a claim is empirical or control (functional vs contentful) | System (with external reviewer) + Curator | Medium to high — the control role specifically depends on what the claim does in the paper’s argument |
| Structural inference (hypothesis layer) | Surface the implicit hypothesis the paper bets on | System (with external reviewer) | Medium — the external reviewer recovers most explicit hypotheses; deeply implicit ones are still curator territory |
| Structural inference (prediction layer) | Surface predictions implicit in the modelling structure | System (with external reviewer) | Medium — same as above; reviewer recovers ~all of Headley’s, partial recovery on more complex papers |
| Edge mapping | Identify which claims requires, supports, tests, dissociates-with which others | Curator (system emits empty edge sections) | High — Step 6 is intentionally analyst work |
| Epistemic assessment | Mark a claim’s overall support as strong / moderate / weak / contested | Curator (system emits tentative) | High — requires reading the paper’s argument structure, not just its prose |
Auxiliary phrasings (displayClaim, shortClaim) | Write a softer rendering for body text and a tooltip-sized clause | Curator (system doesn’t author) | High — needs domain feel for which phrasings work in which contexts |
| Concept tagging | Add controlled vocabulary terms ([dopamine-transporter, clearance, dorsal-striatum]) | Curator (system emits empty list) | High — requires familiarity with your concept vocabulary |
What this means in practice
Section titled “What this means in practice”For the system’s automated path (--review-mode external):
- The output is a complete claim graph with claim text, panel anchors, role classifications, evidence quotes — everything needed for downstream rendering and search.
- It is not a complete corpus entry. Edge sections are empty. Epistemic field defaults to
tentative. Auxiliary phrasings are absent. Concepts are unspecified. - You can ship this as the claim graph for review purposes (the structure is there), but for full corpus integration you should plan for analyst time on the schema’s “high-curator-value” fields.
For the curator-in-the-loop path (--review-mode interactive):
- The system saves you ~95% of the work. Surface extraction, panel assignment, role classification, evidence quoting, reconciliation — all done.
- Your time is spent on the intellectual layer: edge mapping (Step 6), epistemic assessment, auxiliary phrasings, concept tagging.
- Estimated curator time per paper: 15-30 minutes for the system-prepared review, vs 4 hours from scratch.
For the eLife journal-augmentation path (the system runs unattended on submitted papers):
--review-mode externalproduces drafts that are ~96% role-correct on average. The remaining 4% of role disagreements are mostly defensible (the curator and the reviewer have different but reasonable readings).- For editorial use cases — surfacing claim structure for reviewers, anti-hallucination on cited references — this quality is sufficient.
- For permanent corpus inclusion, an analyst pass adds the high-value fields and resolves the role-classification ambiguities.
The empirical case for the external reviewer
Section titled “The empirical case for the external reviewer”The methodology says Step 5 review is a hard gate. The empirical question is: how much quality do you lose if you substitute Opus for the curator at that gate?
On the Headley round-trip:
- Without external reviewer: 65% role agreement with the curated reference
- With external reviewer: 96% role agreement
On the 10-paper sweep:
- With external reviewer: 78% mean role agreement (median 76%), 95% mean recovery
For the operational shapes that the methodology’s design implicitly assumed wouldn’t be possible (batch operation at journal scale), the external reviewer is the substitute. The remaining 22% role disagreement is the residual analyst value — for projects where the corpus quality matters more than throughput, that’s where the curator goes.
Operational summary
Section titled “Operational summary”| Operation | Review mode | Curator effort per paper |
|---|---|---|
| One paper, full corpus quality | interactive | 15-30 min |
| Bulk corpus build, full quality | interactive | 15-30 min × N papers |
| Bulk corpus build, journal use | external | 0 (per-paper); spot-check sample |
| Editorial augmentation | external | 0 (system-only); review by reviewers in their normal workflow |
| Validation / iteration | evaluate | 0 (automated benchmark) |
Next steps
Section titled “Next steps”- Review modes — when to use each — the operational decision in CLI terms
- The external reviewer — what Step 4.5 does and why it exists
- Validation results — empirical performance across the 10-paper public eLife corpus