Design decisions

This page is an architectural-decision-record (ADR)-style summary of the substantive choices the system embeds. Each decision answers: what was the choice, why was it made, what alternatives were considered, what would change if revisited.

For the operational consequences of each decision, see the architecture and CLI sections. This page is for the why.

ADR-1: Three independent agents, not one big call

Decision. The extraction step partitions the paper into three slices (abstract+results, captions, methods) and runs three independent agents on the slices.

Rationale. Empirical: a single agent reading the whole paper exhibits characteristic failure modes (quantitative hallucination, wrong panel assignment, missed methodological claims). Each failure mode is reduced when the agent is constrained to read a focused slice. The three slices map naturally to three classes of claims (interpretive prose, panel-grounded numerics, methodological structure).

Alternatives considered.

• One large agent reading the whole paper. Tested in early experiments; produced more hallucination at quantitative grounding and worse role classification.
• Per-section agents (abstract, results, methods, discussion). Considered; rejected because the abstract/results split offers no real partition (the same agent should see both).
• Per-figure agents. Considered; rejected because per-figure granularity loses the cross-figure synthesis claims the Results-reader catches.

What would change if revisited. A fourth agent specifically for the discussion section could surface speculative-interpretation claims the Results-reader is told to skip. Cost: one more Sonnet call per paper. Worth testing if the curated corpus’s discussion-derived claims are systematically missed by the current setup.

ADR-2: Sequential agent execution, not concurrent

Decision. The three extraction agents run sequentially in the current implementation.

Rationale. Simplicity. Sequential execution makes error handling, retry logic, and progress logging straightforward. Each agent is independent (no shared state); they could trivially run concurrently via asyncio.gather().

Alternatives considered.

• Concurrent via asyncio. Would cut Step 3 wall time by ~2x. Tested informally; works but adds complexity for marginal benefit.
• Concurrent via threading. Same outcome as asyncio for I/O-bound calls.

What would change if revisited. For corpus-scale operation where wall time matters more than debug simplicity, switch to asyncio. The change is ~10 lines in agents.run_all_agents().

ADR-3: Opus for reconciliation and review, Sonnet for extraction

Decision. Sonnet 4.6 for the three extraction agents; Opus 4.6 for reconciliation and external review.

Rationale. Empirical: extraction is a constrained “read prose, surface propositions” task that Sonnet handles well at lower cost. Reconciliation and external review are holistic synthesis tasks where Opus is meaningfully better. The Phase F prompt iteration on Sonnet for structural inference produced no improvement; the Opus reviewer pass produced a 30-point lift on Headley.

Alternatives considered.

• Opus for everything. Tested; ~3x cost, ~10% better extraction quality (within noise). Not worth it.
• Sonnet for everything. Tested; the reconciler and reviewer noticeably worse without Opus’s holistic judgment.
• Haiku for extraction. Tested partially; produces measurably worse role classification, particularly on the methodological/scope distinction.

What would change if revisited. When Anthropic releases a new model tier (e.g., a hypothetical Sonnet 5 with stronger structural inference), the Opus role might collapse into Sonnet. Re-evaluate via evaluate --model-reconcile claude-sonnet-5 when models change.

ADR-4: Step 4.5 (external reviewer) as a system extension to the methodology

Decision. The system adds a reviewer pass between Step 4 (reconciliation) and Step 5 (review gate) that the methodology doesn’t include.

Rationale. The methodology was designed for a curator-and-agents loop. Step 5 review is a hard human gate. The system runs in environments without a curator (batch, integration, journal-scale extraction). Without a substitute, auto-approve mode produces drafts with systematic role-classification gaps. The reviewer pass closes most of that gap empirically (65% → 96% role agreement on Headley).

Alternatives considered.

• Make the curator optional but mandatory in spirit. Initial position; broke down operationally for batch use.
• Push more capability into the extraction agents. Tested in Phase F (revised Sonnet prompts to surface predictions). Produced no improvement — the deductive structure isn’t in the prose for the agents to extract.
• Skip Step 4.5; rely on per-paper analyst review at Step 5. Acceptable for low-volume use; impractical for journal-scale.

What would change if revisited. If a future model tier handles structural inference natively at extraction time (so the Sonnet agents could surface predictions and hypotheses correctly), Step 4.5 becomes redundant. Until then, it’s load-bearing for unattended operation.

ADR-5: Pydantic schema enforcement at every step

Decision. Each step’s wire format is a Pydantic model. JSON outputs are validated before passing to the next step; validation failures halt the pipeline.

Rationale. Schema-invalid outputs are the easiest class of failure to detect cheaply. Halting on validation failure prevents downstream corruption. The Pydantic models also serve as documentation — schema.py is the canonical reference for what each step produces.

Alternatives considered.

• Trust the agents; pass through whatever JSON they produce. Brittle; an agent’s malformed output corrupts the downstream draft.
• Validate at the boundary only (input to write step). Less precise — failures surface late.

What would change if revisited. If we add a tier of agents whose output isn’t well-formed JSON (e.g., free-text reasoning chains), the schema becomes more lenient at that step. Currently every step’s output is structured.

ADR-6: Extended ClaimType vocabulary in implementation but not (yet) in the official schema

Decision. The methodology document (docs/method.md § 4.1) lists 5 claim-type values. The actual curated corpus uses 9 (additionally hypothesis, prediction, scope, methodological). The CLI’s Pydantic Literal initially mirrored the corpus reality; the user reverted to the 5-value methodology spec.

Rationale. The schema should be the source of truth, but the corpus and the methodology disagree. The methodology spec is authoritative; corpus claims using extended values may need adjustment at the review step.

Alternatives considered.

• Match the corpus reality. Schema accepts 9 values; the methodology doc is updated to match.
• Match the methodology. Schema accepts 5 values; corpus claims with extended values get adjusted at review.

What would change if revisited. This is an open tension. The methodology should be revised to match observed practice (9 values), or the corpus should be re-curated to match the spec (5 values). Either path resolves the disagreement; the CLI follows whichever wins.

ADR-7: Streaming for all LLM calls

Decision. All LLM calls use the Anthropic SDK’s streaming interface, even when the response is collected in full before processing.

Rationale. The SDK requires streaming for outputs that may exceed 10 minutes. The reconciler and reviewer can produce large outputs (~30k+ tokens). Using streaming unconditionally avoids the time-budget cliff.

Alternatives considered.

• Non-streaming with a 10-minute timeout. Brittle for the reviewer pass on large papers.
• Conditional streaming based on expected output size. More complex; no benefit over always streaming.

What would change if revisited. None likely.

ADR-8: Step 6 (dependency mapping) scaffolded, not auto-inferred

Decision. Claim files emit with empty belongings: and no top-level edge keys. The methodology calls for the analyst to map edges at write time; the CLI doesn’t attempt automated edge inference.

Rationale. Edge mapping requires holistic argument-structure judgment — which claim requires which other, which tests which prediction, which dissociates-with which counterpart. This is harder than role classification (per-claim) because it’s relational. The empirical question of whether an LLM can produce analyst-quality edges hasn’t been answered; until it has, scaffolding is the honest position.

Alternatives considered.

• Stub edges with placeholder values. Worse than empty — analysts would have to identify and remove placeholder noise.
• LLM-suggested edges with confidence scores. Sketched; not implemented. Would require a new prompt and evaluation methodology.
• Programmatic edges via slug pattern matching (e.g., interprets- slugs always interpret cited prior work). Limited but could populate some edges without LLM cost.

What would change if revisited. If a --review-mode external already produces curator-quality role classifications, the same Opus call could be extended to map edges. This would be a meaningful new capability; cost would be ~$1-2 more per paper.

ADR-9: Heuristic PDF metadata extraction (title, year, authors)

Decision. prepare.py extracts paper title, year, authors via heuristic regex on the first ~3000 chars of the PDF text. Override via --paper-slug.

Rationale. eLife PDFs have variable layout; structured XML metadata isn’t always available via simple extraction. The heuristic is good enough for slug derivation in most cases; failures are recoverable via the override flag.

Alternatives considered.

• Use the eLife API for metadata. More reliable for eLife papers; doesn’t generalize to other publishers. Worth implementing as a parallel path.
• Use a structured PDF metadata library (Crossref API by DOI, GROBID). More accurate but adds a dependency.

What would change if revisited. When publisher-API integration is added (eLife API for eLife papers, generic CrossRef for others), metadata extraction shifts to the API, and the heuristic becomes a fallback.

ADR-10: Markdown for state and config

Decision. Configuration and state are markdown (with YAML frontmatter where structured data is needed), not JSON / TOML / YAML files.

Rationale. The HAAK platform’s conventions (Foundation 13: code-and-reasoning.md) specify markdown as the default carrier of agent-readable state. The CLI inherits this convention. Markdown is reviewable by humans, version-controllable, diffable, and parseable.

Alternatives considered.

• JSON/YAML config files. The Python ecosystem default. Rejected per HAAK convention.
• Database-backed state. Overkill for the CLI’s data scale.

What would change if revisited. None likely; this is an architectural commitment that pervades the system.

Implementation references

• home/collabs/elife/claim-trees/jobs/extract-cli.md — the worklog tracking each phase’s design decisions
• home/collabs/elife/claim-trees/.haak/method.md — HAAK-side method doc bridging methodology and implementation
• prompts/external-reviewer.md — embodies ADR-4 in concrete prompt form

Next steps

• Code structure — module organization that follows from these decisions
• Adding a prompt variant — the extension point that respects the architecture
• Iteration discipline — the empirical discipline that validates each decision