Skip to content

Code structure

This page is the orientation for someone who will read or modify the source. For the public API, see API reference. For the operational behavior, see the architecture pages.

Repository layout

Section titled “Repository layout”

The system lives in a single Python package at extract/ (in the haak repo: home/collabs/elife/claim-trees/extract/; in the published shared repo: under the corpus root).

extract/
├── pyproject.toml                  # package metadata, entry point, deps
├── README.md                       # developer entry point
├── INTEGRATION.md                  # eLife-collaborator-facing recipe
├── DOCS-SPEC.md                    # documentation site spec (this site)
├── prompts/                        # the load-bearing artifacts
│   ├── results-reader.md
│   ├── caption-reader.md
│   ├── structure-reader.md
│   ├── reconciler.md
│   └── external-reviewer.md
├── elife_extract/                  # the Python package
│   ├── __init__.py                 # version
│   ├── cli.py                      # argparse + subcommand dispatch (~400 lines)
│   ├── config.py                   # Config dataclass + env resolution (~120 lines)
│   ├── schema.py                   # Pydantic models for the wire format (~80 lines)
│   ├── prepare.py                  # Step 1: PDF fetch + slice (~200 lines)
│   ├── agents.py                   # Steps 2-3: extraction agents (~150 lines)
│   ├── reconcile.py                # Step 4: reconciliation (~120 lines)
│   ├── external_review.py          # Step 4.5: Opus reviewer pass (~110 lines)
│   ├── review.py                   # Step 5: human review gate (~110 lines)
│   ├── write.py                    # Steps 6-7: edges + claim files (~200 lines)
│   ├── verify_refs.py              # CrossRef DOI resolution (~250 lines)
│   └── evaluate.py                 # round-trip scoring (~300 lines)
└── tests/
    ├── headley_roundtrip.py        # standalone Headley scorer (legacy)
    ├── headley-roundtrip.md        # v1 scorecard (auto-approve baseline)
    ├── headley-roundtrip-v2.md     # v2 scorecard (revised prompts)
    ├── headley-roundtrip-v3-external.md  # v3 scorecard (external reviewer)
    ├── aggregate-scorecard-10paper.md     # 10-paper sweep result
    └── kammer-iterated-scorecard.json    # kammer iteration delta

Total: ~2200 lines of Python, ~1500 lines of prompt text, plus the docs site under site/src/content/docs/docs/ (this site).

Module organization

Section titled “Module organization”

The package is organized by methodology step, not by abstraction layer. Each module corresponds to one or two steps of the 8-step methodology:

ModuleMethodology stepLines
prepare.pyStep 1 (Prepare)~200
agents.pySteps 2-3 (Abstract scan + three extractions)~150
reconcile.pyStep 4 (Reconciliation)~120
external_review.pyStep 4.5 (External reviewer)~110
review.pyStep 5 (Review gate)~110
write.pySteps 6-7 (Edge mapping + write)~200
verify_refs.py(Adjunct) CrossRef DOI resolution~250
evaluate.py(Adjunct) Round-trip scoring~300

Plus three cross-cutting modules:

  • cli.py — the argparse-based CLI; one handler per subcommand, dispatching to the per-step modules
  • config.py — Config dataclass + env-var resolution; consumed by every other module
  • schema.py — Pydantic models for the wire format between steps

Reading order

Section titled “Reading order”

If you’re new to the codebase, read in this order:

  1. pyproject.toml — see what dependencies and entry points are defined
  2. config.py — understand the Config dataclass; every other module takes one
  3. schema.py — the Pydantic models. These are the wire format; they tell you what data flows between steps
  4. cli.py — the user-facing entry point; each cmd_* handler shows the orchestration
  5. prepare.py through verify_refs.py in that order — follow the pipeline in execution order
  6. evaluate.py — the validation harness; ties everything together

The prompts directory is the next thing to read — those are the load-bearing artifacts of the system.

Naming conventions

Section titled “Naming conventions”
  • Module names match methodology steps where possible (prepare, reconcile, review)
  • Function names are imperative (run_agent, verify_refs, score_against_reference)
  • Class names describe the data they hold (PreparedPaper, DraftClaimTable, PaperScorecard)
  • CLI subcommand names are direct verbs (extract, write, evaluate)
  • Pydantic field names use snake_case to match Python convention; YAML serialization preserves these as-is

What’s intentionally not abstracted

Section titled “What’s intentionally not abstracted”

A few places where the code is more concrete than a “production” Python package would be:

  • No retry framework. Each LLM call has inline retry-on-429 with exponential backoff (max 2 retries). A retry decorator would be cleaner but adds indirection without much benefit.
  • No async. The pipeline runs sequentially. The three extraction agents could run concurrently but don’t currently. Adding async would help latency by ~2x but complicates error handling.
  • No dependency injection. The Vertex client is cached as a module-level global; the prompt loader reads from a hardcoded path relative to __file__. Both are fine for a single-tenant CLI; would need DI for embedding in a multi-tenant service.
  • Minimal logging. Each subcommand prints progress to stderr; no structured logging, no metrics emission. Sufficient for the current use case; adopt a logging framework when the operational surface grows.

These are explicit tradeoffs, not oversights. See design decisions for the rationale.

Where the docs live

Section titled “Where the docs live”

Two documentation surfaces:

  • In-package docs — README.md, INTEGRATION.md are intended as quick-reference for someone with the package open. README is for developers; INTEGRATION is for eLife collaborators.
  • The site you are reading — site/src/content/docs/docs/ in MDX. Astro Starlight builds it as a sub-site at /docs/ of the existing eLife claim-trees site.

The two surfaces overlap intentionally — the README’s content is duplicated and expanded across the site’s pages. Keep them in sync when behavior changes.

Where the data lives

Section titled “Where the data lives”

Three data locations:

  • ~/.cache/elife-extract/ — fetched PDFs cached for re-runs. Created automatically; safe to delete.
  • <output-dir>/ (default: ./out) — JSON intermediates. draft-<slug>.json is the reconciled draft; agents-<slug>.json is per-agent raw output; <draft>.reviewed.json is the post-Step-4.5 revised draft.
  • <corpus-dir>/<paper-slug>/ — the claim files. index.md is the paper metadata; <claim-slug>.md files are individual claims.

Testing

Section titled “Testing”

There is no pytest test suite. The tests/ directory contains round-trip scorecards (the empirical validation; expensive to re-run, ~$100 per sweep) and the legacy standalone scorer. Unit tests are deferred — the system’s correctness is validated end-to-end via evaluate, not via per-function assertions.

If you add a function that needs unit testing (e.g., complex slug derivation logic), add a pytest fixture in tests/. The pyproject.toml already declares pytest>=7.0 as a dev dependency.

How to extend

Section titled “How to extend”

Three common extension points:

  • New CLI subcommand: add a cmd_<name>(args) handler in cli.py, declare its argparse subparser in build_parser(), set func=cmd_<name> on the subparser, implement the handler logic in a new module under elife_extract/. See cmd_evaluate for a worked example.
  • New prompt variant: see adding a prompt variant.
  • New review mode: add the mode name to the --review-mode choices in cli.py, add a branch in cmd_write that handles the new mode, implement the mode logic (likely in review.py or a new module). See the external mode for a worked example.

Implementation references

Section titled “Implementation references”
  • home/collabs/elife/claim-trees/extract/ — the source
  • pyproject.toml — package metadata
  • home/collabs/elife/claim-trees/jobs/extract-cli.md — the worklog tracking what was built when

Next steps

Section titled “Next steps”