Batch operation

The CLI is built sequential by default — each subcommand processes one paper at a time. This is the right default for development and small-scale extraction. For larger corpora, you’ll want shell-level parallelism, resumability, and a strategy for monitoring quality across the batch. This page covers the operational mechanics.

Sequential baseline

The simplest batch is a shell loop:

for doi in $(cat papers.txt); do
  elife-extract extract --doi $doi --corpus-dir ./claims --output-dir ./out
  elife-extract write --draft ./out/draft-*.json --corpus-dir ./claims --review-mode external
done

Time: ~15 min per paper × N papers, sequential. For 10 papers: ~2.5 hours. For 100: ~25 hours.

Cost: ~$7 per paper × N. For 10 papers: ~$70. For 100: ~$700.

Shell-level parallelism

The CLI is process-safe — each invocation is independent. Run multiple papers in parallel via xargs, GNU parallel, or just &:

# Three at a time with xargs
cat papers.txt | xargs -n 1 -P 3 -I {} sh -c '
  elife-extract extract --doi {} --corpus-dir ./claims --output-dir ./out
'

Or with GNU parallel:

parallel -j 3 \
  'elife-extract extract --doi {} --corpus-dir ./claims --output-dir ./out' \
  :::: papers.txt

Time at 3-way parallelism: roughly 1/3 of sequential, so ~50 min for 10 papers, ~8 hours for 100.

Vertex AI rate limits

Anthropic’s Vertex AI quotas are per-project, per-region, per-model. Default quotas are typically:

• Sonnet 4.6: ~50 requests / minute / region
• Opus 4.6: ~5-10 requests / minute / region

Each paper’s pipeline makes 4 LLM calls (3 Sonnet + 1 Opus reconcile, plus 1 Opus reviewer in external mode). At 3-way parallelism with the external reviewer:

• 9 Sonnet requests/min concurrent (well under quota)
• 6 Opus requests/min concurrent (at the edge of quota — may throttle)

If you hit rate limits, the CLI retries with exponential backoff (max 2 retries before raising). For sustained throughput at >3-way parallelism, request a quota increase from GCP.

Resumable sweeps with evaluate --skip-existing

For validation runs across the curated reference corpus, evaluate is the right tool. It walks per-paper, persists per-paper scorecards, and supports --skip-existing for resumability:

elife-extract evaluate \
  --reference-dir ~/Projects/mainenlab/elife-claim-trees/claims \
  --work-dir /tmp/eval-$(date +%Y%m%d-%H%M) \
  --all \
  --review-mode external \
  --skip-existing

If the sweep is interrupted (network drops, OOM, kill), the per-paper scorecard.json files persist. Re-running with --skip-existing skips papers that already have a scorecard and continues with the rest. Failed papers (those whose scorecard contains an error: field) are skipped too — you’d need to delete those scorecards and re-run to retry them.

The aggregate aggregate-scorecard.md file is regenerated on each run from whatever scorecards exist in <work-dir>/. Mid-sweep aggregates are sensible reads of partial progress.

Resuming a partial extract sweep manually

For non-evaluate batches (e.g., extracting a fresh corpus, not validating against a reference), there’s no built-in resumability. Three patterns work:

Pattern A — Check before run. Inspect the --corpus-dir for already-populated paper directories before invoking:

for doi in $(cat papers.txt); do
  slug=$(/Users/zach/anaconda/bin/python -c "import re; print(re.sub(r'.*\.', '', '$doi'))")
  if [ -d "./claims/${slug}-*" ]; then
    echo "skip $doi (already extracted)"
    continue
  fi
  elife-extract run --doi $doi --corpus-dir ./claims
done

Pattern B — Two-phase batch. Run all extract calls first, then all write calls. The extract phase only writes JSON intermediates, so re-running it on already-extracted papers re-fetches the PDF (cached) but doesn’t damage the corpus. The write phase refuses to overwrite existing paper directories (raises FileExistsError), so re-running it skips done papers naturally:

for doi in $(cat papers.txt); do
  elife-extract extract --doi $doi --output-dir ./out --corpus-dir ./claims
done

for draft in ./out/draft-*.json; do
  elife-extract write --draft $draft --corpus-dir ./claims --review-mode external 2>/dev/null || echo "skip (already written)"
done

Pattern C — Use evaluate even for non-validation batches. If you point --reference-dir at the corpus you’re building (rather than a curated reference), evaluate --skip-existing gives you free resumability. The “scoring” output is meaningless without a reference to compare against, but the extract+review+write portion still runs correctly.

Cost projections at scale

Reference numbers from the 10-paper sweep on the curated corpus (mean values):

Corpus size	Cost (external mode)	Time sequential	Time at 3-way parallel
10 papers	$70	~2.5 hrs	~50 min
50 papers	$350	~12 hrs	~4 hrs
100 papers	$700	~25 hrs	~8 hrs
500 papers	$3.5K	~5 days	~40 hrs
1000 papers	$7K	~10 days	~3 days
3000 papers (panel-claim-unification.md horizon)	$21K	~30 days	~10 days

These are with the external reviewer enabled. Drop $2/paper if running auto-approve mode without the reviewer.

Observability

Each subcommand logs its progress to stderr. For batch operation, capture both stdout and stderr per-paper for later inspection:

mkdir -p logs
for doi in $(cat papers.txt); do
  slug=$(echo $doi | sed 's|.*/||')
  elife-extract run --doi $doi --corpus-dir ./claims \
    > logs/${slug}.stdout 2> logs/${slug}.stderr
done

Look for:

• extract failed: errors in stderr — usually network or auth issues
• external review failed: errors — schema validation issues or rate limits
• Per-paper claim count totals at the end of extract — sudden drops can indicate prompt regression

Failure modes you’ll see at scale

Symptom	Cause	Fix
404 Publisher Model not found	Vertex model not enabled	Enable in GCP, or --vertex-region to a region where it is
429 Resource exhausted	Rate limit	Reduce parallelism, or request a quota increase
[Errno 8] nodename nor servname	DNS / network failure (transient)	Wait, retry. Often resolves within minutes
Streaming required for operations longer than 10 minutes	max_tokens too high without streaming	Already handled in the CLI; report as a bug if you see it
JSON parse failed after extract	Agent emitted non-JSON output (rare)	Re-run; usually transient. Persistent failures indicate prompt issue

Implementation references

• elife_extract/cli.py — subcommand handlers
• elife_extract/evaluate.py — evaluate_paper() and aggregate_report() for batch validation

Next steps

• Cost and performance — per-step breakdown for tuning
• Validation results — what to expect from a 10-paper sweep
• Iteration discipline — how to validate prompt changes before deploying