Skip to content

Adding a prompt variant

The --prompt-variant flag lets you swap in alternative prompts for any agent without modifying the defaults. Variants are first-class — version-controlled, comparable via evaluate, swappable at run time. This is how you A/B test prompt revisions, calibrate prompts for non-default paper types, or experiment with different reviewer-prompt designs.

When to author a variant

Section titled “When to author a variant”

Three common cases:

  1. Testing a prompt revision before it becomes the default. A new bias rule for the reviewer; a more aggressive multi-panel preservation instruction for the Caption-reader; a domain-specific extraction guidance for the Results-reader.
  2. Calibrating prompts for a non-default paper type. Methods papers, observational atlases, computational papers, papers from non-eLife sources — these may need prompts tuned to their argument structure.
  3. Maintaining a stable production variant. When you want the default to be the bleeding edge but production runs to use a known-stable older variant, freeze the production version under prompts/production-vN/ and run with --prompt-variant production-vN.

The directory convention

Section titled “The directory convention”
extract/prompts/
├── results-reader.md            # default
├── caption-reader.md            # default
├── structure-reader.md          # default
├── reconciler.md                # default
├── external-reviewer.md         # default
└── <variant-name>/
    ├── results-reader.md        # variant override (only if changed)
    ├── caption-reader.md        # variant override (only if changed)
    ├── ...                      # any subset of the prompts

A variant is a directory under prompts/. Inside it, you place override versions of any subset of the five prompts. Prompts not present in the variant directory fall back to the defaults. This means you can author a variant that changes only one prompt, without copy-pasting the others.

The CLI’s Config.prompt_path() resolves variant lookups:

def prompt_path(self, agent: str) -> Path:
    """Resolve the prompt file for a given agent in the active variant."""
    if self.prompt_variant == DEFAULT_PROMPT_VARIANT:
        return self.prompts_dir / f"{agent}.md"
    return self.prompts_dir / self.prompt_variant / f"{agent}.md"

(Currently this falls back to default only at the top level — full subset-fallback would need a small change to check both paths and prefer variant.)

Authoring workflow

Section titled “Authoring workflow”

  1. Identify what you want to change. Be specific: “I want the Caption-reader to preserve panel: fig4, fig5 as a multi-panel claim when the caption text references multiple panels.” The change is concrete.

  2. Create the variant directory.

    Terminal window
    mkdir -p prompts/multi-panel-fix
    cp prompts/caption-reader.md prompts/multi-panel-fix/caption-reader.md

  3. Edit the variant prompt. Add the new instruction. Keep the rest of the prompt untouched (or note what else you’re changing).

  4. Run on a single paper to inspect. Pick a paper where the change should make a difference. For multi-panel preservation, pick wengert (lowest panel score in the 10-paper sweep at 36%):

    Terminal window
    elife-extract extract \
      --doi 10.7554/eLife.<wengert-doi> \
      --corpus-dir /tmp/variant-test \
      --output-dir /tmp/variant-test/out \
      --prompt-variant multi-panel-fix

    Inspect the draft. Did the Caption-reader output multi-panel claims for the relevant content? If yes, proceed. If no, the prompt change wasn’t aggressive enough; iterate.

  5. Validate via evaluate. Compare the variant against the baseline:

    /tmp/elife-eval-baseline/aggregate-scorecard.md
    # Baseline (already exists from prior sweep)
    # Variant
    elife-extract evaluate \
      --reference-dir ~/Projects/mainenlab/elife-claim-trees/claims \
      --work-dir /tmp/elife-eval-variant \
      --all \
      --review-mode external \
      --prompt-variant multi-panel-fix

    Diff the aggregate scorecards. Did panel agreement go up? Did anything else move (recovery, role)?

  6. Decide. If the variant is a net win across the corpus, consider promoting it to default. If it helps the targeted papers but hurts others, keep it as a variant for those papers specifically. If it shows no improvement, revert and try a different approach.

  7. Document. Inscribe the variant’s purpose, what it changed, and the measured delta in home/collabs/elife/claim-trees/jobs/extract-cli.md worklog. The empirical record is part of the discipline.

Cost of variant testing

Section titled “Cost of variant testing”
OperationCost
Author the variant prompt$0
Single-paper test run~$5 (extract only)
Single-paper round-trip validation~$10 (extract + matcher)
Full 10-paper sweep validation~$100

Targeted iteration on a single paper before committing to the full sweep is the right pattern. The kammer iteration (iteration discipline) is the worked example: ~$10 to validate a prompt change on the target paper, ~$100 to confirm the change is a net win across the corpus.

Promotion to default

Section titled “Promotion to default”

If a variant proves to be a net win, promote it by:

  1. Copying the variant prompt over the default: cp prompts/multi-panel-fix/caption-reader.md prompts/caption-reader.md
  2. Re-running evaluate --all with no --prompt-variant flag to confirm the new default is what you intended
  3. Inscribing the promotion in the worklog with the measured delta and the rationale
  4. Optionally: keeping the variant directory as a frozen reference (e.g., rename to pre-multi-panel-fix/) so the previous behavior is recoverable

Testing a model substitution alongside a prompt variant

Section titled “Testing a model substitution alongside a prompt variant”

Variants compose with model overrides. To test “a Haiku-tier Caption-reader with a multi-panel-fix prompt”:

Terminal window
elife-extract evaluate \
  --reference-dir ~/Projects/mainenlab/elife-claim-trees/claims \
  --work-dir /tmp/elife-eval-haiku-multi-panel \
  --all \
  --review-mode external \
  --prompt-variant multi-panel-fix \
  --model-caption claude-haiku-4-5

This composes orthogonally — the variant changes the prompt, the model flag changes which model loads it. Both feed into the same evaluate harness for measurement.

Pitfalls

Section titled “Pitfalls”
  • Variants need their reciprocal pages written. If you add a new prompt file under prompts/<variant>/ for a role that didn’t have a default (e.g., a new agent type), you also need to wire the agent into agents.py. The variant directory only overrides existing prompts; new agent roles need code changes.
  • The reconciler and reviewer prompts are interdependent with the schema. A variant reviewer that emits new claim_type values not in the ClaimType Literal will fail Pydantic validation. Either extend the schema or stay within the existing vocabulary.
  • Validation cost adds up. Each full sweep is $100. Don’t run a sweep for every prompt-character change; iterate on a single paper first, run the sweep when the targeted change clearly works.
  • Don’t rely on stochastic noise as signal. A 1-2 percentage point change between runs is within LLM stochasticity. Look for ≥5 point changes as meaningful; ideally validate by running each configuration twice.

Implementation references

Section titled “Implementation references”
  • extract/prompts/ — default prompts
  • elife_extract/config.py Config.prompt_path() — variant resolution logic
  • elife_extract/agents.py load_prompt() — runtime prompt loading
  • elife_extract/external_review.py load_reviewer_prompt() — variant-aware reviewer loading

Next steps

Section titled “Next steps”