Experiments & benchmarks

Why this exists

Before this, every comparison was a script someone wrote once. Twenty of them accumulated in one repository. Each re-invented how to run the arms, how to poll for results, how to price them, and how to draw a report. The results lived in untracked files with four different names. The findings ended up as dated comments beside the constants they changed, which is fine until you want to ask “is that still true?” — and one published report was deleted and had to be rebuilt from scratch.

Then the cost of that showed up. A production quality gate was set from a bench of nineteen subjects whose scores did not look like real work. For two days, packs lost between nine and thirty-nine items each, and the code comment beside the number said the number was calibrated. It was — against the wrong thing.

The six nouns

Learn these and the rest of the page is detail. Nothing here is a new kind of execution — an experiment borrows the machinery that already runs your work.

The rules, and why each one is there

Running one, start to finish

1. Write the design

   A JSON file, committed next to the code. It is the create body plus, optionally, an inline case set. Keeping it in the repository is what makes “run last month’s benchmark again” a command rather than an archaeology project.

2. Dry-run it

   Prices every arm × case × repeat with the same estimator a single run uses, and executes nothing. If the estimate exceeds the budget cap, freezing tells you before you spend.

3. Freeze it

   Pins versions, validates every case against every arm’s intake schema and every model override, computes the design hash. A bad case fails here, not after forty renders.

4. Run it

   Trials are planned interleaved, executed under the experiment’s own budget ceiling and the workspace budgets, and settled from each run’s record.

5. Read the report

   Pass rate at the gate, median, p10, worst, the per-case delta against the baseline with an interval, cost, latency, and what each arm lost. Drag the gate to see what a different threshold would have done.

6. Record the decision

   Ship, hold or revert, with the rationale and a link to the report. The next canary reads this to know what the current baseline is.

{
  "name": "Difference matte: which model repaints its own render",
  "kind": "benchmark",
  "hypothesis": "2.5 Sunburst repainting its own render holds the subject still at least as well as the split pairing, at the same cost per item.",

  "primaryMetric":    { "scorer": "diffmatte_pair", "key": "pairConsistency", "direction": "higher", "threshold": 0.95 },
  "guardrailMetrics": [{ "scorer": "run_metrics", "key": "cost_usd", "direction": "lower", "maxDegradation": 0.005 }],
  "analysisPlan":     { "paired": true, "bootstrapSamples": 2000, "correction": "holm", "alpha": 0.05 },

  "repeats": 1,
  "budgetLimitUsd": 3.0,
  "concurrency": 4,

  "caseSet": {
    "name": "diffmatte-pair-prompts",
    "casesFile": "experiments/cases/diffmatte-pair-prompts-2026-09-10.json"
  },

  "arms": [
    { "key": "A", "label": "Sunburst render, gpt-image-2 repaint", "isBaseline": true,
      "workflowId": "generate-clip-art-diffmatte",
      "intakeOverrides":   { "style": "flat", "quality": "low" },
      "providerOverrides": { "imageEditor": "openai/gpt-image-2-2026-04-21" } },

    { "key": "B", "label": "Sunburst render, Sunburst repaint",
      "workflowId": "generate-clip-art-diffmatte",
      "intakeOverrides":   { "style": "flat", "quality": "low" },
      "providerOverrides": { "imageEditor": "openai/gpt-image-2.5-sunburst-2026-09-08" } }
  ]
}

# what it would cost, and nothing else
python -m scripts.experiment run experiments/designs/diffmatte-model-pair.json --dry-run

# run it and wait, then write the report next to you
python -m scripts.experiment run experiments/designs/diffmatte-model-pair.json --wait --out report.html

python -m scripts.experiment status exp-1a2b3c4d
python -m scripts.experiment report exp-1a2b3c4d --out report.html

Designing one that answers something

Pick one primary metric

One number decides, and you name it before you look. Everything else is a guardrail: reported, watched, and allowed to be the reason you say no, but never the reason you say yes. Guardrails carry a maxDegradation — how much worse than the baseline is still acceptable.

Choose the baseline deliberately

One arm is the baseline, and it should be what runs in production today. Every other arm is reported as a difference from it, computed per case: the same prompt, the same seed policy, arm B minus arm A. Per-case differences cancel out how hard each case is, which is most of the variation in a generation benchmark.

Cases, and the hold-out

A case set is versioned and private. Seed one from production — the runs that failed last night are the best benchmark you will ever have — and keep a hold-out slice that never feeds prompt tuning. Hold-out cases are excluded unless the plan asks for them, and a report says when they were included. Once a decided experiment has used a version, its hold-out flags cannot be flipped in place; that would make the split meaningless.

{
  "caseId": "s02",
  "intake": { "prompt": "A noble fir Christmas tree in three-quarter view, …" },
  "tags":   { "category": "christmas", "elementType": "subject" },
  "holdout": false,
  "source": { "runId": "run-7717b86b" }
}

Repeats, not seeds

The image providers do not expose a seed, so repeats are the seed policy: the same case run N times gives you the spread. Cost is linear in repeats, so start at one for a screening comparison and raise it when the difference you care about is close to the noise.

Multiple arms

Every extra arm is another chance for one to look good by luck, so the analysis applies a Holm correction across the arms on the primary metric, and marks a delta as significant only if it survives. Guardrails are reported without correction — they exist to be looked at, not to be tested.

Reading the report

{
  "primary": { "metric": "diffmatte_pair.pairConsistency", "direction": "higher", "threshold": 0.95 },
  "baselineArmKey": "A",
  "perArm": {
    "A": { "passRate": 0.50, "median": 0.951, "p10": 0.922, "min": 0.919, "scored": 26, "costUsd": 0.295 },
    "B": { "passRate": 0.50, "median": 0.958, "p10": 0.894, "min": 0.703, "scored": 26, "costUsd": 0.298 }
  },
  "deltas": {
    "B": { "mean": 0.004, "low": -0.010, "high": 0.019, "p": 0.62,
           "pairedCases": 26, "significant": false }
  },
  "thresholdSweep": { "grid": [0.80, 0.82, …], "perArm": { "A": [1.0, 1.0, …], "B": […] } },
  "failures": { "perArm": { "A": { "byCode": {}, "rate": 0.0 } }, "srm": null },
  "verdictHints": ["arm B vs A on diffmatte_pair.pairConsistency: +0.0040 …"]
}

The scorers

A scorer is a named, versioned function that turns a settled trial into numbers. It never decides anything — thresholds belong to the design, verdicts to the analysis — so the same scorer serves an experiment that wants a hard gate and one that wants a distribution.

Adding one is a function plus a registry entry. Changing what an existing one returns for the same input is a version bump — that is the whole contract, and it is what lets you re-score a year of history and compare it honestly.

When not to use an experiment

Anti-drift rules

API

What exists today