Claude DigestTier benchmark ← Back
Harness-based development · 2026-10-03

Does the cheaper tier hold?

The same coding tasks, run headless in Claude Code on Opus, Sonnet and Haiku. Each task checks itself, so a pass is a pass. Headline numbers are plain Claude Code with nothing extra loaded: 132 runs across 3 task sets, plus 144 more comparing configs.

01All tasks
TierPassedPass rateCost / taskCost per pass vs OpusAvg time
Opus44/44100%$0.082baseline18 s
Sonnet44/44100%$0.03847%11 s
Haiku41/4493%$0.05572%38 s

Parity is the tier's pass rate divided by Opus's on the same tasks. Cost is cost per passed task, at API prices, relative to Opus.

02coding-v1

coding-v1

Twenty small, well-specified tasks: off-by-one fixes, one-line edits, single-function features, simple extractions.

Task typeOpusSonnet · parity · costHaiku · parity · cost
fix-test5/55/5 · 1.00 · 53%5/5 · 1.00 · 47%
small-edit5/55/5 · 1.00 · 53%5/5 · 1.00 · 53%
feature5/55/5 · 1.00 · 50%5/5 · 1.00 · 55%
refactor5/55/5 · 1.00 · 50%5/5 · 1.00 · 54%

20 tasks · no failures on any tier · manifest e9527dd003a0 · last run 2026-10-03

Every task, every tier
TaskOpusSonnetHaiku
feature/01✓✓✓
feature/02✓✓✓
feature/03✓✓✓
feature/04✓✓✓
feature/05✓✓✓
fix-test/01✓✓✓
fix-test/02✓✓✓
fix-test/03✓✓✓
fix-test/04✓✓✓
fix-test/05✓✓✓
refactor/01✓✓✓
refactor/02✓✓✓
refactor/03✓✓✓
refactor/04✓✓✓
refactor/05✓✓✓
small-edit/01✓✓✓
small-edit/02✓✓✓
small-edit/03✓✓✓
small-edit/04✓✓✓
small-edit/05✓✓✓
03coding-v2

coding-v2

Twelve harder tasks: an LRU recency bug, RFC 4180 CSV quoting, a concurrency limiter that leaks slots, prototype-pollution hardening, a leading-edge debounce, cursor pagination, an expression parser, semver ranges, topological sort with cycle detection, and multi-file and nesting refactors.

Task typeOpusSonnet · parity · costHaiku · parity · cost
fix-test3/33/3 · 1.00 · 49%3/3 · 1.00 · 51%
small-edit3/33/3 · 1.00 · 50%3/3 · 1.00 · 63%
feature3/33/3 · 1.00 · 48%3/3 · 1.00 · 84%
refactor3/33/3 · 1.00 · 56%3/3 · 1.00 · 52%

12 tasks · no failures on any tier · manifest a9190d302477 · last run 2026-10-03

Every task, every tier
TaskOpusSonnetHaiku
feature/v2-01✓✓✓
feature/v2-02✓✓✓
feature/v2-03✓✓✓
fix-test/v2-01✓✓✓
fix-test/v2-02✓✓✓
fix-test/v2-03✓✓✓
refactor/v2-01✓✓✓
refactor/v2-02✓✓✓
refactor/v2-03✓✓✓
small-edit/v2-01✓✓✓
small-edit/v2-02✓✓✓
small-edit/v2-03✓✓✓
04coding-v3

coding-v3

Twelve tasks with hidden tests, written into the folder only after the session ends: tax rounded once not per line, business days with holidays, async memoization that must not cache failures, globstar matching, a quadratic dedupe made fast without changing results, strict INI errors with line numbers, cron with the day-of-month OR day-of-week rule, a provably minimal diff, a hand-written JSON parser with exact error positions, breaking an import cycle, a dual callback/promise API, and a long function split up and checked against the original on 3,000 random inputs.

Task typeOpusSonnet · parity · costHaiku · parity · cost
fix-test3/33/3 · 1.00 · 44%3/3 · 1.00 · 61%
small-edit3/33/3 · 1.00 · 48%1/3 · 0.33 · 154%
feature3/33/3 · 1.00 · 33%2/3 · 0.67 · 171%
refactor3/33/3 · 1.00 · 47%3/3 · 1.00 · 59%

12 tasks · 3 tasks failed on at least one tier · manifest 3ffc6a672c57 · last run 2026-10-03

What failed

Every task, every tier
TaskOpusSonnetHaiku
feature/v3-01✓✓✓
feature/v3-02✓✓✓
feature/v3-03✓✓✗
fix-test/v3-01✓✓✓
fix-test/v3-02✓✓✓
fix-test/v3-03✓✓✓
refactor/v3-01✓✓✓
refactor/v3-02✓✓✓
refactor/v3-03✓✓✓
small-edit/v3-01✓✓✗
small-edit/v3-02✓✓✗
small-edit/v3-03✓✓✓
05Configs

Does your setup help?

The same tasks again, changing only what Claude Code loads. Each row compares a config with clean, on exactly the tasks both ran.

ConfigTierTasksPassed: clean → configCost / taskTokens / task
ambientOpus4444 → 44$0.082 → $0.172 (+111%)73K → 103K (+41%)
ambientSonnet4444 → 44$0.038 → $0.091 (+140%)58K → 103K (+76%)
ambientHaiku4441 → 40$0.055 → $0.062 (+13%)146K → 164K (+12%)
clean+verify-mdHaiku129 → 10$0.099 → $0.109 (+10%)217K → 296K (+36%)
06Method

How it's measured

  1. Each task is a small Node project written to a fresh temp folder: the files, a prompt, and a verify command.
  2. Claude Code runs it headless with claude -p, --model opus|sonnet|haiku, and only the tools the task allows. One attempt, no retries, no hints.
  3. The clean config isolates the session with --setting-sources project --strict-mcp-config --disable-slash-commands: no user settings or hooks, no plugins enabled by the user, no MCP servers, no skills. (--bare would go further but only accepts an API key, not a subscription login.) The first runs were made without this isolation; they are kept as the ambient config and compared under Configs.
  4. The verify command runs after the session ends. Exit code 0 is a pass. Tests the model was told not to edit are part of the fixture.
  5. From coding-v3 on, each task also has hidden tests that are written into the folder only after the session ends, so a fix that only satisfies the visible test fails. One refactor is checked against a frozen copy of the original code on 3,000 random inputs.
  6. The grader belongs to the benchmark, not the model: visible tests and check scripts are restored from the original task before verify. An audit of every transcript found seven sessions that had rewritten a test file; re-graded against the originals, all five passes still pass, and corrections are listed under each set.
  7. Every task is checked offline before any run: it must fail on the starting files and pass on a reference solution, so doing nothing never passes.
  8. Cost is the API-equivalent price of the tokens in each session's transcript, deduplicated and priced exactly the way the Claude Digest HUD prices your own sessions. The manifest hash pins the exact task set each table came from.