The same coding tasks, run headless in Claude Code on Opus, Sonnet and Haiku. Each task checks itself, so a pass is a pass. Headline numbers are plain Claude Code with nothing extra loaded: 132 runs across 3 task sets, plus 144 more comparing configs.
01All tasks
Tier
Passed
Pass rate
Cost / task
Cost per pass vs Opus
Avg time
Opus
44/44
100%
$0.082
baseline
18 s
Sonnet
44/44
100%
$0.038
47%
11 s
Haiku
41/44
93%
$0.055
72%
38 s
Parity is the tier's pass rate divided by Opus's on the same tasks. Cost is cost per passed task, at API prices, relative to Opus.
Sonnet passed all 44 tasks at 47% of Opus's cost per task.
Haiku cost as much as or more than Sonnet per successful task on every set: it takes enough extra turns to cancel its lower token price.
On coding-v3, Haiku passed 9 of 12 and cost more per success than Opus ($0.132 vs $0.123).
Running the same tasks inside the author's everyday setup (19 MCP tools, 63 skills) made Opus 2.1× and Sonnet 2.4× as expensive, with identical results.
20 tasks · no failures on any tier · manifest e9527dd003a0 · last run 2026-10-03
Every task, every tier
Task
Opus
Sonnet
Haiku
feature/01
✓
✓
✓
feature/02
✓
✓
✓
feature/03
✓
✓
✓
feature/04
✓
✓
✓
feature/05
✓
✓
✓
fix-test/01
✓
✓
✓
fix-test/02
✓
✓
✓
fix-test/03
✓
✓
✓
fix-test/04
✓
✓
✓
fix-test/05
✓
✓
✓
refactor/01
✓
✓
✓
refactor/02
✓
✓
✓
refactor/03
✓
✓
✓
refactor/04
✓
✓
✓
refactor/05
✓
✓
✓
small-edit/01
✓
✓
✓
small-edit/02
✓
✓
✓
small-edit/03
✓
✓
✓
small-edit/04
✓
✓
✓
small-edit/05
✓
✓
✓
03coding-v2
coding-v2
Twelve harder tasks: an LRU recency bug, RFC 4180 CSV quoting, a concurrency limiter that leaks slots, prototype-pollution hardening, a leading-edge debounce, cursor pagination, an expression parser, semver ranges, topological sort with cycle detection, and multi-file and nesting refactors.
Task type
Opus
Sonnet · parity · cost
Haiku · parity · cost
fix-test
3/3
3/3 · 1.00 · 49%
3/3 · 1.00 · 51%
small-edit
3/3
3/3 · 1.00 · 50%
3/3 · 1.00 · 63%
feature
3/3
3/3 · 1.00 · 48%
3/3 · 1.00 · 84%
refactor
3/3
3/3 · 1.00 · 56%
3/3 · 1.00 · 52%
12 tasks · no failures on any tier · manifest a9190d302477 · last run 2026-10-03
Every task, every tier
Task
Opus
Sonnet
Haiku
feature/v2-01
✓
✓
✓
feature/v2-02
✓
✓
✓
feature/v2-03
✓
✓
✓
fix-test/v2-01
✓
✓
✓
fix-test/v2-02
✓
✓
✓
fix-test/v2-03
✓
✓
✓
refactor/v2-01
✓
✓
✓
refactor/v2-02
✓
✓
✓
refactor/v2-03
✓
✓
✓
small-edit/v2-01
✓
✓
✓
small-edit/v2-02
✓
✓
✓
small-edit/v2-03
✓
✓
✓
04coding-v3
coding-v3
Twelve tasks with hidden tests, written into the folder only after the session ends: tax rounded once not per line, business days with holidays, async memoization that must not cache failures, globstar matching, a quadratic dedupe made fast without changing results, strict INI errors with line numbers, cron with the day-of-month OR day-of-week rule, a provably minimal diff, a hand-written JSON parser with exact error positions, breaking an import cycle, a dual callback/promise API, and a long function split up and checked against the original on 3,000 random inputs.
Task type
Opus
Sonnet · parity · cost
Haiku · parity · cost
fix-test
3/3
3/3 · 1.00 · 44%
3/3 · 1.00 · 61%
small-edit
3/3
3/3 · 1.00 · 48%
1/3 · 0.33 · 154%
feature
3/3
3/3 · 1.00 · 33%
2/3 · 0.67 · 171%
refactor
3/3
3/3 · 1.00 · 47%
3/3 · 1.00 · 59%
12 tasks · 3 tasks failed on at least one tier · manifest 3ffc6a672c57 · last run 2026-10-03
What failed
Haiku · small-edit/v3-01 — an escaped * produced an invalid regular expression; **/ did not match zero directories
Haiku · small-edit/v3-02 — counted NaN as a duplicate (Set semantics, not ===)
Haiku · feature/v3-03 — lost an own __proto__ key and reported wrong error positions (hit the 6-minute cap)
Every task, every tier
Task
Opus
Sonnet
Haiku
feature/v3-01
✓
✓
✓
feature/v3-02
✓
✓
✓
feature/v3-03
✓
✓
✗
fix-test/v3-01
✓
✓
✓
fix-test/v3-02
✓
✓
✓
fix-test/v3-03
✓
✓
✓
refactor/v3-01
✓
✓
✓
refactor/v3-02
✓
✓
✓
refactor/v3-03
✓
✓
✓
small-edit/v3-01
✓
✓
✗
small-edit/v3-02
✓
✓
✗
small-edit/v3-03
✓
✓
✓
05Configs
Does your setup help?
The same tasks again, changing only what Claude Code loads. Each row compares a config with clean, on exactly the tasks both ran.
clean — Plain Claude Code: no user settings, hooks, plugins, MCP servers or skills.
ambient — The author's everyday setup: 19 MCP tools (Vercel, Google Drive, Calendar, Gmail and others), 63 skills and 102 slash commands loaded in every session.
clean+verify-md — Clean, plus a five-line CLAUDE.md: read the rules first, never edit tests, enumerate the edge cases the spec implies and test each one, re-run the tests before stopping.
Config
Tier
Tasks
Passed: clean → config
Cost / task
Tokens / task
ambient
Opus
44
44 → 44
$0.082 → $0.172 (+111%)
73K → 103K (+41%)
ambient
Sonnet
44
44 → 44
$0.038 → $0.091 (+140%)
58K → 103K (+76%)
ambient
Haiku
44
41 → 40
$0.055 → $0.062 (+13%)
146K → 164K (+12%)
clean+verify-md
Haiku
12
9 → 10
$0.099 → $0.109 (+10%)
217K → 296K (+36%)
06Method
How it's measured
Each task is a small Node project written to a fresh temp folder: the files, a prompt, and a verify command.
Claude Code runs it headless with claude -p, --model opus|sonnet|haiku, and only the tools the task allows. One attempt, no retries, no hints.
The clean config isolates the session with --setting-sources project --strict-mcp-config --disable-slash-commands: no user settings or hooks, no plugins enabled by the user, no MCP servers, no skills. (--bare would go further but only accepts an API key, not a subscription login.) The first runs were made without this isolation; they are kept as the ambient config and compared under Configs.
The verify command runs after the session ends. Exit code 0 is a pass. Tests the model was told not to edit are part of the fixture.
From coding-v3 on, each task also has hidden tests that are written into the folder only after the session ends, so a fix that only satisfies the visible test fails. One refactor is checked against a frozen copy of the original code on 3,000 random inputs.
The grader belongs to the benchmark, not the model: visible tests and check scripts are restored from the original task before verify. An audit of every transcript found seven sessions that had rewritten a test file; re-graded against the originals, all five passes still pass, and corrections are listed under each set.
Every task is checked offline before any run: it must fail on the starting files and pass on a reference solution, so doing nothing never passes.
Cost is the API-equivalent price of the tokens in each session's transcript, deduplicated and priced exactly the way the Claude Digest HUD prices your own sessions. The manifest hash pins the exact task set each table came from.