Technical Blog

Lab Notes

Writing from inside the lab — every note opens with a TL;DR and links back to the experiments it draws on.

2026-07-18

Agent Architecture

From Hermes and OpenClaw to Remy: consolidating the lab's agents

Two agents became one. Remy imported Hermes's Telegram and email identities and OpenClaw's channels, wrapped them in a deterministic policy engine with git-backed reversible edits, and added a budget-aware sub-agent hive — verified live across four providers with a $0.02 budget correctly blocking a cloud call.

2026-06-25

Lessons Learned

What 10M parameters actually buys you

SLM-10M — 9.97M params, 25B curated tokens, GQA + QK-norm — scores 32.38% average on the small-model leaderboard and can't generate open-ended text without looping. Both facts are in the model card, because both are true.

2026-06-10

Engineering

Hot-swapping Metal kernels into a live training run

We replaced the chunked MLX recurrence path with fused Metal kernels mid-flight in a live SSM+GLA run: peak memory fell 23.88GB → 10.34GB, throughput rose ~1,074 → ~1,500 tok/s, and gradient parity held at ~1e-7. The loss curve never noticed.

2026-06-07

Model Training

Going all-local: a 4.75B-token pretraining pipeline

D-CSIL-3 pretrains a 235.7M selective-state model on a seven-stage, 4.747B-token corpus — FineWeb-Edu, Wikipedia, SlimPajama, OpenWebText, WikiText-103, then SFT — with every byte staged locally. Measured throughput: 1,270 tok/s. Estimated wall-clock: 43.26 days.

2025-08-10

Research Notes

Training with plasticity phases: what 215 cycles taught us

HibbieFormer trains in explicit phases — hebbian, synaptic scaling, pruning, on a 50/30/20 schedule — with an adaptive neuromodulator. The TinyStories run completed 215 cycles and 21,476 steps without a single divergence, ending at PPL ~2.1–2.9.