Project File
HELIX
A family of hybrid recurrent language models — HarmonicStateLayer, ElasticAttentionGate, and ResonantExpertRouting blocks — trained entirely on Apple Silicon.
Overview
OBSERVEDHELIX is the lab's flagship model family. Each model stacks a repeating H-E-R pattern — HarmonicStateLayer (recurrent state), ElasticAttentionGate (sparse attention), ResonantExpertRouting (top-2 of 4 mixture-of-experts) — over a GPT-2 BPE vocabulary of 50,257 tokens.
Current Run: HELIX-200M Foundation
MEASUREDThe helix-200m-foundation-35b run is training a 209M-parameter, 28-layer model on a 40B-token corpus (90% FineWeb-Edu, 10% WikiText-103, per-doc repeat cap 4, 13-gram decontamination). At step 120,140: training PPL 16.85, validation PPL 39.4, ~2,900 tokens/sec sustained on the M3 Max in BF16 at ~12GB peak memory.
Scaling Discipline
MEASUREDToken budgets are Chinchilla-matched (~20 tokens/parameter). HELIX-56M established the baseline (val PPL 48.6; training throughput in the 8,000–9,000 tok/s range, operator-reported), with matched transformer baselines run in Google Colab (HELIX_Transformer_Comparison notebooks, June 2026). HELIX-120M is configured as a capacity-control variant under a strict 20GB MLX memory cap, and a 350M preset is defined but gated on the 200M run's results.
Model Lineage
Current Limitations
- −HELIX-200M is mid-training; all its numbers are checkpoint snapshots, not final results.
- −Published external baselines (transformer 33.0 PPL @44M, Mamba-2 ~28.5 @130M on WT-103) have not yet been matched under identical data conditions.
Next Steps
- +Complete the 200M foundation run and publish final validation numbers.
- +Run HELIX-120M as the capacity-control comparison.
- +Decide the 350M step from measured scaling behavior, not enthusiasm.