TL;DR
SLM-10M — 9.97M params, 25B curated tokens, GQA + QK-norm — scores 32.38% average on the small-model leaderboard and can't generate open-ended text without looping. Both facts are in the model card, because both are true.
The recipe
12 layers at 256 dim with grouped-query attention (8 query heads, 2 KV), QK-norm to prevent logit explosion, RoPE at θ=100k, SwiGLU, weight tying, and an 8,192-token vocabulary. Data: 55% FineWeb-Edu, 25% Cosmopedia-v2, 10% FineWeb-HQ, 10% FineMath — 25B tokens at 512K tokens/step on a borrowed NVIDIA GB10 — the lab's only non-local training run to date.
Evaluating honestly at tiny scale
At 10M parameters, open-ended generation degenerates into repetition loops — so the eval uses log-likelihood ranking over multiple-choice options, and the published card says exactly that. PIQA at 50.92% and ArithMark at 24.32% are modest numbers reported plainly; the contribution is a clean, reproducible recipe at a scale anyone can train.