TL;DR
HELIX-127M and HELIX-200M both hit a validation-loss plateau. At 127M the model was parameter-bound — more tokens made it worse. At 209M the model was schedule-bound — a real WSD decay recovered ~9 PPL with zero architecture changes. Same symptom, opposite diagnosis, both confirmed by controlled before/after runs.
The first plateau: 127M runs out of capacity
After a clean 2.5B-token foundation run, the lab tried to push HELIX-127M further with a 5B-token continuation. The first two attempts carried a silent bug — warmup_steps was set to an absolute global step rather than a per-run value, so every resumed schedule silently kept training near peak learning rate. The bug recurred the day after it caused the first failure; it was a process failure as much as a code one.
With the schedule finally corrected, the model's true behavior surfaced: validation improved to a best of 37.84 at step 161k, then degraded to 39.68 by step 187k. No 5B base was ever finished. The verdict: 127M is parameter-bound. More data could not fix it. That result is the direct reason HELIX-200M exists — a from-scratch generation at nearly double the parameters.
The second plateau: 209M looks identical, isn't
HELIX-200M's 40B-token foundation run settled into its own plateau — validation perplexity stuck in a 39.1–40.3 band for tens of thousands of steps. The obvious read, given what had just happened at 127M, was that the model had again run out of capacity.
It hadn't. The run branched into an 'anneal' phase that was supposed to decay its learning rate — and never did. Every line in its console log shows the same learning rate, 3e-4, from start to finish. A run literally named '1x-wsd-anneal' did no annealing at all.
grep -o "lr=..." console.log | sort -u
# -> lr=0.000300 (only value in the entire 3,312-line log)The recovery, and the lesson
A recovery run resumed from a protected pre-anneal checkpoint with an explicit optimizer reset and a real WSD decay schedule this time: stable phase to step 219,720, then a genuine linear decay over the final 24,421 steps. Nothing about the model changed — same architecture, same data, same parameter count.
Trainer validation PPL went from 37.32 to 28.45. Held-out validation went from 35.54 to 27.50. Facttrack accuracy rose from 10/19 to 14/19. All of it came from finally running the schedule as designed. The handoff note says it plainly: 'the plateau was the schedule, not the model.'
Put the two results side by side and you get the program's most reusable finding: identical symptoms — a stalled validation curve — had opposite causes at consecutive scales. At 127M, under a correct schedule, the plateau meant the model needed to be bigger. At 209M, under a broken schedule, the plateau meant nothing about the architecture was wrong at all. Diagnosing which one you're looking at, before reaching for the expensive fix, is the actual skill.