Experiment Record · 2026-07-17
HELIX-200M-C1X-RECOVERY
The WSD recovery anneal: 'the plateau was the schedule, not the model'
Hypothesis
The 209M foundation's validation plateau (39.1–40.3) can be broken by actually running the WSD decay schedule the run was designed for — the prior 'anneal' branch never decayed its learning rate at all.
Setup
Resumed the failed-anneal branch's step-132,000 checkpoint with an explicit AdamW optimizer reset and 1,000-step re-warmup, then ran a real WSD schedule: stable to step 219,720, linear decay over the final 24,421 steps to completion at 244,141.
Variables
- ◆Schedule only (flat LR vs. real WSD decay) — architecture and data held constant
Results
- →Training complete: 244,141/244,141 steps, 7d2h31m, 1,837,318,144 tokens this leg.
- →Trainer val PPL: 37.32 (step 219,000) → 28.45 (step 244,000) — ~9 PPL from the decay phase alone.
- →Held-out (10M-token slice): 35.54 → 27.50. Facttrack rank-1: 10/19 → 14/19. Benchmark avg 41.1%.
Observations
- ·A pre-decay checkpoint (step 219,720) was protected specifically to enable this before/after comparison.
- ·Zero architecture changes between the two measurements — the entire gain came from actually running the schedule as designed.
Conclusion
The plateau was the schedule, not the model. Paired with the opposite diagnosis at 127M (parameter-bound under a correct schedule), this is the program's most reusable methodological result: identical symptoms, opposite causes, both confirmed.
Next Experiment
Post-training on this checkpoint; the architecture tax vs. a matched transformer remains unmeasured at this scale.