Skip to content
D-CSIL

Experiment Record · 2026-07-17

HELIX-200M-C1X-RECOVERY

The WSD recovery anneal: 'the plateau was the schedule, not the model'

SUCCESSREPLICATED

Hypothesis

The 209M foundation's validation plateau (39.1–40.3) can be broken by actually running the WSD decay schedule the run was designed for — the prior 'anneal' branch never decayed its learning rate at all.

Setup

Resumed the failed-anneal branch's step-132,000 checkpoint with an explicit AdamW optimizer reset and 1,000-step re-warmup, then ran a real WSD schedule: stable to step 219,720, linear decay over the final 24,421 steps to completion at 244,141.

Variables

  • Schedule only (flat LR vs. real WSD decay) — architecture and data held constant

Results

  • Training complete: 244,141/244,141 steps, 7d2h31m, 1,837,318,144 tokens this leg.
  • Trainer val PPL: 37.32 (step 219,000) → 28.45 (step 244,000) — ~9 PPL from the decay phase alone.
  • Held-out (10M-token slice): 35.54 → 27.50. Facttrack rank-1: 10/19 → 14/19. Benchmark avg 41.1%.

Observations

  • ·A pre-decay checkpoint (step 219,720) was protected specifically to enable this before/after comparison.
  • ·Zero architecture changes between the two measurements — the entire gain came from actually running the schedule as designed.

Conclusion

The plateau was the schedule, not the model. Paired with the opposite diagnosis at 127M (parameter-bound under a correct schedule), this is the program's most reusable methodological result: identical symptoms, opposite causes, both confirmed.

Next Experiment

Post-training on this checkpoint; the architecture tax vs. a matched transformer remains unmeasured at this scale.