Project File
D-CSIL Foundation Models
The D-CSIL-1/2/3 training line plus the published SLM-10M: selective state-space and GLA models pretrained on all-local pipelines.
SLM-10M — Published
MEASUREDA 9.97M-parameter causal transformer (12 layers, GQA 8Q/2KV, QK-norm, RoPE θ=100k, SwiGLU, 8,192 vocab) trained on 25B tokens (FineWeb-Edu 55%, Cosmopedia-v2 25%, FineWeb-HQ 10%, FineMath 10%) on a borrowed NVIDIA GB10 (a neighbor's machine — the lab's one non-local run). Published to HuggingFace as liodon-ai/slm-10m with a full zero-shot scorecard: 32.38% average across HellaSwag, ARC, PIQA, and ArithMark.
The D-CSIL Line
MEASUREDD-CSIL-1 (55.9M, GLA 4-head) carried the lab's first DPO/LoRA experiments, with a brutally honest epoch-8 inference report (val PPL 455.8, exact-match 37.5%). D-CSIL-2 built the benchmark harness — its model reached PPL 58.0 by step 82.5k with dialogue accuracy climbing from 40% to 80% across checkpoints. D-CSIL-3 scales to 235.7M parameters over a 4.747B-token, seven-stage all-local corpus at a measured 1,270 tok/s.
Current Limitations
- −SLM-10M shows repetition loops in open-ended generation — evaluated by log-likelihood ranking, as appropriate at 10M scale.
- −D-CSIL-1's post-training results were weak (PPL 455.8); recorded as a lesson, not buried.
Next Steps
- +Finish the D-CSIL-3 pretrain+SFT schedule (est. 43 days at measured throughput).
- +Port the D-CSIL-2 benchmark harness to evaluate HELIX checkpoints.