Experiment Record · 2026-09-27
HXE-EVIDENCE-ABC
HELIX evidence selection: Milestones A–C — deterministic baseline holds, learned checkpoint does not clear serving
Hypothesis
Paul's HELIX architecture can learn retrieval over local documents well enough to beat a tuned BM25 baseline on a frozen benchmark.
Setup
Thin adapter (encode_queries / encode_blocks / score_candidates) over the audited PyTorch H/E/R blocks; 200-step CPU smoke training on a seeded 5-family dataset; frozen benchmark (20+3 queries) with arms A BM25, B deterministic selector, C HELIX hybrid.
Variables
- ◆Pooling (last-token vs mean)
- ◆Abstention threshold (cal-tuned, frozen for test)
Results
- →Milestone A: single entry point verified; full suite 48/48 tests green.
- →Smoke training (200 steps): loss 4.586 → 2.268; checkpoint sha fd3914f55b8d8070; resume bitwise identical.
- →Held-out eval: trained ≈ control, delta 0.0 on recall@k and MRR — honest negative, no generalization. Train split: recall@10 1.00 / MRR 0.87 vs control 0.16 / 0.08.
- →Frozen benchmark: B vs A INCONCLUSIVE (median reduction 0.504, CI straddles 0.50); C vs A FAIL — learned checkpoint NOT cleared for serving; production stays on the labeled deterministic baseline.
Observations
- ·Last-token pooling is tail-dominated (H-block carries ~no content to the final position); mean pooling recovers content sensitivity (16/60 unique embeddings vs 3–4).
- ·Abstention (cal-tuned threshold): F1 0.667, precision 0.5, recall 1.0 on both pool modes.
Conclusion
Milestone A is done and the pipeline is real, but the learned model did not generalize to held-out queries. The honest result is the negative one — and it is recorded as such.
Next Experiment
Run the real 56M pilot on Mac/MLX or an authorized CUDA GPU when Paul points at one; do not rent compute.