Skip to content
D-CSIL

Experiment Record · 2026-09-27

HXE-EVIDENCE-ABC

HELIX evidence selection: Milestones A–C — deterministic baseline holds, learned checkpoint does not clear serving

PARTIALMEASURED

Hypothesis

Paul's HELIX architecture can learn retrieval over local documents well enough to beat a tuned BM25 baseline on a frozen benchmark.

Setup

Thin adapter (encode_queries / encode_blocks / score_candidates) over the audited PyTorch H/E/R blocks; 200-step CPU smoke training on a seeded 5-family dataset; frozen benchmark (20+3 queries) with arms A BM25, B deterministic selector, C HELIX hybrid.

Variables

  • ◆Pooling (last-token vs mean)
  • ◆Abstention threshold (cal-tuned, frozen for test)

Results

  • →Milestone A: single entry point verified; full suite 48/48 tests green.
  • →Smoke training (200 steps): loss 4.586 → 2.268; checkpoint sha fd3914f55b8d8070; resume bitwise identical.
  • →Held-out eval: trained ≈ control, delta 0.0 on recall@k and MRR — honest negative, no generalization. Train split: recall@10 1.00 / MRR 0.87 vs control 0.16 / 0.08.
  • →Frozen benchmark: B vs A INCONCLUSIVE (median reduction 0.504, CI straddles 0.50); C vs A FAIL — learned checkpoint NOT cleared for serving; production stays on the labeled deterministic baseline.

Observations

  • ·Last-token pooling is tail-dominated (H-block carries ~no content to the final position); mean pooling recovers content sensitivity (16/60 unique embeddings vs 3–4).
  • ·Abstention (cal-tuned threshold): F1 0.667, precision 0.5, recall 1.0 on both pool modes.

Conclusion

Milestone A is done and the pipeline is real, but the learned model did not generalize to held-out queries. The honest result is the negative one — and it is recorded as such.

Next Experiment

Run the real 56M pilot on Mac/MLX or an authorized CUDA GPU when Paul points at one; do not rent compute.