Skip to content
D-CSIL™

Lab Note · Engineering · 2026-11-20

HELIX 209M: fused prefill and cached Metal decode in one request

TL;DR

One frozen HELIX checkpoint, two inference changes. On an M3 Max, a 4,032-token prompt with 64 generated tokens took 1.190 s instead of 3.279 s (2.757× paired). All 44 matched timed runs produced the same token IDs. This is a local latency result, not a model-quality improvement.

November 19 Central / November 20 UTC: what changed

We tested a frozen 209.05M HELIX multiturn SFT checkpoint at step 800 on a 36 GB M3 Max (MLX 0.30.0, FP32, batch size one). The optional fused rotational scan handles eligible prefill shapes. During cached one-token decode, a new opt-in path packs expert weights once and dispatches two Metal kernels: selected gate/up projections with SwiGLU, then selected down projection with weighted merge. The combined experiment hands the prefill cache directly to decode in the same model request.

The fused scan activates only for prompt lengths of at least 128 tokens divisible by 32. The 44-token control stayed on reference prefill and exercised only accelerated decode. No prompt padding, model training, checkpoint changes or deployment.

A separate warmed prefill-only test of the existing fused scan measured a 2.79× paired median across nine matched runs at 128, 160 and 192 tokens. The rounded median rates by shape were 1,923 to 3,820, 2,399 to 5,453, and 2,333 to 7,367 tokens/s, respectively. Each eligible prefill invoked the scan five times; the scan was not used during cached decode.

Four conditions, one request

For 4,032 input tokens and 64 generated tokens, the warmed median full-request times were 3.279 s stock, 1.510 s with fused prefill only, 2.987 s with cached Metal decode only, and 1.190 s with both. The median speedup paired within four rounds was 2.757× (2.624–2.795×). Time to first token fell from 2.590 s to 0.822 s. Median prefill throughput rose from 1,571 to 4,945 tokens/s and fixed-budget cached decode from 91.0 to 171.2 steps/s.

The 44 matched timing rounds across seven prompt lengths preserved exact greedy output IDs across all conditions. Separate numerical checks compared logits and the recurrent and attention caches. The timing starts with tokenized input, excludes loading and one-time weight packing, and uses a fixed decode budget. The packed weights add 480 MiB of memory. An application that already uses fused prefill should compare against the 1.510 s prefill-only control, not stock.

Weight packing did most of the work

In a separate matched application loop with ordinary stopping, three short retrieval-augmented prompts ran at 179.66 versus 93.29 output tokens/s with the same output token IDs. The controls show most of the gain came from packing expert matrices. In the fixed-step decode test, the two custom Metal kernels added just 2.3% paired median over the already optimized selected-expert MLX path. That makes the simpler MLX path a sensible fallback.

What did not improve

In one greedy story continuation, both tested foundation branches repeated phrases. The earlier-to-later checkpoint comparison also found repetition on that prompt, including at the selected multiturn checkpoint. In a separate 10-case retrieval pilot, the model produced 0/10 strictly grounded outputs with lexical retrieval and 0/10 with gold evidence. The deterministic checker was separate: its verdict matched the answer key in 9/10 lexical-retrieval cases and 10/10 gold-evidence cases. These are narrow probes, not capability rankings. Faster inference did not fix these failures.