A smaller operator is not automatically a better model: the HELIX-Ω edge-training experiment

TL;DR

HELIX-Ω replaced dense MLP matrices with compact tensor-axis transforms on an M3 Max. The operator saved roughly 22% of parameters, 21% of estimated FLOPs, and 30–32% of peak MLX memory—but lost the validation-quality frontier. A matched diagonal rescue recovered essentially none of the gap, so the branch was stopped.

The goal: make edge training cheaper without pretending cheaper means equal

The experiment began with a practical edge-hardware question: can a language model replace some of its largest dense matrices with structured transforms that store and apply far less state? The first target was deliberately narrow. Attention stayed dense. Only the three projections inside each gated MLP changed, so any gain or loss could be attributed to the new operator.

The accounting was separated from capability from the start. We recorded physical trainable parameters, active state, dense operator dimensionality, estimated arithmetic, wall time, throughput, peak memory, validation loss, and perplexity as different measurements. Saving memory is useful; it is not evidence that the model retained what the dense matrix could learn.

Y = X ×₁ A₁ ×₂ A₂ … ×ₘ Aₘ

change: dense MLP projections only
held constant: attention, width, depth, data, tokenizer, optimizer, precision, hardware

The first real-data pilot

Three order-swapped dense / TensorAxis seed pairs trained on the same WikiText-2 stream for 4,096 steps each. The structured arm did exactly what the complexity analysis predicted on resources: 22.37% fewer physical parameters, 20.68% fewer estimated FLOPs per token, and 32.19% less mean peak MLX memory.

It did not preserve quality. Dense mean validation loss was 5.56445; TensorAxis was 5.72141, worse in every seed. Even the wall-clock story turned against the structured model: dense reached the fixed validation landmark substantially earlier. The operator was cheaper per step but needed more learning to reach a worse place.

One bounded rescue, with stricter controls

Before closing the branch, we tested the smallest plausible repair: a learned diagonal scale after each tensor-axis projection, initialized to exact identity. The rescue used hash-pinned shared initialization, not merely matching seed numbers. Within each triad, every embedding, attention weight, and norm was byte-identical; pure and diagonal factor matrices were identical; initial logits matched exactly.

The pass condition was intentionally demanding: recover at least half of the pure-to-dense loss gap, improve in both seeds, keep every seed above a 25% recovery floor, preserve the resource savings, and stay within 10% of pure TensorAxis throughput.

The rescue result

It failed cleanly. Mean dense / pure / diagonal loss was 5.73598 / 5.83298 / 5.83290. One seed recovered 0.366% of the gap; the other went backward by 0.317%. Mean recovery was 0.025%—effectively zero—and the matched pure-to-dense gap remained large enough to rule out initialization luck as the explanation for the original result.

The diagonal still saved 22.35% of physical parameters, 20.65% of estimated FLOPs per token, and 29.78% of peak MLX memory versus dense. But it also reduced compute throughput by 10.40% versus pure TensorAxis. It passed the resource gates and failed the gates that mattered for continuing.

mean loss      dense 5.73598 | pure 5.83298 | diagonal 5.83290
mean recovery  0.025%         | required 50%
compute tok/s  pure 36,023    | diagonal 32,275
decision       STOP

What we keep

This is not a failed research process. It is a successful falsification. The experiment proved the implementation was correct, proved the structured operator really reduced resources, proved the quality penalty survived matched initialization, and prevented custom-kernel work from optimizing an architecture that had not earned it.

The tested one-stage pure and diagonal TensorAxis branch is closed. HELIX-Ω remains the name of the broader research platform, but a future experiment must start from a materially different mathematical hypothesis with its own fair control and stop rule—not an automatic pile-up of permutations, routing, residual rank, or kernel complexity.