Project File
FieldTransformer
A physics-inspired architecture: multi-scale field layers with flux, Schwarzian, and latent-diffusion regularizers over causal attention.
Overview
EXPERIMENTALFieldTransformer layers a MultiScaleFieldLayer (k=3 + k=7 Laplacian convolutions) under causal self-attention, trained with physics-shaped penalties: normalized spatial gradient energy (flux), a Schwarzian curvature stability term, depth-smoothness latent diffusion, and an entropy bonus.
Results So Far
MEASUREDv1.2 (1.36M params, char-level) fit Tiny Shakespeare to loss 0.0109 / PPL 1.01 over 5 epochs. v1.3c was killed at 3,500 steps — recorded as a failure. v1.4 moves to a 4096-token BPE (3.26× compression) at 9.0M params. Small-corpus memorization is expected at these sizes; generalization tests are the open question.
Current Limitations
- −All results are on tiny corpora; PPL 1.01 on Shakespeare is memorization, and is labeled as such.
- −Custom IP terms apply to the WaveFieldLayer and Schwarzian regularizer (see repo notice).
Next Steps
- +Held-out generalization evals for v1.4.
- +Ablate each regularizer against a plain transformer control.