The wins that don't survive a fair baseline
Seven papers on efficient attention and linear recurrence. At tiny scale, all seven beat a parameter-matched transformer on the same data. Give the transformer its own best learning rate and double the width, and six of the seven wins collapse into a tie.
The setup is deliberately boring. Same parameters (within a few percent), same tokens, same tokeniser and corpus, three seeds — and the part that matters, a learning-rate sweep so each model runs at its best, not the paper's default. A drawn architecture trained against a drawn transformer. One number: validation loss.
Small scale (~1M params): every architecture wins, by 0.08–0.12 nats/char (MixFormer the lone near-tie). That part reproduces.
Scale ×2 (~3M params): the picture changes.
| paper | small | scale ×2 |
|---|---|---|
| State Soup | held | held |
| Liger | held | tie (nose-ahead) |
| Griffin | held | tie (nose-ahead) |
| From Sparsity to Simplicity | held | tie |
| Parallax | held | tie |
| ELSAA | held | tie |
| MixFormer | tie | baseline ahead |
Only State Soup holds at scale outside the seed spread. Everything else lands in a tie, and MixFormer's baseline actually pulls ahead.
But look at the order. Rank the seven by their margin at scale, and the three gated-linear-recurrence models — State Soup, Griffin, Liger — take the top three slots (all still nose-ahead, +0.01 to +0.04). The four attention-approximation models — Parallax, ELSAA, From Sparsity to Simplicity, MixFormer — take the bottom four (dead tie to −0.02). Most of the individual gaps are inside the noise, so this is a direction, not a verdict. But it's a clean family-level split, and it's invisible until you tune the baseline.
The honest reading: at tiny width, almost anything beats an under-tuned transformer. What you're measuring when you skip the LR sweep is the baseline's handicap. Remove it, and most "wins" turn out to be the width being too small to separate architectures — with a faint, consistent lean toward gated recurrence as you scale.
Three seeds, character-level, 2× — this is not a scaling law. It's a question worth more compute: does the recurrence lean hold at 4×, 8×? That's the next run.
Papers (own implementations, matched compute):
- Parallax — Zuo et al., arXiv:2605.29157
- ELSAA — Heidari et al., arXiv:2607.20214
- From Sparsity to Simplicity — Ren et al., arXiv:2605.18865
- MixFormer — Guo et al., arXiv:2608.09468
- Griffin — De et al., arXiv:2402.19427
- State Soup — Pióro et al., arXiv:2406.08423
- Liger — Lan et al., arXiv:2503.01496
The full table — both scales, seed spreads — is on the results page.