Research
Findings about small holographic models: how they use context, how they grow and how they compare. Each article leads with its finding and says what would refute it.
All articles
-
The Four-Token Ceiling
A holographic language model pinned to four tokens of context stops improving its top-guess margin at about round 48,000 and stops adding spelled words between rounds 15,000 and 21,000, while calibration and the specialisation of one weight bank keep changing through more than 80,000 rounds.
-
Growing the Wrong Way
Growing a four-token model that had plateaued to eight tokens of context gave no recovery in about 100,000 rounds, and growth by appending new position slots after the most-trained one demotes that slot on every grow, whereas inserting new slots before the oldest one keeps it in place.
-
The Model Learns Back To Front
In a holographic model grown from one layer to six, only the last layer's two passes carried information: reading every intermediate state against the vocabulary shows the earlier passes name a single token at confidence 0.9740 to 1.0000, and the four-layer and six-layer snapshots are identical to two decimal places over their last four read points.
-
Smaller Than TinyStories
A holographic language model with a parameter count of 107,776 is about a tenth the size of the smallest published TinyStories reference model and spells 88.5% of its words correctly, but it does not write readable stories.
-
Nobody Read the Warning
The effective number of independent judgements in a group falls as its members become correlated, to about one when they are fully coordinated, and the surviving accounts of collapse are consistent with that mechanism but were preserved as scripture, not read as analysis.