Home/Articles
WritingLonger pieces than a product page has room for: why something is built the way it is, what we learned getting there, and ideas that don't fit neatly on a package's own page. No fixed schedule, no newsletter to sign up for — just check back, or come from a link.
newest first
A snapshot of the six-layer lineage that died this week shows two of its six layers, plus most of a third, were a mathematically exact no-op the whole time — growing from four layers to six added identical padding, not capability, before the run destroyed itself.
A checkpoint in this lineage just crossed 107,776 parameters, already well under every published TinyStories model, on a token granularity that costs it three decisions for every one a word-level model spends. Long-form coherence isn't there yet, and I've included a real sample.
The grow to 8 tokens didn't pay off the way hard-mining ctx=4 first was supposed to — and chasing down why led to a real structural bug in how context growth has always worked.
I hard-trained a tiny model pinned at 4 tokens of context to find out if my last one grew up too fast — where it plateaued, what kept improving anyway, and where my own read of the data was wrong.
Every civilisation that collapsed left us a report, and it's still readable — a look at why the warnings from Bronze Age collapse to Babel to the Talmud keep getting preserved perfectly and understood not at all.