Smaller Than TinyStories

Evaluated Applications · Published

A checkpoint in this lineage just crossed 107,776 real parameters, already smaller than every published TinyStories reference model, the smallest of which is roughly ten times its size. On size, that's just true. On writing ability, it isn't there yet, and the honest way to say this is that beating TinyStories on both counts is a bet I'm making, not a result I'm reporting. Here's a real sample of what it currently writes, then the actual numbers behind the size claim and the actual numbers behind why I think the bet is a reasonable one to make anyway.

On this page

Why TinyStories is the comparison

TinyStories is a 2023 paper (Eldan and Li, arXiv 2305.07759) built around a simple, useful idea: train very small language models on a synthetic corpus of simple children's stories and see how small a model can get while still producing text that reads as coherent English. Their published reference models come in five sizes: 1M, 3M, 8M, 28M and 33M parameters. It's a fair yardstick for what I'm building here, because the whole point of this lineage is the same question from the opposite direction: how much can you make a model do at a size TinyStories never tried to go.

This checkpoint is real, weight-tied, and runs entirely in the browser on the Prism tool page on this site — nothing server-side, nothing hidden. That's the same standard I'm holding it to here: if I say a number about it, you can go pull the same checkpoint and check it.

The size, plainly

The current checkpoint is 107,776 parameters. Shape: d=96, 6 layers, 16 shifts, a context window of 192 tokens, a 256-token vocabulary, and 2 weight-tied passes over the same weights per forward step. As a file it's 862,244 bytes raw, about 471KB gzipped, which is what actually ships to your browser when you load the tool.

Against TinyStories' own five sizes, that puts this model at roughly a tenth the size of their smallest published model (1M parameters) and roughly three hundredth the size of their largest (33M parameters). That comparison is solid and I'm comfortable stating it as a fact rather than a hope: on raw parameter count, this is already smaller than anything in their lineup, by a wide margin, not a close one.

There's a second dimension to that size gap that isn't obvious from the parameter count alone, and it's worth stating here before the sample below, because it changes how you should read everything that follows: this model isn't just smaller, it's also working with a much finer-grained vocabulary than a typical TinyStories-style model, which means it has to make more individual correct decisions to produce the same visible output. I'll come back to that with real numbers further down.

What it currently writes

Before anything else, here's an actual, unedited sample pulled from this checkpoint at a middling temperature. I'm putting it before the good numbers on purpose, because this site's whole voice is candid about what isn't built yet, and a milestone piece that let you infer more than this would be dishonest the moment anyone clicked through to the live tool and generated their own sample:

"One doging the saw the started to the started to the very. They w..."

That is not TinyStories-quality prose. It isn't close. There's no subject that survives more than a couple of words, "doging" isn't a word, and the whole thing decays into a repeat pattern ("the started to the started to") that's exactly the kind of loop I've written about in earlier pieces in this lineage. I want that sample sitting right here, not softened, not cherry-picked for a better run, because the rest of this piece is going to talk about real, measured improvement, and none of that improvement means the model currently writes readable stories. It doesn't. Not yet.

The actual milestone: spelling and bits per character

What did actually move, over the last stretch of training: bits per character, the measure of how much genuine uncertainty the model still has about the next character, sat flat at 2.6 to 2.7 for days, through several internal restructures of the architecture that didn't move it either way. It has now dropped to about 2.3, at roughly round 190,000 of continuous training. I want to flag immediately that this number isn't directly comparable to the bits per character the TinyStories paper itself reports — different tokenizer, different measure entirely — so I'm not claiming a benchmark win here, just describing a real drop in this model's own metric on its own terms.

The more legible number, to me, is spelling. Across a held-out sample, 88.5% of words the model produces are spelled correctly. Broken down by word length: 94.1% for 3-letter words, 87.4% for 4-letter words, and 49.5% for words of 5 letters or more. That last figure is the one I actually watch, because short words are easy to get right by luck or by memorizing a small closed set, and long words aren't. Long-word spelling has roughly tripled since the last time I measured it, from 16.6% to 49.5%.

Degenerate looping — the model falling into a short repeating cycle and staying there, the thing the sample above does — has also fallen sharply. 59.0% of vocabulary-seeded generations now produce no repeating loop at all, up from about 40% the last time I checked. That's real progress on the specific failure mode you can see in the sample above, even though, as that sample shows, the failure mode is still common enough to be the first thing you'll hit.

Why spelling here isn't the easy half

I want to correct my own framing here, because I nearly wrote this piece treating "spells words correctly" as the lesser achievement, the thing you get before the real work of stringing sentences together. That's wrong for this specific model, and it's wrong for a measurable reason: I went and actually checked the vocabulary instead of assuming it. This model's tokenizer has 256 entries, and it is a bigram vocabulary, not a word-level one. Mean token length across the whole vocabulary is 1.63 characters. 95 tokens are a single character, 160 are two characters, and the longest token in the entire 256-entry vocabulary is two characters. There is no token longer than that.

What that means in practice is that every word this model produces is built a couple of characters at a time, with no lookahead past the current token:

Once   ->  O | nc | e
upon   ->  u | po | n
little ->  li | tt | le
named  ->  na | me | d

Measured across real prose, it takes an average of 2.84 tokens to produce one word at this granularity. A TinyStories-style model, by contrast, typically uses a large word-level vocabulary where one token is roughly one word. That difference compounds fast once you're stringing words together. Take a five-word phrase like "going to there with them": at word granularity that's 5 correct decisions in a row. At this model's granularity, producing that same five words correctly is 15 correct token decisions in a row, three times as many, each one made without being able to see past it to the token after.

So when I say 88.5% of words come out spelled correctly, and 49.5% of five-plus-letter words come out spelled correctly, that's not the model doing the easy warm-up task before real composition starts. Correctly spelling a five-letter word at this granularity typically takes two or three sequential, blind token decisions in a row, each one consistent with the ones before it and none of them visible in advance. That's already the same kind of multi-step composition a word-level model is doing when it strings a short phrase together, just measured at a finer grain. It doesn't make the long-form sample above read any better today, and I'm not using this to talk that sample up. But it does mean the size comparison earlier in this piece has a second axis worth stating plainly: this model is roughly ten to three hundred times smaller in parameters than TinyStories' own range, and it's doing that while working at a token granularity fine enough that matching the same visible output costs it about three times as many sequential decisions.

What changed in the weights, not in the output

The more interesting story, and the actual reason I think the ability bet is reasonable, is underneath the output entirely. I compared six checkpoints spanning this whole stretch and looked at how the weights themselves moved, not just what they produced. The finding is consistent across all 42 bank-and-layer combinations in the model: the weights rotated, they did not grow. Every one of those 42 combinations moved 10 to 1,000 times more by rotation (changing what direction a weight vector points) than by rescaling (changing how big it is), with cosine similarity between the old and new direction sitting between 0.977 and 0.999 — barely different in raw angle, but consistently, deliberately different, checkpoint after checkpoint, in the same direction.

Magnitudes, meanwhile, barely moved at all: under 1.5% change on every weight bank across the entire window, and the share of the total work done by each layer stayed essentially flat throughout. Two other things stayed essentially fixed the whole time: the token embeddings and the readout geometry. Effective dimensionality of the embeddings held between 14.7 and 14.9 the entire window, and the readout geometry correlated between 0.996 and 0.998 across checkpoints. In plain terms: the model didn't relearn what tokens mean, and it didn't relearn how to read its own internal state back out into a prediction. What it did was reorganise how it relates things to each other in between those two fixed points. That's a genuinely different kind of change than "the model got bigger" or "the model saw more examples of the same pattern" — it's restructuring, not accumulation.

The plateau that was never actually stuck

This is the single most interesting thing I found this pass, and I want to say it plainly because it runs against what "plateau" usually implies. For days, bits per character sat flat at 2.6 to 2.7 and looked, from the loss curve alone, exactly like a model that had stopped learning. It hadn't. When I measured the rotation rate specifically during that flat stretch, it was 0.0198 per round — faster than the 0.0159 per round average across the whole window that includes the actual jump down to 2.3. The model was rotating toward whatever it eventually landed on faster during the plateau than it was on average overall. The loss just hadn't caught up to the reorganisation yet. If I'd stopped training when the loss curve looked flat, I would have stopped exactly when the underlying structure was moving fastest.

The part that went the wrong way

I don't want to write a milestone piece that only lists the numbers that moved in the direction I wanted, so here are the two that didn't, or that I can't yet be sure about.

The bet

So here's where this actually stands. On size, the claim is closed, and it's now two-dimensional: 107,776 parameters is roughly ten times smaller than TinyStories' own smallest published model and about three hundred times smaller than their largest, and it's getting there on a bigram vocabulary that costs it roughly three sequential token decisions for every one a word-level model spends on the same output. Both halves of that are true today, not a projection. On long-form ability, nothing in this piece should read as "it writes like TinyStories now." It doesn't. The sample near the top of this piece is what it actually produces stitched across a full generation, and it's well short of readable prose.

What I do have is a specific, measured reason to think ability is catching up rather than stalled, and I'd rather state it as what it actually is than round it up: this model is already sustaining long, blind runs of correct multi-token spelling decisions (88.5% overall, with long words roughly tripling to 49.5%) at a granularity where each one of those decisions is comparable in kind to a word-level model choosing its next word. Add to that loop-free generations up from about 40% to 59.0%, bits per character finally breaking a days-long plateau, and — the part I actually trust most — direct evidence that the weights kept reorganising fast the entire time the loss curve looked stuck, so the plateau was never really a stall. Set against that, honestly, is long-form coherence moving the wrong way over the same stretch and a corpus change I can't fully untangle from one of the more dramatic recent numbers yet.

That's the actual position: already smaller on both parameter count and token granularity, not yet better at stringing what it can already spell into something that reads as a sentence, with real evidence the underlying structure is still doing productive work rather than sitting still. Surpassing TinyStories on both size and ability is the bet this lineage is making. I'm not going to call it won before it's won — I'd rather publish the sample above now and let the next piece in this series show whether the bet paid off.

← Back to Articles