Smaller Than TinyStories

The yardstick is TinyStories [1]. That paper trains very small language models on a synthetic corpus of simple children's stories to see how small a model can be while still producing text that reads as coherent English. Its published reference models come in five sizes: 1M, 3M, 8M, 28M and 33M parameters. This lineage asks how much a model can do at a size TinyStories did not try.

The checkpoint is weight-tied and runs entirely in the browser in the Prism tool at /tools/prism/, with nothing server-side. Its shape is d=96, 6 layers, 16 shifts, a context window of 192 tokens, a 256-token vocabulary, and 2 weight-tied passes over the same weights per forward step. The file is 862,244 bytes raw and about 471KB gzipped, which is what a browser downloads to load the tool.

Bits per character is the measure of the model's remaining uncertainty about the next character. It is not comparable to the bits per character the TinyStories paper reports, which uses a different tokenizer and a different measure. No benchmark comparison is made on it.

The checkpoint has a parameter count of 107,776. Against the five TinyStories sizes, that is roughly a tenth the size of the smallest (1M parameters) and roughly a three-hundredth the size of the largest (33M parameters). On parameter count it is smaller than anything in that lineup, by a wide margin.

A second dimension is vocabulary granularity. The tokenizer has 256 entries and is a bigram vocabulary, not a word-level one. Mean token length across the vocabulary is 1.63 characters; 95 tokens are a single character, 160 are two characters, and no token is longer than two. Every word is built a couple of characters at a time, with no lookahead past the current token. The word Once is O, nc, e; upon is u, po, n; little is li, tt, le.

Measured across real prose, it takes an average of 2.84 tokens to produce one word. A word-level TinyStories-style model typically spends one token per word. A five-word phrase such as going to there with them is 5 correct decisions in a row at word granularity and 15 correct token decisions in a row here, three times as many, each made without sight of the next token. Spelling a word of five or more letters takes two or three sequential blind token decisions, each consistent with the ones before it.

A sample from the checkpoint at a middling temperature, unedited: "One doging the saw the started to the started to the very. They w..."

This is not TinyStories-quality prose. No subject survives more than a couple of words, doging is not a word, and the text decays into a repeat loop (the started to the started to). The measured improvements below do not mean the model writes readable stories.

Across a held-out sample, 88.5% of the words the model produces are spelled correctly. By word length: 94.1% for 3-letter words, 87.4% for 4-letter words, and 49.5% for words of 5 letters or more. Long words are the figure to watch, because short words can be right by luck or by memorising a small closed set. Long-word spelling rose from 16.6% to 49.5%, roughly a tripling.

Degenerate looping, where the model falls into a short repeating cycle and stays, fell sharply. 59.0% of vocabulary-seeded generations produce no repeating loop, up from about 40%.

Bits per character sat flat at 2.6 to 2.7 through several restructures of the architecture that moved it in neither direction, then dropped to about 2.3 at roughly round 190,000 of continuous training.

Six checkpoints spanning the stretch were compared on how the weights moved. In all 42 bank-and-layer combinations in the model the weights rotated and did not grow. Every combination moved 10 to 1,000 times more by rotation (a change in the direction a weight vector points) than by rescaling (a change in its size). Cosine similarity between old and new direction sits between 0.977 and 0.999, barely different in raw angle but consistently different in the same direction checkpoint after checkpoint.

Magnitudes changed by under 1.5% on every weight bank across the window, and each layer's share of the total work stayed essentially flat. The token embeddings and the readout geometry stayed fixed: the effective dimensionality of the embeddings held between 14.7 and 14.9, and the readout geometry correlated between 0.996 and 0.998 across checkpoints. The model did not relearn what tokens mean or how to read its internal state back into a prediction. It reorganised how it relates things between those two fixed points, which is restructuring and not accumulation.

The flat stretch of the loss curve was not a stall. Rotation rate during the plateau at 2.6 to 2.7 bits per character was 0.0198 per round, faster than the 0.0159 per round averaged over the whole window that includes the drop to 2.3. Stopping when the loss curve looked flat would have stopped training when the structure was moving fastest.

Long-form coherence got worse over the same window. The longest run of valid English the model sustains shrank across the checkpoints, and a corruption test (feeding a deliberately broken input and seeing how much the model relies on the good context around the break to recover) suggests it leans less on context than before. That test runs across only 60 windows, too coarse to call either a real regression or noise.

The corpus changed on the same day as some of the measurements. TinyStories data is 55% of the 3.77GB training corpus. The spelling gains predate the corpus change, so they are not an artefact of seeing more TinyStories-shaped text. A later acceleration of roughly six-fold in the rate of weight rotation sits entirely inside the window after the corpus change, and how much of it comes from the architecture and how much from the changed training mix cannot be separated.

A model under a million parameters can spell most of its words correctly on a held-out sample, while still failing at stringing words into sentences. Spelling and coherence are separate measurements and need separate reports.

Compare size with vocabulary granularity in view. A bigram vocabulary needs about three token decisions where a word-level vocabulary needs one, so a parameter count compares two models only at equal granularity.

Do not stop a run because the loss curve is flat. Measure weight rotation against rescaling, and read a flat loss with a fast rotation rate as reorganisation in progress.

Keep a corruption test at more than 60 windows if it will carry a claim about context use, and record corpus changes against the measurement dates so they can be separated from architecture effects.

The size claim is closed and holds on parameter count and on token granularity. Nothing here shows the model writes like a TinyStories model, and the sample above shows it does not. No quality comparison against the TinyStories models was made, and bits per character is not comparable to theirs. Long-form coherence moved in the wrong direction on a coarse test, and the corpus change confounds the weight-rotation acceleration.

The vocabulary is described as 256 entries but the single-character and two-character counts (95 and 160) sum to 255.