The Four-Token Ceiling

The model is a fresh set of weights with the same shape and the same real corpus as the main lineage. Its context is fixed at exactly 4 tokens for the whole run, and no schedule grows it. The architecture runs one fixed weight matrix through eight refinement passes on every token, and the state after each pass can be inspected as well as the final answer.

Four tokens of context cannot carry long-range coherence: a model that sees only the last four tokens has no mechanism for staying consistent with a subject from six tokens back. The run is therefore graded on spelling and on local structure, not on whether it writes a sensible paragraph.

The extrapolation multiplier is the length of the longest correct stretch of output divided by the four-token window. A correct stretch longer than the window cannot be read off the input and has to be reconstructed from the weights, so the multiplier separates building from repeating within the window.

A permanent battery of tests runs directly against saved checkpoints. It was run every 3,000 rounds on a fresh checkpoint out past 80,000 rounds, and every snapshot from the run is kept.

Three measures stopped moving. The primary signal is the gap between the model's top guess and its second-best guess, a proxy for whether the runner-up is still catching the winner. It plateaued at about round 48,000. Splitting the run into thirds and comparing the middle third's average with the final third's gives equal values: the signal is flat, not slowly rising.

The effective vocabulary, the set of corpus words the model reliably spells, capped between rounds 15,000 and 21,000 and did not grow in the 60,000 or more rounds after. The measurement was made twice, independently, and both gave the same result.

The rate at which plain generation falls into a repeat loop reached a floor. At the latest checkpoints a plain generation run falls into a repeat loop in the large majority of runs. A long-form test on this run shows the collapse rate was already high at the earliest checkpoints and stayed in that band. A lower collapse rate at larger contexts, seen in a different and longer run, does not carry over to this one.

Two things never plateaued across the 80,000 or more rounds. The first is the specialisation of one weight bank: its channels specialise harder throughout the run, with zero reversals. One channel in that bank, the no-relative-shift channel, was the most active channel in two separate investigations on two different checkpoints.

That channel is consistent with a gate: it appears to decide, per token, how much of the eight-pass iterative budget is spent on real work and how much is served from an embedding-level guess. Part of the model's numeric encoding is a fixed codec that has not been trained since round one, and the model routes through it more over time. The codec does not change; the model leans on it more. This reading is an interpretation of the channel's activity, not a separately measured result.

The second is calibration. When the top guess is wrong, the rank of the correct answer and the probability mass placed near it kept improving after the primary metric went flat. The model did not get better at being right; it got better at being usefully wrong.

The specialisation trend is steady and linear, not accelerating. Acceleration is the signature of an approaching sudden transition of the kind called grokking, so the trend gives no sign of one approaching. The general phenomenon of a flat visible output over a reorganising interior is the grokking pattern; what this run adds is its observation in this architecture.

Raw accuracy under corrupted input rises with more training, which reads as growing robustness. It is not robustness: the model increasingly echoes the last visible token regardless of what was corrupted, and the corruption filler used coincided with the model's favourite echo target. The rise is an artefact of the test design.

The reduced collapse rate remembered from a different run does not appear in this run when tested directly with a long-form test (see the plateau section).

Compare the middle and final thirds of the margin signal before deciding a size is finished; a slow climb and a plateau look alike on a curve and differ in that comparison.

Once the effective vocabulary has capped, more rounds at the same context do not add spelled words. Further capability needs a change other than more rounds at that size.

Track calibration separately from top-guess accuracy. It can keep improving under a flat headline metric.

Do not read a rise in accuracy under corrupted input as robustness unless the filler is different from the token the model echoes.

One lineage, one shape and one corpus. The window is four tokens, so nothing here concerns long-range coherence. The extrapolation multiplier is defined here and used as the stopping test, but its measured values are not reported in this article. The gating and codec-reliance readings of the no-relative-shift channel are interpretations of activity, not isolated causal tests.