Growing the Wrong Way
Evaluated Applications · Published
The last piece ended with a real plateau and a decision to grow. This one is about what actually happened when I did, why my own hunch about it turned out wrong, and a structural bug I only found because I was willing to admit that.
On this page
The grow, and the long wait for a payoff that never came
The four-token model plateaued, cleanly enough that I trusted it, and the plan going in was straightforward: seed a new model from those final trained weights, grow it to eight tokens of context, and see whether all that hard-mining at the smaller size gave the bigger one a head start. That was the actual bet, not just "grow it," but "grow it from something that was genuinely finished, and see if finished pays off."
It didn't show up. Not in the first 20,000 rounds after the grow, not in the next 70,000. The primary metric, the same rank-gap number that told me ctx=4 was really done, never settled into a clean recovery. It oscillated in a wide band the entire time, past the point where the ctx=4 run's own entire training life would have finished. Calibration got worse before it got better. Long-form generation stayed collapsed into repeat loops at every single checkpoint I pulled, dozens of them, spread over nearly 100,000 rounds.
The one thing that didn't wobble was the same weight bank I'd already flagged as doing something general rather than context-specific. It kept specializing, checkpoint after checkpoint, completely unbothered by the grow. That was real, and it held up. Everything else about "is this settling in" didn't.
Was it the training procedure? No. Was the hunch wrong? Also yes.
My first instinct, when the numbers wouldn't move, was to blame the mechanics. I found a real, fixable gap: a learning-rate boost the training engine is supposed to give a model right after a context grow had never actually fired for this particular run, because the fork bypassed the normal path that would have triggered it. That felt like the answer for about five minutes.
It wasn't. The ctx=4 model had produced every bit of its own real improvement running at that exact same floor learning rate, the whole time. If a starved learning rate explained the stall, ctx=4 shouldn't have worked either, and it plainly had. I had to drop that explanation the moment I actually checked it against the model that disproved it.
With that gone, I didn't have anywhere left to hide from the actual conclusion: fully mining a small context before growing did not make the bigger context easier to absorb. That was the whole premise of the previous experiment, and the evidence, once I stopped explaining it away, didn't support it. I'd go further and guess the opposite might be closer to true. A model sitting in a fully converged minimum for one task may have less real room to redirect toward new capacity than a less-settled one would, not more. I can't prove that part yet, but the milder claim, "hard-mining first doesn't obviously help," is one I'm now confident in.
Growing wasn't the mistake, either
It would have been easy to read all that as "growing context doesn't work, stick with tiny models." I don't think that's right, and I think the same data that killed the hard-mining hunch also points at why. The one thing that never stopped improving through the whole ctx=8 mess was the general, non-position-specific structure, the same mechanism that seemed to be building reusable skill rather than something tied to a particular context size. If a piece of what the model learns really is generic, it shouldn't need a context size to be "finished" before it's worth having, and the evidence I actually have is consistent with that.
So the new plan wasn't slower training, it was faster growing: a context tier every 200 rounds instead of every 20,000, on the logic that the cheap, early, token-level stuff doesn't need to be relearned from scratch at a bigger size, and growing quickly through it should cost less than training a thousand-token model cold. Early results on that front actually look decent, comparable capability shows up at comparable round counts to what the four-token run needed, which is exactly what "generic skill transfers regardless of size" would predict.
The bug I only found by comparing round counts
Here's the part that only showed up once I started actually comparing the fast-growing run against the fixed-size one, round for round. The fixed-size model's problem was obvious: it plateaus, because there's nowhere left to go. The fast-growing model doesn't have that problem, it can keep going all the way to a real, useful context length. But growing quickly toward a thousand tokens exposes something the slow version never had time to show me.
Every position slot in the context window has its own dedicated, separately-trained representation, and the highest-indexed slot always means "the token immediately before the one I'm about to predict," the single most load-bearing position there is. Every time you grow the context, new capacity gets appended past that slot, which sounds harmless until you notice what it actually does: it takes the position that mattered the most, the one that had accumulated the most training, and demotes it to the middle of the window, replacing it with a brand new, untrained slot in the one role that mattered most.
That happens on every single grow, forever, no matter how carefully you pace it. Growing slowly doesn't fix it, it just spreads the damage out. By the time a model has grown all the way to a thousand tokens of context, the position doing the most important job, reading what was just said, has had less training time than almost any other slot in the entire window. That's backwards from what you'd want, and it's not a training problem at all, it's a structural one, baked into how growing has always worked.
Growing from the front instead of the back
The fix is a genuine reversal, not a tuning knob. Instead of appending new, untrained capacity past the most important slot, you insert it before the oldest one, and shift every existing position up to make room. The slot that means "immediately before prediction" keeps its own already-trained values and never gets bumped out of that role again, no matter how many more times the model grows. New capacity always lands representing context further into the past, which is exactly where it's cheapest to start from zero, since the far past was already the least useful part of a small window's memory.
I want to be honest about a wrong turn along the way. My first explanation for why this would be safe leaned on an assumption about how the model's attention mechanism handles relative distance between tokens. That assumption turned out to be wrong when it was actually checked against the real code, the mechanism doesn't work on relative position at all. The real reason the fix is safe is simpler and better grounded: the model's sense of position lives in exactly one place, a straightforward lookup table added onto each token, nowhere else. There was nothing else for the change to accidentally break, which is a cleaner argument than the one I started with, and I only have it because someone checked rather than trusted my reasoning.
It shipped as a real default, not a manual toggle you have to remember: any newly built model gets it automatically, and every existing checkpoint is explicitly pinned to the old behavior on load, so nothing already trained changes underneath itself. Old and new can sit side by side without either one silently drifting.
Where it stands now
I restarted training from scratch under the new convention. It's extremely early, a few hundred rounds in as of writing, nowhere near enough to say the fix delivered on its promise. What I can say honestly: the raw numbers at this stage don't look dramatically different from how the old approach looked at the same age, which is expected, the problem this fixes is about what happens over many grows, not about the first one. But the generated text is already producing real, cleanly space-separated word-shaped chunks this early, the individual words aren't spelled right yet, but the model has clearly already worked out where word boundaries go. Whether that's the fix paying off early or just ordinary progress I can't fully separate yet. I'll know more the longer this runs, specifically whether the long-form collapse that never once recovered under the old growth scheme finally does under this one.