AlgFormer how holoformer works

How HoloFormer works

HoloFormer is the holographic attention core in AlgFormer. This page explains it in plain terms: how a token is stored, how ideas are combined, how the model thinks longer and how it remembers a whole context in one place. Each idea ends by naming the part of the model it describes.

A token is a chord

Give the model a letter, a word or a number and it writes it down as a chord: a particular stack of pure tones played together. Each tone is one channel of a complex-valued vector, and the chord is the whole vector. One word is one chord and another word is another.

How many tones can a chord stack? That is the size of the instrument. A small instrument has few strings, so chords blur into one another and the model cannot tell things apart. A bigger instrument keeps chords crisp and distinct.

In the model: The tones are the channels, set by dModel when you construct the model. More channels keep meanings distinct.

Combining is playing together

The model has essentially one core move. To join two things it plays their chords together, and they merge into a single new chord. That is how a word is tied to its role, or a question to its subject.

Numbers are chords tuned so that playing two of them together adds their values. The chord for 5 played with the chord for 3 decodes as 8. The arithmetic is in the sound itself, so the model does not have to memorise a times table the way an ordinary transformer would. The program below is the Phasor codec that supplies the chords.

Binding two numbers and reading a bundle back

using Phasor;

// Encode a number and some symbols
double[] five = PhasorCodec.Encode("5");
double[] three = PhasorCodec.Encode("3");
double[] age = PhasorCodec.Encode("age");
double[] name = PhasorCodec.Encode("name");
double[] hello = PhasorCodec.Encode("hello");

// Bind two numbers: adds them on the linear band
double[] fivePlusThree = PhasorCodec.Bind(five, three);
Console.WriteLine($"5 + 3 = {PhasorCodec.DecodeSum(fivePlusThree, max: 10)}");

// Bind each value to a role, bundle the pairs into one record, then unbind a role to read its value back
double[] record = PhasorCodec.Bundle(PhasorCodec.Bind(age, five), PhasorCodec.Bind(name, hello));
double[] ageField = PhasorCodec.Unbind(record, age);
Console.WriteLine($"age field matches 5 better than 3: {PhasorCodec.Correlate(ageField, five) > PhasorCodec.Correlate(ageField, three)}");

// For larger workloads and similarity search, HoloCodec is SIMD and configurable in width
var codec = new HoloCodec(dim: 1024);
double[] encoded = codec.Encode("hello");
Console.WriteLine($"HoloCodec self-similarity is positive: {HoloCodec.Dot(encoded, codec.Encode("hello")) > 0}");

Output

5 + 3 = 8
age field matches 5 better than 3: True
HoloCodec self-similarity is positive: True

In the model: This one move is binding. For numbers, binding is addition. The codec comes from the Phasor package.

Chord moves

Stacking chords is not enough. The model also has to transform them, steering a question toward its answer. It does this with a small set of learned moves, each of which blends in tones from elsewhere on the instrument.

More moves let it express richer relationships. The moves are spread across the instrument rather than only nudging neighbouring strings, so it can connect distant ideas and not only adjacent notes. At least two are needed: with one move every tone only ever reads itself, and nothing can route between tones.

In the model: The moves are the shifts of the relation bank, spread by golden-ratio spacing so a few of them cover the whole range. HoloShape.ShiftsFor chooses how many a given context needs.

A chain of arrangers

The sound is not shaped all at once. It flows through several stages in a row, and each stage restyles it with its own set of moves before handing it on, the way a melody passes through a chain of arrangers.

In the model: The stages are the layers. Depth is how many times the sound is reworked.

Play it through twice

Thinking, for this model, is replaying. Each stage can play its part again over what it just laid down, and only then hand the sound on. The first pass sketches and the second develops the sketch.

Replaying costs no extra parameters, because the same stage is simply applied again. The model can think harder about a short input by playing it through more times, and a newly added stage can be eased in by letting its replay weight start at zero and ramp up.

In the model: The repeats are the K weight-tied iterations of each layer (the StackIter family), with a per-layer alpha that ramps a new layer in.

One shared sound you tune into

This is where HoloFormer parts company with an ordinary transformer. A transformer makes every position compare itself with every earlier position: a crowded room in which everyone talks to everyone, and the effort grows with the square of the conversation.

HoloFormer folds everything it has heard into one running sound per layer. Each position binds its key to its value and adds the result to the bundle. To recall something, the query is unbound from the bundle: the matching part rings out and the rest averages toward silence, like finding one station on a radio dial.

One bundle holds the whole context, so reading a prompt grows with its length rather than its square, and each new token costs the same however long the context has become. The chart on the overview page measures both cores. The held memory can also be listened to: the inspectors read off what the final query resonated with and decode it to tokens.

In the model: This is holographic memory: retrieval by resonance instead of every-position-to-every-position comparison. The incremental cache of this core is one running bundle per layer.

How much one sound can hold

Many bound pairs summed into one vector crosstalk: each read comes back with a little noise from all the others. A bundle can therefore hold a limited number of items cleanly. HoloShape reports two numbers for a shape. The crosstalk edge is the number of shifts times the width, halved. The clean capacity is the number of tokens that read back at least 90 percent clean, which is about 12 percent of the edge.

Models are sized so that the context sits at a chosen fraction of the edge, and the default is a quarter. At a width of 128, HoloShape derives a context of 2,048 tokens. The crosstalk edge there is 8,192 tokens and 983 tokens read back clean. The constants behind the derived shapes are single measurements, so treat the shape as a starting point.

Sizing a shape with HoloShape

using PrismFormer;

int ctx = 64, dim = 32;
int shifts = HoloShape.ShiftsFor(ctx, dim);                       // the normal way to choose Shifts
bool collides = HoloShape.ShiftsCollide(shifts, dim, golden: true);
Console.WriteLine($"ShiftsFor({ctx},{dim}) = {shifts}, collides = {collides}");
Console.WriteLine($"bind rank {HoloShape.BindRank(shifts, dim)}, reads back clean {HoloShape.CleanCapacity(shifts, dim)}");

// A starting shape from a width alone. Treat it as a place to begin, not a guarantee.
foreach (var width in new[] { 64, 128, 256 })
{
    var s = HoloShape.Derive(width, stackK: 2);
    Console.WriteLine($"Derive({width}): shifts {s.Shifts}, context {s.Context}, layers {s.Layers}");
}

Output

ShiftsFor(64,32) = 16, collides = False
bind rank 256, reads back clean 30
Derive(64): shifts 64, context 512, layers 3
Derive(128): shifts 128, context 2048, layers 6
Derive(256): shifts 256, context 8192, layers 8

Side by side with an ordinary transformer

What differs between the two attention cores
Aspect	Ordinary transformer (softmax attention)	HoloFormer (holographic attention)
Meaning	A point in a learned space.	A chord: a vector of phases.
Combining ideas	Learned tables, taught every combination.	Binding: chords played together. For numbers it adds.
Remembering	Every position compares with every earlier position.	One running bundle per layer, read by resonance.
Cost of attention	Reading a prompt grows with the square of its length, and the cache grows with it.	Reading a prompt grows with its length, and the cache stays the same size.
Thinking longer	More layers, so more parameters.	Replay the same layer, with no new parameters.
Looking inside	Hidden numbers.	Inspectors read off what a query resonated with and decode it to tokens.

An ordinary transformer memorises the music. This one plays it.

Try it

The holographic core takes the same constructor as the softmax core. This program trains one, serves it through the incremental cache and reads what its final query resonated with.

HoloFormer: train, serve and inspect

using PrismFormer;

// The holographic core has the same constructor shape. Swap the type, keep the workflow.
var holo = new HoloFormer(vocab: 16, shifts: 4, layers: 2, maxContext: 6, dModel: 32, frozenPrefix: 0);

var data = new List<(int[] Ctx, int Target)>
{
    (new[] { 1, 2, 3 }, 4),
    (new[] { 2, 3, 4 }, 5),
};
double loss = holo.TrainEpoch(data, batchSize: 2, lr: 5e-2, shuffleSeed: 1);

// Serve with the incremental cache: one running holographic memory per layer, so each token costs the same
// however long the context has become.
var cache = holo.NewCache();
double[] afterPrompt = holo.Prime(cache, new[] { 1, 2, 3 });
double[] afterNext = holo.Step(cache, 4);

// Read the model's mind: the resonance of the final query against every earlier position, and what it retrieved.
var alpha = new double[holo.Layers]; Array.Fill(alpha, 1.0);
int[] toks = { 1, 2, 3 };
double[][] resonance = holo.InspectAttention(toks, K: 2, alpha);
double[] retrieved = holo.InspectRetrieval(toks, K: 2, alpha);
double[] decoded = holo.DecodeFace(retrieved);          // per-token logits for what the query pulled out

Console.WriteLine($"loss={loss:F3} cache length={cache.Length} logits lengths {afterPrompt.Length},{afterNext.Length}");
Console.WriteLine($"resonance rows={resonance.Length}, positions per row={resonance[0].Length}, decoded logits={decoded.Length}");

Output

loss=2.777 cache length=4 logits lengths 16,16
resonance rows=4, positions per row=3, decoded logits=16

Back to AlgFormer