Attention Is All You Need

Perhaps you've played Chinese whispers, where a sentence passed ear to ear down a line comes out the far end as delightful nonsense. That's roughly how older language models read a sentence: each word handed its understanding to the next, one link at a time. By the time meaning from the first word reached the twentieth, it had passed through nineteen pairs of hands and blurred along the way — so distant words easily lost their connection. The Transformer's trick, self-attention, does away with the chain. It gives every word a direct line to every other word, so even the farthest two are just a single hop apart and nothing gets muddled in relay. It turns out that letting everyone talk to everyone, all at once, beats a long game of telephone.

No relay, no whisper — everyone hears the original.

Perhaps you, like me, have joined the one open checkout in a busy shop, watching the queue crawl while five other tills sit dark. That, more or less, is how an old recurrent network reads a sentence — one word at a time, and it can't start on "cat" until it's finished with "the," because each word's turn depends on the one before it. Self-attention throws open every till at once. Every word is served in the same instant, so a whole sentence is read in one shot rather than shuffled through a single lane. It's not a cleverer reader so much as a wider counter — and it's the plain reason the Transformer trains in days where the old models took far longer.

Don't queue the words — open every till.

Ask a large language model how it makes sense of a word, and the honest answer is: it goes shopping. Every word in the sentence quietly holds up a little label advertising what it offers — "I'm a place," "I'm an animal," "I'm a time" — and behind each label sits its actual content. When the word "sat" wants to know who did the sitting, it doesn't pick one word and ignore the rest. It reads all the labels at once, then pours itself a drink: a big glug from the bottle whose label fits best, and a thin trickle from each of the others. What it walks away knowing is that blend in the glass — mostly the closest match, faintly flavoured by everything else. That's all "attention" really is: not a single choice, but a mixture measured out by how well each label answered the question.

A big pour from the best bottle, a splash from the rest.

Perhaps you, like me, have read a sentence quickly and caught its gist — but missed the quiet link that changes everything. "The cat she fed." Who is "she"? What did the feeding attach to? A single read grabs one thread and lets the others slip. The Transformer's trick, in its "multi-head attention," is to refuse to read just once. It runs the same sentence through several readers at once — the paper uses eight — and each has learned to notice a different kind of relationship: one tracks which word is the subject of which verb, another which pronoun points back to which noun, another simply what sits nearby. None of them is right on its own; together they hand back a far richer sense of the line than any could alone. Then their notes are pooled into one.

One pair of eyes reads a line — a panel reads between them.

Perhaps you've noticed that "the cat sat on the mat" and "the mat sat on the cat" are built from exactly the same words. Same bag of tiles, wildly different picture. The only thing that tells them apart is the order. Now, the clever attention mechanism at the heart of modern language models has a curious blind spot: it treats its input as a bag of words, with no built-in sense of which came first. On its own it genuinely can't tell those two sentences apart. So before the words go in, each one is quietly stamped with a signal for its position — first, second, third — the way you'd number cards before shuffling them so you can always lay them back in order. Restore the numbers, and you restore the sentence.

No numbers, no word order — so number the words.

Perhaps you, like me, have started a sentence without quite knowing how it would end — and only found the ending once the beginning was down on paper. That is exactly how a Transformer produces an answer: one word at a time, left to right, each new word chosen by looking back over the words already written, never forward at the ones it hasn't produced yet. The paper enforces this with a rule it calls masking — while the model writes a word, the rest of the sentence is curtained off, out of reach. It's the same everyday attention from the earlier sketches, now pointed in one direction only. And it's precisely what a chatbot is doing when you watch its reply appear word by word: building a sentence it can't yet see the end of, out of everything it has already said.

It writes forward, but it can only look back.