AI for Everybody - Lesson 11

How a Language Model Actually Works: The Transformer in One Diagram

For three lessons now we have been describing what the model does. It tokenizes its input (Lesson 7), runs a probability loop over the vocabulary (Lesson 8), produces output one token at a time from that probability distribution (Lesson 9), and ended up with those probabilities through training (Lesson 10). What we have not described is what kind of machine performs the probability computation in the middle. That machine is called the transformer, and the entire current generation of frontier AI rests on a single 2017 paper that named it (Vaswani et al., 2017).

This week’s lesson is unusual. The transformer is the most-discussed neural-network architecture in the history of the field, and the temptation to teach the math is strong. I am going to resist it. The math is calculus on a graph of matrix multiplications; if you want it, the references at the end of this lesson hand you the canonical accessible treatment. What you actually need from this week is a one-paragraph mental model of what the transformer is doing, and the single piece of vocabulary that every later lesson will use: attention.

The transformer in one paragraph

When the transformer reads a sequence of tokens (Lesson 7’s chunks), it does not read them strictly left-to-right the way you are reading this sentence. It reads them all at once. Every token in the input looks at every other token in the input and asks, which of the other tokens should I pay closer attention to when I am figuring out what comes next here? The transformer answers that question with a number for every other token: a high number means “highly relevant to this position right now,” a low number means “not relevant right now.” Then every token’s internal representation is updated based on the weighted blend of the tokens it decided were most relevant. That weighted-blending procedure runs in parallel across the whole sequence, then runs again as another layer, then again, dozens of layers deep. The output of the last layer is the probability distribution Lesson 8 described.

That is the transformer. The trick is the which-other-tokens-should-I-pay-attention-to step. The published name for that step is attention, and the title of the 2017 paper that named it was a confident declaration: Attention Is All You Need. The authors meant it. The architecture they proposed had thrown out the older sequential-processing machinery and replaced it with attention layers stacked on top of attention layers. The bet looked aggressive in 2017. By 2020 every leading model used the same architecture, and the bet looks obvious in hindsight.

What “attention” actually does, in plain English

Here is a small worked example. Take the sentence the bank by the river was crumbling under the weight of the heavy rain. When the model is predicting what kind of word comes after bank, the word bank on its own is ambiguous: it could mean the financial institution or the side of a waterway. The model needs to know which.

In an attention layer, the position holding bank gets to look at every other position in the sentence and assign each one a relevance score. The gets a low score (not informative for disambiguation). Was crumbling gets some weight. River gets a high score. Heavy rain gets weight too. The model blends in information from the high-attention positions, and the internal representation at the bank position shifts toward “side of a river” rather than “place where money is kept.” Now when the model predicts the next few tokens, it is predicting from a representation that has already absorbed the contextual cue from elsewhere in the sentence.

That is one attention step. The transformer applies this maneuver to every token, in parallel, then stacks the result through another layer that does the same thing on the updated representations, then another, and another. After dozens of layers, every position in the input holds a representation that has been informed by the most relevant pieces of the rest of the input, at multiple levels of abstraction. The probability distribution at the final layer reflects that thorough contextualization.

The reason this matters for you, as a user, is that the model’s behavior on a prompt is sensitive to what else is in the prompt. Add an irrelevant sentence and the attention weights shift. Add a tone-setting opening and the weights shift. Frame a question one way and certain other-word relevance scores spike; frame it differently and different ones do. Prompt engineering, which lesson by lesson we will start mechanizing in Part 3, is in the simplest reading a craft of choosing inputs so that the right attention scores fire.

The one diagram

If you want a visual, search “the illustrated transformer.” Jay Alammar’s 2018 essay of that title (Alammar, 2018) is the canonical accessible diagram of the architecture, taught at intro ML courses worldwide. The essay walks through the same flow this lesson described, with each box labeled. Three things to notice in the diagram if you go look:

Embeddings at the bottom. Every token enters the transformer as a vector of numbers, called an embedding, which is the architectural unit Lesson 12 will own. For now, take it on faith that there is a numerical address for every token in the vocabulary, and the address tells the model something about what the token means.

Repeated identical blocks stacked vertically. The transformer is a stack of the same block, applied over and over. Each block does attention plus some additional simple processing. The depth of the stack (how many blocks) is one of the architectural parameters that distinguishes a small model from a frontier model.

The probability output at the top. After the stack, the final representation at each position runs through one more transformation that maps it to a probability distribution over the vocabulary. That is the distribution Lesson 8 named and Lesson 9 examined.

Everything else in the diagram is either an engineering detail (positional encodings, layer normalization, residual connections) or a refinement of the attention step (multi-head attention, scaled dot-product attention). The engineering details are necessary for the model to train at scale; they are not necessary for the mental model. The refinements are interesting but the core idea is the same: every token looks at every other token, decides which are relevant, and blends.

Why the transformer changed everything

Before 2017, the dominant architectures for language tasks were recurrent neural networks: machines that read tokens one at a time, in order, maintaining a running internal summary as they went. The summary-as-you-go design meant that the model’s attention to early tokens decayed as the sequence got longer. Long-range dependencies (the technical term for caring about a token from way back in the sequence) were the architecture’s structural weakness. Recurrent networks were also hard to parallelize, since each step’s output depended on the previous step’s running summary. Training was slow.

The transformer architecture changed both of those at once. Every token can attend to every other token directly, regardless of distance, so long-range dependencies are no harder than short-range ones. And the all-tokens-at-once computation is parallelizable across modern hardware in a way the sequential recurrent loop never was. The combination unlocked the ability to train much larger models on much more data, in much less time, which is the immediate technical precondition for everything Lessons 12 through 14 will cover.

The Vaswani et al. 2017 paper is one of those rare cases where the title is honest. Attention really was, mostly, all they needed. Six years later, the same architecture is running every chatbot you have ever talked to.

Going Deeper (optional)

The two technical refinements of attention that show up in the diagram and matter for understanding modern systems are multi-head attention and Query-Key-Value attention.

Multi-head attention runs several copies of the attention step in parallel, each with its own learned set of weights, then concatenates the result. The idea is that different copies (the “heads”) can learn to attend to different kinds of relationships at once: one head might learn to track grammatical agreement, another might learn to track topic continuity, another might learn co-reference. The paper used eight heads in its base model. Modern models use considerably more.

Query-Key-Value attention is the specific mathematical form attention takes inside the transformer. Each token produces three vectors from its representation: a query (what it is looking for), a key (what it offers to others), and a value (what it actually contributes when attended to). The relevance score between two tokens is computed from the dot product of one token’s query with another’s key; that score then weights how much of the other’s value flows in. The math is one matrix multiplication per pairing, scaled by a factor for numerical stability. The Vaswani paper’s §3.2 walks through this in three pages.

The canonical, gentle, math-included walk-through is Alammar’s Illustrated Transformer. The canonical “with code” walk-through is Sasha Rush and the Harvard NLP group’s Annotated Transformer, which runs through the paper line by line with executable PyTorch. If you ever want to know exactly what the architecture computes, those two together are the path.

What you have, what comes next

You now know what the transformer is at the level of mental model: stacked attention layers, every token looks at every other token, the probability distribution falls out at the top. Attention is the single piece of vocabulary you have to carry forward; the rest of the course will use the word without re-defining it.

Lesson 12 picks up the embeddings at the bottom of the diagram and asks the natural question: what does it mean for a word to be a vector of numbers? The answer involves a beautiful geometric idea: in a well-trained model’s embedding space, words that mean similar things sit near each other, and certain analogical relationships (king is to queen as man is to woman) literally turn into vector arithmetic. That is the subject of Lesson 12, and after that we have the last two pieces of the Part 2 puzzle: how much the model can see at once (Lesson 13: context window) and what happens when you scale all of this up (Lesson 14: size and capability).

If You Want to Dig Deeper

For the gentlest illustrated walk-through of the transformer architecture, with diagrams of attention at every step, Jay Alammar’s 2018 essay remains the field-standard introduction. It is what most graduate courses send students to before they read the original paper. Alammar, J. (2018, June 27). The illustrated transformer. https://jalammar.github.io/illustrated-transformer/

For a code-included treatment that walks through the Vaswani et al. paper line by line with executable PyTorch, the Harvard NLP group’s “Annotated Transformer” is the standard reference. It is the source most graduate students learn the actual implementation details from. Rush, A. M., et al. (2018, April 3). The annotated transformer. Harvard NLP. http://nlp.seas.harvard.edu/2018/04/03/attention.html

For the original paper itself, the writing is clearer than its reputation suggests; readers who got through Lessons 7 through 11 can follow the introduction and §3 without much friction. The math gets dense in §3.2, but the intuition is what we covered above. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is all you need (preprint). arXiv. https://doi.org/10.48550/arXiv.1706.03762

Author Note:

Grace Ann Hansen is an independent researcher and writer, and an MBA & PhD graduate student in health informatics and artificial intelligence. She is also a published author, a professional musician, a gymnastics coach, and a queer transgender woman living in Sioux Falls, South Dakota. She corrects all her papers and articles with Grammarly, because even though she has deep thoughts, she has shallow patience for punctuation. She uses Anthropic’s Claude in Research mode for source location and verification on cited factual claims; all interpretation, argument, and prose are her own. Correspondence concerning this article should be addressed to Grace Ann Hansen at grace@graceannhansen.com.

References

Alammar, J. (2018, June 27). The illustrated transformer. https://jalammar.github.io/illustrated-transformer/

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is all you need (preprint). arXiv. https://doi.org/10.48550/arXiv.1706.03762