AI for Everybody — Lesson 12

How a Language Model Actually Works: Embeddings: Words as Coordinates Continue reading on Artificial Intelligence in Plain English »

AI LITERACY

How a Language Model Actually Works: Embeddings: Words as Coordinates

At the bottom of last week’s transformer diagram, every token entered the network as a vector of numbers. That vector has a name. It is called an embedding, and the geometric idea behind it is one of the prettiest things in the whole field. This lesson is short by Part 2 standards since the idea is small, but it carries a load of intuition that the rest of the course will keep coming back to.

The one-line summary: an embedding is a location in space, and words that mean similar things end up at nearby locations. That is the whole concept. The rest of this lesson is what that means, why it works, and the famous result that put it on the map.

A small picture you can hold in your head

Imagine a piece of graph paper with two axes. Now imagine taking every word in the English language and writing each one at a particular spot on the paper, in such a way that words with similar meanings sit near each other. Dog near puppy. Cat near kitten. Run near sprint. Banana somewhere in the fruit corner, near apple and pear. The locations are not arbitrary; the closeness encodes how related the words are.

Two-dimensional graph paper is too small for this to work well in real life. Real language models use embedding spaces with somewhere between 100 and 10,000 dimensions, depending on the model. You cannot picture a 4,096-dimensional space. You do not need to. The two-dimensional intuition is correct; there is just more room. Each word, or more precisely each token in the model’s vocabulary (Lesson 7), is assigned a point in this high-dimensional space. The point is its embedding. The numbers that make up the point are what the transformer (Lesson 11) actually reads when it processes that token.

The reason embeddings sit at the bottom of the transformer diagram is that the model needs a numerical input to operate on. A word like banana on its own is not something a network of mathematical operations can multiply or add. The embedding turns banana into about a thousand numbers (give or take), and those numbers are what the network manipulates.

Where the closeness comes from

You might reasonably ask how the model knows that dog and puppy should sit near each other. Nobody hand-labels the embeddings. They are not derived from a dictionary. They emerge, during training, from a simple statistical observation: words that tend to appear in similar contexts probably mean similar things.

The intuition is older than computers. Think about how you would teach a child what banana means without using a dictionary. You would show them bananas, and you would talk about bananas in the kinds of sentences bananas show up in: we are having banana bread for dessert, the bananas are not ripe yet, peel the banana first. After a few hundred such sentences, the child has a working sense of what banana means without anyone having sat them down with a definition. The meaning is built up from the company the word keeps.

Word embeddings work the same way at the statistical level. During training (Lesson 10), the model is shown enormous amounts of text and adjusted to predict each next token from its context. As a side effect of getting good at that prediction, the model has to learn what kinds of contexts each word appears in. Words that appear in similar contexts get pushed toward similar locations in the embedding space, since the model is using their embeddings to make similar predictions. The geometry of the space ends up reflecting the statistics of how words are used in the corpus.

This was first demonstrated cleanly in a 2013 paper introducing a method called word2vec (Mikolov et al., 2013), which produced word embeddings simple enough to study and good enough to be useful. Modern transformer embeddings are richer and more contextual, but the underlying idea is the same: the geometry of the embedding space reflects the statistics of how words are used.

The famous arithmetic

The Mikolov et al. paper also surfaced a striking property of the learned embedding space. Relationships between words showed up as directions in the space. If you took the embedding vector for king, subtracted the vector for man, and added the vector for woman, the resulting vector landed strikingly close to the embedding for queen. In other words:

king − man + woman ≈ queen

This was the result that made embeddings famous. The arithmetic suggested that the relationship “the male-to-female axis” was encoded as a particular direction in the space, and you could navigate along that direction to move between related word pairs. The same arithmetic worked for Paris − France + Italy ≈ Rome (the capital-of-country axis), for walked − walking + swimming ≈ swam (the past-tense axis), and for many other relationships the researchers tested.

The result was philosophically interesting because it suggested that meaning-like structure was emerging from pure prediction training, without anyone having designed it in. It was also practically important because it showed that embeddings were not just opaque numbers; they had geometric structure that corresponded to semantic structure, and that geometric structure could be used.

A reasonable note of caution. The famous analogies work cleanly in the simplest demonstrations and become noisier on harder cases. Subsequent work has shown that the analogy effect is sometimes weaker than the initial demonstrations suggested, and depends on how the analogies are evaluated (Nissim et al., 2020). The geometric structure is real. It is not as clean as the cleanest examples make it look.

What embeddings are used for, beyond the transformer

Inside a transformer (Lesson 11), embeddings are the input format. They flow into the bottom of the architecture and get progressively refined through the attention layers. That is one use.

A separate use, which has become a workhorse of applied AI systems, is semantic search. The idea is direct. If embeddings encode meaning as location in space, you can search for documents that mean something similar to a query by computing the query’s embedding, computing each document’s embedding, and finding the documents whose embeddings are closest to the query’s. This is how modern search inside large document collections works. It is how AI-assisted code search works. It is the engine behind a piece of architecture called retrieval-augmented generation that lets a chatbot look things up in a private document set before answering. The retrieval-augmented generation pattern is the subject of Lesson 30; you can think of this lesson as setting up the geometry it relies on.

The technical specialty that makes search-by-meaning fast at scale is called vector search or vector similarity search. The canonical open-source library for it is Facebook AI Research’s FAISS (Johnson et al., 2021), which is what you would reach for if you were building such a system. The library is technical infrastructure; the geometric idea behind it is what this lesson is.

Why this matters

Embeddings are one of the conceptual handles that let you reason about what a model knows. Where in space does this token live? What is nearby? What direction encodes the relationship I care about? These are not idle questions. They have been used to audit models for bias (do gendered word pairs sit symmetrically or asymmetrically in the space?), to probe for hidden structure (what concepts does the space contain that nobody put there explicitly?), and to build practical retrieval systems used by millions of people every day.

The mental model to carry forward is small. Tokens live at locations in a high-dimensional space. Locations encode meaning, by virtue of how the locations were learned during training. When you read about a model embedding a document, or about vector databases, or about semantic search, that is the picture you should hold.

Going Deeper (optional)

Modern transformer embeddings have a refinement that word2vec did not: they are contextual. In word2vec, the word bank always has the same embedding, regardless of whether it is the financial institution or the side of a river. The transformer (Lesson 11) updates each token’s embedding through the attention layers based on the context around it, so the bank token in deposit at the bank and the bank token in fishing along the bank end up at different locations in the higher layers. This is what contextual embeddings means, and it is one of the things attention bought the field. The opening word2vec demonstrations were on static embeddings; everything in current systems is contextual.

If you want one more piece of vocabulary that comes up: the technical operation for comparing two embeddings (asking “how similar are these two locations?”) is usually cosine similarity, which measures the angle between the two vectors rather than the straight-line distance. The intuition is that direction in the space matters more than magnitude. You will see cosine similarity in many practical references.

What you have, what comes next

You now know what embeddings are at the level of mental model: locations in a high-dimensional space, learned from how words appear in training text, with geometric structure that mirrors semantic structure. The famous king − man + woman ≈ queen result is one example of what that mirroring buys; everyday semantic search is another.

Lesson 13 takes a step back from inside-the-model architecture and asks a practical question: how much text can the model hold and attend to at once? This is the context window, the second-to-last piece of the Part 2 puzzle. After that, Lesson 14 closes Part 2 by examining what changes (and what surprises everyone) when you scale the architecture up.

If You Want to Dig Deeper

For the original demonstration of word embedding arithmetic, including the king − man + woman ≈ queen result, the Mikolov et al. word2vec paper is short, readable, and full of examples. It is the historical anchor. Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013). Distributed representations of words and phrases and their compositionality (preprint). arXiv. https://doi.org/10.48550/arXiv.1310.4546

For the canonical open-source library that makes vector similarity search work at scale (used by virtually every modern retrieval-augmented generation system), the FAISS paper from Facebook AI Research is the standard technical reference. Johnson, J., Douze, M., & Jégou, H. (2021). Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, 7(3), 535–547. https://doi.org/10.1109/TBDATA.2019.2921572

For an accessible visual treatment of how word embeddings encode meaning geometrically, Jay Alammar’s Illustrated Word2vec essay (a companion to the Illustrated Transformer cited in Lesson 11) is the standard introduction. It walks through the math with diagrams at every step. Alammar, J. (2019, March 27). The illustrated Word2vec. https://jalammar.github.io/illustrated-word2vec/

Want to start from the beginning?

AI for Everybody — Lesson 1

What Is This Thing? The Search Box and the Strange New Box

Author Note. Grace Ann Hansen is an independent researcher and writer, and an MBA & PhD graduate student in health informatics and artificial intelligence. She is also a published author, a professional musician, a gymnastics coach, and a queer transgender woman living in Sioux Falls, South Dakota. She corrects all her papers and articles with Grammarly, because even though she has deep thoughts, she has shallow patience for punctuation. She uses Anthropic’s Claude in Research mode for source location and verification on cited factual claims; all interpretation, argument, and prose are her own. Correspondence concerning this article should be addressed to Grace Ann Hansen at grace@graceannhansen.com.

References

Johnson, J., Douze, M., & Jégou, H. (2021). Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, 7(3), 535–547. https://doi.org/10.1109/TBDATA.2019.2921572

Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013). Distributed representations of words and phrases and their compositionality (preprint). arXiv. https://doi.org/10.48550/arXiv.1310.4546

Nissim, M., van Noord, R., & van der Goot, R. (2020). Fair is better than sensational: Man is to doctor as woman is to doctor. Computational Linguistics, 46(2), 487–497. https://doi.org/10.1162/coli_a_00379