AI for Everybody - Lesson 9
How a Language Model Actually Works: Probability, Not Truth
Ask a chatbot which Supreme Court justice wrote the dissent in Bush v. Gore. There is a strong chance you will get a fluent, confident, citation-shaped answer. There is some chance the answer will be wrong. There is no chance the chatbot knows which case you are in, since the chatbot never asks itself that question.
Last week we opened the engine. The model’s entire job is to pick the next token from a probability distribution, then the next, then the next, until it stops. This week we look closely at what that probability is. The short version of this lesson is: the number is a guess about pattern-match against training data. It is not a guess about truth. The two are usually correlated. They are not the same. Where they pull apart, the system produces fluent confident error. That gap, owned at the mechanism level here, is the conceptual heart of hallucination, which we will name and unpack starting in Lesson 15.
What the model’s confidence actually measures
When the model assigns a high probability to a candidate next token, it is making a statement about its training data. Roughly: in text patterns like this one, that token tended to be the right next one. That is what the probability is. It is a measure of how confidently the system can fit the present input into the patterns it has previously seen. It is not a measure of whether the resulting sentence will be true.
For most everyday questions, those two things line up. Banana bread recipes tend to be true in the same way they tend to follow predictable patterns; the model’s high-probability completion use ripe bananas is both plausible and correct. The correlation is strong enough that the system feels useful most of the time. The trouble is what happens at the gap.
Ask the system a question where the truth requires looking up a specific fact, and the prediction loop does what it always does: it picks the most plausible next token. Bush v. Gore sounds like a case. The dissent slot, in case-citation patterns, gets filled by a justice’s name. The model has seen many case citations. It will produce a fluent name. Whether the specific name it produces is the right one depends on whether that exact pattern is strong in its training data and whether nothing else has displaced it. Sometimes the answer comes back right. Sometimes the answer comes back wrong, with the same fluency and the same surface confidence as the correct version.
The gap is structural. The loop was never built to check the answer. It was built to predict the next plausible token. Plausible and true are different categories, and the model is operating on the first.
Sampling: even confidence is variable
There is one more wrinkle worth holding in mind. The model does not always pick the highest-probability candidate. Most consumer chat products use sampling: the model rolls dice, weighted toward the high-probability tokens but with room for surprise. Two effects follow.
The first is that two identical prompts in two fresh sessions can produce two different answers. Sometimes both correct. Sometimes one right and one wrong. Sometimes both wrong in different ways. The variability is not noise in the engine. It is the engine working as designed; the design includes a roll of the dice at every step.
The second is that the system can produce a high-confidence-sounding answer that the model itself, on a different roll, would have rejected. There is no internal consistency check across sessions, since each session is a fresh loop starting from the same probability mechanism with a new sequence of rolls.
This means that a confident answer this morning and a contradicting confident answer this afternoon are both genuine outputs of the same machine. The morning answer is not more authentic than the afternoon answer. They are both the system doing its job. The system’s job does not include picking the one that is true.
Three consequences worth holding
A few specific patterns fall out of probability-not-truth that you will meet repeatedly for the rest of this course.
Fluent surface, hollow inside. When the system invents a fact, the invented fact is delivered in the same prose register as the real facts. The court case that does not exist will be cited with a plausible docket number, a plausible judge, and a plausible paragraph of legal analysis. The recipe that does not work will be written in the same warm tone as the recipe that does. The book that was never written will be attributed to the author who never wrote it with the same fluency the system uses for the author’s real books. Fluency is not a signal of correctness. Fluency is the constant background; correctness is the variable on top. This is the inversion of the cue most readers carry from talking with humans, where halting fluency often signals uncertainty. The system has no halting mode. It is fluent always.
Plausible specific is more dangerous than implausible specific. The system rarely invents wildly unlikely facts, since the training data does not support them. The system invents plausibly specific facts, since those are what the prediction loop produces from the probability distribution it learned. Justice O’Connor wrote the dissent is more likely to be the kind of error you get than Justice Snowflakeisplover wrote the dissent, since the first is statistically near the case-citation neighborhood and the second is not. This is exactly the kind of error that survives a quick read.
The model does not know that it does not know. The prediction loop produces a probability distribution every time, including times when it should not have an answer at all. If you ask the model what was on the cover of a magazine that was never published, the loop does not stop and say “I have no information about a magazine that does not exist.” It produces a fluent description of a plausible cover. The system has no internal I-don’t-know signal that consistently fires at the right times. There is research on getting models to flag when they are out of their depth, which we will touch on in Going Deeper. The honest summary is: the research helps, sometimes; it does not fix the underlying problem; the underlying problem is the loop itself.
Why this is the conceptual heart of hallucination
Most of the next several weeks build out of this lesson.
Lesson 15, Hallucination: Why It Lies With a Straight Face, names the phenomenon directly and gives you the practical verification habits that turn this lesson’s diagnosis into a usable defense against the surface-look of fluent confident error.
Lesson 26, How to Catch a Hallucination, hands you a three-move workflow for detecting fabricated content before it does damage.
Lesson 43, Hallucination: What It Is and What It Isn’t, returns to the topic with a working definition and names the two types (intrinsic and extrinsic) that the research literature distinguishes.
All three of those build on what this lesson hands you: the system does not check its work, since its work is to produce plausible tokens, and plausibility was never truth. Once you see the gap clearly, the rest of the hallucination conversation makes structural sense rather than feeling like a list of random failures.
Going Deeper (optional)
There is a small but important research literature on calibration and on whether models can be made to know when they don’t know. Three pieces are worth knowing exist.
Guo, Pleiss, Sun, and Weinberger (2017) showed that modern deep neural networks are systematically overconfident: when a network outputs 90 percent probability for a class, the actual frequency of that class being correct, across many such outputs, is often lower than 90 percent. The networks act more sure than they are. This was the canonical empirical result that calibration is a real problem and a default failure mode of large neural networks, not a quirk of any specific architecture. Calibration is the technical term for closing this gap; many techniques have been developed to improve it, none of them solving the fluent-confident-error problem in language generation since calibration is about the probability numbers, not about the truth of the sentences those numbers underlie.
Kadavath and colleagues at Anthropic (2022) ran the most-cited test of whether language models can assess their own confidence. The headline finding, in the paper’s careful wording, is that language models mostly know what they know. The qualifier carries the weight. The models are partially able to flag their own uncertainty when prompted to do so. The capability is imperfect; the failure pattern is not symmetric; and the practical implication is that asking the model are you sure? is a weak defense against confident error, since the system that produced the confident error is the same system you are asking to second-guess it.
The two findings together explain why “just have the model tell you when it doesn’t know” is not the fix that intuition suggests it should be. The system’s confidence number is uncalibrated. Its self-assessment is imperfect. Both are improving. Neither has closed the gap. The gap is structural, and the structural gap is what this lesson opens up at the conceptual level. Lesson 25 (When to Trust the Output) and Lesson 26 (How to Catch a Hallucination) take the practical workflows that follow from these limits.
What you have, what comes next
You now know what the model’s confidence actually measures, why confident wrongness is structural rather than accidental, and why two re-asks of the same prompt can disagree. You have met the term hallucination here as a conceptual seed; the term gets named, defined, and made useful in Lesson 15.
In Lesson 10, we pull the camera back one more step. The probabilities the model uses to predict the next token were not handed to it by anyone. They were learned, from a procedure called training, which is the central engine of every machine-learning system in the world. Next week’s lesson is what training actually is, in plain English, with no calculus. Once you understand training, the central principle from Lesson 4 (data is the model) will move from a phrase to a piece of machinery you can see working.
If You Want to Dig Deeper
For the technical-paper-but-readable treatment of why neural networks are systematically overconfident, Guo et al. (2017) is the canonical empirical reference and the source of the calibration vocabulary in the field. Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017). On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning (PMLR 70, pp. 1321–1330). https://doi.org/10.48550/arXiv.1706.04599
For the cleanest treatment of whether asking a model are you sure? works (the careful answer being: partially), Kadavath et al. (2022) is the relevant Anthropic study. Kadavath, S., Conerly, T., Askell, A., Henighan, T., Drain, D., Perez, E., Schiefer, N., Hatfield-Dodds, Z., DasSarma, N., Tran-Johnson, E., Johnston, S., El-Showk, S., Jones, A., Elhage, N., Hume, T., Chen, A., Bai, Y., Bowman, S., Fort, S., … Kaplan, J. (2022). Language models (mostly) know what they know (preprint). arXiv. https://doi.org/10.48550/arXiv.2207.05221
For the broader conceptual essay this lesson sits on top of, Murray Shanahan’s piece in Communications of the ACM remains the cleanest short explainer. Shanahan, M. (2024). Talking about large language models. Communications of the ACM, 67(2), 68–79. https://doi.org/10.48550/arXiv.2212.03551
Author Note:
Grace Ann Hansen is an independent researcher and writer, and an MBA & PhD graduate student in health informatics and artificial intelligence. She is also a published author, a professional musician, a gymnastics coach, and a queer transgender woman living in Sioux Falls, South Dakota. She corrects all her papers and articles with Grammarly, because even though she has deep thoughts, she has shallow patience for punctuation. She uses Anthropic’s Claude in Research mode for source location and verification on cited factual claims; all interpretation, argument, and prose are her own. Correspondence concerning this article should be addressed to Grace Ann Hansen at grace@graceannhansen.com.
References
Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017). On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning (PMLR 70, pp. 1321–1330). https://doi.org/10.48550/arXiv.1706.04599
Kadavath, S., Conerly, T., Askell, A., Henighan, T., Drain, D., Perez, E., Schiefer, N., Hatfield-Dodds, Z., DasSarma, N., Tran-Johnson, E., Johnston, S., El-Showk, S., Jones, A., Elhage, N., Hume, T., Chen, A., Bai, Y., Bowman, S., Fort, S., … Kaplan, J. (2022). Language models (mostly) know what they know (preprint). arXiv. https://doi.org/10.48550/arXiv.2207.05221
Shanahan, M. (2024). Talking about large language models. Communications of the ACM, 67(2), 68–79. https://doi.org/10.48550/arXiv.2212.03551
