AI for Everybody - Lesson 7
How a Language Model Actually Works: Tokens: How a Computer Reads Words
Open a fresh chat window in ChatGPT or Claude and ask, “How many r’s are in the word strawberry?” There is a documented chance you will be told two. There is a smaller documented chance you will be told four. There is some chance you will be told three, which is the right answer. Then ask the same question again in a new conversation. The answer may change. The model is not stupid, and it has not lost interest. It is doing its job exactly as designed, and its job does not involve looking at letters.
Welcome to Part 2. Last week we closed Part 1 with the question of whether the system understands what it is doing. This week we open the engine. The first thing to notice, the smallest thing inside the machine, is the unit it actually reads. That unit is not a word. It is not a letter. It is something called a token, and almost every weird thing about how language models behave starts here.
