Stage 1 · Measured · Act I
One word at a time
An AI model doesn’t write its whole reply at once. It writes one small piece, then the next, then the next. Watch a reply build itself.
Each glowing tile is one piece of text. Before a tile appears, the faint beams above it are the other pieces it could have been. The wider the beam, the more likely that choice.
Then the beams collapse onto one choice and it drops into the line. The soft tick you hear is that moment: a higher note when the choice was likely, a lower one when it was a long shot.
The pieces are called tokens. A token is often a whole word, sometimes part of one, sometimes a comma. The line of tokens is the reply so far, and each new token is chosen by looking at everything before it.
These odds come from GPT-2, a small model from 2019 that anyone can download and run. Today’s large models take the same step: read the text so far, give every possible next token a probability, pick one, repeat.
That is the whole trick, repeated once for every token of an answer. Everything below explains this one step: where the odds come from, and what it takes to make them good.
Stage 2 · Measured · Act I
70 years of trying
People have tried to build thinking machines for as long as there have been computers. For decades it went slowly, with two long winters. Then the computing poured into training models began to grow about four to five times every year.
Each dot is one AI model that mattered, from a database kept by the research group Epoch AI. Left to right is time. Height is how many calculations it took to train the model: each grid line up means 100,000 times more.
1950: Alan Turing asks whether machines can think. 1956: a summer workshop gives the field its name, artificial intelligence. 1958: the perceptron, one of the first machines that learned from examples instead of following written instructions.
Then two long dim troughs: the AI winters, when promises outran results and the money dried up. In between, in 1986, came backpropagation, the method that is still used to train every large model.
Before 2010 the dots climb slowly, about 1.5 times more computing per year. In 2012 a network called AlexNet won an image-recognition contest by a wide margin, and the climb steepened into a cliff.
2017: the Transformer, the design today’s large models are built on. 2020: GPT-3. 2022: ChatGPT puts a chat model in front of everyone. 2024: reasoning models that think before answering.
Since 2010, the computing used to train notable models has grown about four to five times every year; a straight line through these dots gives 4.3.
The dots on the floor never had their computing published. Most closed labs don’t publish it; the few closed models that float are Epoch’s outside estimates.
Stage 3 · Established · Act I
Rules vs learning
One way to make a computer tell cats from other animals is to write rules: pointy ears, whiskers. Another is to show it many labelled photos and let it find the pattern itself. The second way won.
Each dot is a photo of an animal, placed by two things you could check: how pointy the ears look, left to right, and how clearly whiskers show, bottom to top. Warm dots are cats; cool dots are not.
On the left, a programmer writes a rule: pointy ears and whiskers means cat. The warm box is where the rule says cat. Then photos arrive, and a fox breaks it. A red halo marks every photo the rules get wrong.
So the programmer adds an exception for foxes. Then a cat seen from behind breaks it, then a cat with folded ears. Each fix is another hand-written rule, and each one waits for something new to break it.
On the right, nobody writes rules. A small program is shown the same photos with the right answers and adjusts itself after each one. The glowing line it draws between cat and not-cat is called a boundary. It was learned from the examples.
At the end, cartoon cats turn up. The rules miss them, because nobody thought of cartoons. The learned boundary takes them in, because they were among its examples.
Real photos are not two numbers but millions of pixel values, and nobody can write rules for those. That is why learning from examples replaced programming by hand, and why everything after this is about learning.
Stage 4 · Established · Act II
A neuron is a weighted vote
A modern AI model is built from very simple parts. Each one takes some numbers in, multiplies each by how much it cares about it, adds them up and squashes the total. That is a neuron.
Here is the smallest working part of a language model, deciding one question: go to the beach? Three numbers come in on the left: how sunny it is, whether it is the weekend, how much homework is due.
Each input travels along a wire. The wire’s thickness is its weight: how much that input counts. Warm wires push towards yes, cool wires push towards no. In the middle, everything is added up.
The total then climbs a curve that squashes any number into the range 0 to 1, here how much yes. A big total ends near the top, a very negative one near the bottom.
Now turn one knob: homework counts twice as much against. The inputs have not changed, but the answer flips to no.
Turn another: the weekend counts three times as much. Back to yes. Nothing else changed; only two numbers.
Each weight is one number, called a parameter. This neuron has three. GPT-2 has about 124 million; the largest open models now have more than a trillion.
Stage 5 · Measured · Act II
Learning means nudging knobs
A network starts with random knobs and is useless. Training means showing it examples, measuring how wrong it is, and nudging every knob a little in the direction that makes it less wrong. Then repeating.
This small network is learning live, in your browser. It has 354 knobs, set at random to start. Its job: tell the warm spiral of dots from the cool one.
The glowing landscape shows how wrong the network is. Each spot on it is a different setting of the knobs; higher means more wrong. The ball is where the network’s knobs are right now.
Each step, the network guesses every dot, measures how wrong it was, and works out which way is downhill for every knob at once. Then it nudges them all a little that way, and the ball rolls. This is called gradient descent.
On the right is what the network believes about every point of the plane. As the error falls, its boundary wraps itself around the spirals. With sound on, the falling tone is the error going down.
The real landscape has 354 directions, one per knob, so this shows a two-direction slice of it that passes through the start and the end of this run. The ball is the network’s real position, projected onto that slice.
Large models are trained the same way: guess, measure the error, nudge every knob a little, repeat. The difference is scale: billions of knobs and trillions of examples.
Stage 6 · Measured · Act II
Tokens
Before a model can read anything, text is cut into pieces called tokens, and each token is turned into a number. The model never sees letters. It sees a list of numbers.
Here is a sentence being cut. Each tile is one token, and the small number under it is its ID: its place in the model’s list of all known tokens. Most common words are one token, with the space before them included.
Notice “strawberry”. It is a single tile, number 41236. The model never sees the three r’s inside it, which is one reason counting letters is surprisingly hard for language models.
Code is cut differently: brackets, symbols and short names each take a token, so code costs more tokens than plain English of the same length.
Emoji are rarer still. Each one is stored as several bytes, and the tokenizer, the program that does the cutting, splits it into two or three tiles that only make sense together.
Japanese, in this tokenizer, is the most expensive: half of these characters become two or three tokens each. Newer models have larger vocabularies and do much better, but the gap has not fully closed.
Tokens matter outside the model too. They are what AI companies count to measure how much text a model reads and writes, and what they charge for.
With GPT-2’s tokenizer, an English token is about three quarters of a word. Anthropic says Claude’s newest tokenizer cuts finer: a bit over half a word per token.
Stage 7 · Measured · Act II
Meaning as position
A token’s ID is just a label. The first thing a model does with it is swap it for a long list of numbers, and those numbers place the token somewhere in a space where related words sit close together.
This tile is the token “ king”. Watch it open up: every bar is one number, warm when it is positive, cool when it is negative. GPT-2 gives every token 768 of them. Nobody wrote these numbers; they were learned in training.
Treat each list of 768 numbers as a position, and every word becomes a point. Here are 538 real words from GPT-2. Places gather together; so do verbs, numbers and words from Python code, because words used in similar ways ended up with similar numbers.
Directions can carry meaning too. Take the step from “man” to “king” and take the same step from “woman”. On this squashed map the arrow lands among the words for women. In the full 768 numbers, the closest word to where it lands is “queen”.
It is not magic, and it is not always neat. Paris − France + Italy lands closest to Milan; Rome, the answer you would expect, comes second. The numbers capture how words are used in text, and Milan is used a lot like Paris.
One honest warning about this picture: squashing 768 numbers down to 3 keeps only about a tenth of how the words spread out. It is a loose map of a space with 768 directions, not the space itself.
Stage 8 · Measured · Act II
Attention
A word on its own is ambiguous. To understand it, a model has to look at the words around it. Attention is how each token decides which earlier tokens matter to it, and how much.
“The trophy didn’t fit in the suitcase because it was too big.” GPT-2 reads it one token at a time, left to right. As each token arrives it sends threads back to the tokens before it; a thicker, brighter thread means it is paying more attention to that one.
The dim tile at the start is a marker GPT-2 puts before every text.
Stop at “it”. What does it refer to? A person knows: the trophy, because it was too big. But GPT-2 has not read “big” yet. All it can do here is link “it” back to something.
This thread pattern, called an attention head (this one is head 3 of layer 4), does exactly that: its strongest pull is to “trophy”, the subject of the sentence.
That was one head of many. GPT-2 has 144, 12 in each of its 12 layers, and each one looks for something different. Some pull “it” toward “trophy”, others toward “suitcase”. None can settle the question yet: the clue has not arrived.
The clue is the last word. At “big”, of the words before it, this head (layer 7, head 8) picks “suitcase” most.
Now swap “big” for “small”: the same head reaches for “suitcase” harder still. GPT-2 small only half solves this puzzle, but any model that reads left to right meets the clue at the same place: the word that carries it.
Here is every thread of that head at once. Every token looks at itself and every token before it: 13 tokens make 91 pairs, 1,000 tokens make 500,500. Double the text and the work roughly quadruples. That is part of why long conversations cost more.
Stage 9 · Established · Act II
The stack
A model is the same building block, repeated. Each block lets the tokens look at each other, then reworks each one on its own. Stack enough of them and the numbers at the top know what should come next.
This glass slab is one block. The tokens of “Once upon a time, there was a” rise into it together. Round the last one, a ring of 768 rays: its list of 768 numbers from two stages back.
Inside, two steps. First attention: each token looks back at the others (the threads). Then a feed-forward step: each token’s numbers pass, on their own, through a big grid of learned weights (the lit weave). The tokens leave the block with changed numbers.
GPT-2 small has 12 of these blocks, stacked: the 12 layers of the last stage. Watch the rays change as the tokens rise. Above the last token is the word its numbers would point to if the stack stopped there.
Early on the guesses are nonsense, then “ lot” takes over in the middle (“there was a lot”), then the top blocks spread the odds again. After block 12 the guess is “ man” at 4%: exactly the odds from the very first stage. That is where they come from.
Bigger models are taller. GPT-3 has 96 blocks; Llama 3 405B has 126, and each of their blocks is also much wider. Nobody decides what each block does. Training shapes all of them at once.
Stage 10 · Measured · Act II
Counting parameters
Every weight in every grid you have seen so far is one adjustable number, a parameter. Here they are, counted: first a small model, exactly, then the big ones.
Each glowing speck here stands for 1,000 numbers inside GPT-2 small, a model from 2019. They fill its grids at their real shapes: the floor is the table that turns each token into its list of numbers, and the tower is its 12 layers, filling one by one.
Every one of these numbers was set by training, the knob-nudging of stage 5. Together they are everything the model knows. Count them all and you get 124,439,808.
One layer is lit. Its four grids add up to twelve squares, each 768 numbers wide and 768 deep. Twelve squares a layer, twelve layers: that is where most of the count comes from. The rest is the token table at the bottom.
Now change the scale. One speck becomes a million numbers. GPT-2 shrinks to a little cluster of 124 specks.
GPT-3, from 2020: 175 billion parameters. Each one takes two bytes of memory at the usual precision, so just holding it needs 350 gigabytes.
Llama 3 405B, from Meta in 2024: 405 billion. That is 810 gigabytes: more memory than ten of the AI chips in the next stage have between them.
DeepSeek-V3: 671 billion. It is built from many smaller expert blocks and uses only 37 billion of its numbers for each token, the brighter specks. More on that trick later.
And the big closed frontier models? Their makers don’t publish how many parameters they have, so they can only be shown as a haze with no edges. Any number you see quoted for them is a guess by someone outside.
Stage 11 · Established · Act II
It’s all matrix multiplication
Almost all of a model’s work, reading or writing, is one kind of arithmetic done over and over: multiplying grids of numbers. The chips that do it best were first built to draw video games.
On the left, a grid of numbers coming in (A); above, a grid of weights (B). Warm squares are positive, cool ones negative. Watch them slide through each other.
Wherever a number from A passes over a number from B, the two are multiplied and the result is added to the square underneath. Every square of the result (C) collects eight such products. That whole operation is called a matrix multiplication.
Notice that many squares work at the same moment, and none waits for its neighbour. That is the key: this kind of arithmetic can be split into many small jobs done side by side.
Pull back. That grid was one tensor core, a tiny unit built to do small matrix multiplications in one go. An NVIDIA H100 chip has 528 of them, in 132 blocks, all pulsing at once, with 80 gigabytes of fast memory beside them.
Eight of these chips sit together in one server…
…two servers in a rack…
…and racks fill a hall. This is Meta’s cluster of about 24,000 chips. The warm part is the 16,384 that trained Llama 3 405B together, for weeks.
Why chips made for games? A screen is millions of pixels, each needing the same small sum at the same time. Graphics chips (GPUs) were built for that: thousands of simple calculators side by side. It turned out to be the shape of AI’s arithmetic too.
Stage 12 · Measured · Act III
Training data
Those billions of numbers are set by reading. A model learns by predicting the next token of real text, trillions of times. The amounts are hard to picture, so here is a ruler.
A river of tokens, the text a model reads while it learns. This one is drawn from Meta’s recipe for Llama 3, one of the few large models whose data mix is published.
Half of it is general knowledge: web pages, articles, books. A quarter is maths and reasoning, about a sixth is code, and the last sliver is languages other than English. Each band’s height is its share.
Altogether Llama 3 read 15.6 trillion tokens. A number that big means nothing on its own, so here it is measured in people.
Take one person who reads eight hours a day, every day, for sixty years, at an average adult’s 238 words a minute. Counting three quarters of a word per token, that is about 3.3 billion tokens: this one bright speck.
Each speck in this cube is one such lifetime of reading. Llama 3’s training data fills about 4,700 of them.
The small cluster on the left is GPT-3’s data from 2020: about 90 lifetimes. The haze on the right is an estimate of all the usable text people have ever put online, around 300 trillion tokens. It is drawn blurry because nobody knows the number well.
Training sets have grown so fast that researchers at Epoch AI expect them to use up that whole supply some time between 2026 and 2032.
So models now also learn from text written by other models, called synthetic data, and from practice problems whose answers can be checked.
Stage 13 · Measured · Act III
The token math of training
How much work is it to read all that? There is a simple rule: six operations for every parameter, for every token read. Multiply it out and you get chips, weeks, electricity and money.
Build the sum as an object. Along the back, Llama 3 405B’s 405 billion parameters. Along the front, the 15.6 trillion tokens it read, to the same scale: it read almost 40 tokens for every parameter.
Each parameter does about six operations per token: two to read the token (warm sheets) and four to learn from it (cool). Stack six sheets and the volume is the total: about 3.8 × 1025 operations. Meta reports the same figure.
Now pour that work onto chips. The floor is the 16,384 H100 chips that trained it; the height is how long they ran. The chips manage about 41% of their peak speed in practice, so the run takes about 66 days.
That is about 26 million chip-hours (one chip for one hour), around 18 gigawatt-hours of electricity for the chips alone, and about $52 million at $2 per chip-hour to rent. Meta reports 30.84 million hours for the whole run: real runs lose time this ideal sum leaves out.
With a fixed budget, should you build a bigger model or train it on more text? The terrain shows how well a model learns (higher is better) for each size and budget. The bright crest is the best balance: about 20 tokens per parameter.
GPT-3 sits off the crest: too big for the 300 billion tokens it read. Chinchilla, built in 2022 to test this, sits right on it. Llama 3 405B is close: Meta fitted its own version of this law, which suggested 402 billion parameters for its budget, and chose 405.
Finally, two real runs side by side. Llama 3 405B: 30.84 million chip-hours. DeepSeek-V3: 2.788 million, on H800s, a close cousin of the H100, largely because it uses only 37 billion of its 671 billion parameters for each token.
DeepSeek put the cost at $5.576 million: only the rent for the final training run, at $2 a chip-hour. It leaves out the earlier experiments, the data, the staff and buying the chips.
Stage 14 · Established · Act III
From predictor to assistant
A model fresh from reading the internet only continues text. Here is one small open model at three moments of its training, asked the same question each time.
First snapshot. The model has only learned to predict the next token of web text. Asked “What is the capital of France?”, it says Paris, then keeps going: another question, another capital. Online, a line like that usually opens a quiz page, so that is what it writes.
The repeating colours are repeating tokens: it writes the same questions over and over. It never decides it is finished; a limit of 80 tokens cut it off. A model at this stage is called a base model. It knows a lot, but it isn’t trying to help anyone.
Second snapshot: the same model after more training on example conversations, a question followed by a good answer. Now the question arrives wrapped in markers that say whose turn it is. It answers in seven tokens, writes the marker that ends its turn, and stops.
Training on worked examples like this is called supervised fine-tuning. Notice the first line, “You are a helpful AI assistant”: a standing instruction placed in front of every conversation, called a system prompt.
Third snapshot: two replies to the same prompt branch apart. A judge picks the better one, and the model is nudged so the preferred kind of reply becomes more likely. Repeat that over a huge number of pairs and the model’s habits shift.
When the judges are people, this is called reinforcement learning from human feedback, RLHF. Anthropic says it also trains Claude against a written set of principles, with another model doing much of the judging: a method it calls Constitutional AI.
One question, three snapshots of one model. Pretraining gave it the knowledge; the later training shapes what it does with it: its behaviour, and its values.
Stage 15 · Established · Act III
Reasoning models
Some models can think before answering: they write out working first, often out of your sight, then the answer. It is still one token at a time.
Watch the stream split. Before any answer, a long lane of small tokens runs above it: working, written out step by step. Only then does the stream come back down and the answer arrive.
This lane is called thinking. It is made of ordinary tokens, predicted one at a time like any others. Products often hide it or show a summary, so these tiles are left blank.
Does it help? Here the thinking lane becomes the floor of a chart, with accuracy on a hard maths competition rising above it. The same base model scored 39.2% answering straight away, and 79.8% once trained to think at length.
A second test: one model was simply made to keep going, by appending the word “Wait” whenever it tried to stop. With 7,320 thinking tokens it climbed from 50% to 57%. Longer thinking tends to buy accuracy on hard problems.
How is the habit learned? With problems whose answers can be checked, like maths and code. The model tries, its answer is marked right or wrong, and right answers are rewarded. This is reinforcement learning.
Nobody writes the thinking for it. In one published run, the model taught itself to think longer as training went on, and its score on the maths test rose from 15.6% to 77.9%.
Thinking is not free. Every thinking token is billed as an output token, the same as the answer. In a real Claude Code session, a third of everything the model wrote was thinking. That is why harder questions take longer, and cost more.
Stage 16 · Measured · Act IV
Choosing the next word
So where do the odds come from, and how is one token picked? Here is a single step, slowed down. The ring is every token the model knows: 50,257 of them, one faint beam each.
The model reads “My favourite thing about the ocean is” and gives every token in the ring a score. Higher scores light up brighter. Most of the ring stays dark: thousands of tokens that would make no sense here.
Scores are not yet odds. A step called softmax turns them into probabilities that add up to exactly 100%. Now a beam’s height is its chance of being picked.
The temperature dial changes how adventurous the choice is. Turn it down, and the favourite takes almost everything. Turn it up, and the odds flatten out, so unusual tokens get more of a chance.
Now roll the die. The pointer spins and stops on a token, chosen according to the odds (at a temperature of 0.9, among the 40 likeliest), and it drops into the line. GPT-2 went on to write a whole sentence from it.
Same prompt, second roll. A different token. Nothing about the question changed; the die did.
Third roll. This is why asking a model the same question twice can give two different answers: the odds are fixed, but each pick is a roll.
Stage 17 · Measured · Act IV
The context window
A model doesn’t remember past conversations. Everything it can see of this one has to fit in a fixed stretch of recent tokens: your messages, its replies, any files and tool results.
This channel is everything the model can see right now, and it has a fixed length. As a session goes on, it fills from the left: standing instructions first, then your messages and its replies, then files it opens and results of tools it runs, each band its own colour.
Files and tool results are the big bands. A single file or search can take more room than a whole conversation of messages. This stretch of tokens is called the context window.
How big is it? On Anthropic’s current Claude models the window holds 1 million tokens: about 555,000 words, roughly 1,850 printed pages. Older Claude models held 200,000 tokens, about 500 pages.
Holding all those tokens costs memory. While writing, a model keeps a few notes for every token in the window, called the KV cache, so it doesn’t redo its work on them for each new word. The gauge shows that memory for Llama 3 70B, an open model.
What happens when the channel is full? New tokens still arrive, so the oldest ones fall out of the far end. The instructions stay pinned; the early conversation is simply gone.
The gentler option is compaction: the conversation so far is replaced by a short summary, and work carries on after it. In one real session, 791,533 tokens were compacted to 21,980.
Either way, what leaves the window is gone for the model. Anything it “remembers” from earlier in a long session is whatever survived in here.
Stage 18 · Measured · Act IV
What it costs to run
Every reply runs on a machine: a chip, and a block of memory holding the model’s numbers. Here is that machine, slowed right down, and the tricks that make it cheaper.
On the left, memory holding the model’s weights; on the right, the chip that does the maths; between them, a pipe. To produce anything, the weights have to flow through that pipe to the chip.
Reading your prompt is fast: one pass of the weights serves every prompt token at once, a wide burst. Writing is slow: every new token needs another full pass, so the reply comes out as a drip.
For a model of 8 billion parameters on one H100 chip, that is roughly 31,000 tokens a second read, and at most about 209 written.
Trick one: store each weight with fewer bits (binary digits). At 16 bits the shades are fine; at 8, then 4, they coarsen into steps, the block shrinks, and each pass is shorter. This is called quantization. The model gets slightly less precise, and much cheaper.
Trick two: split the model into expert blocks and use only a few per token. Here two of eight light up for each token, and only they cross the pipe. This is called a mixture of experts.
Mixtral uses 12.9 billion of its 46.7 billion parameters per token; DeepSeek-V3 uses 37 billion of 671 billion.
Trick three: train a small model to imitate a big one. The big model is the teacher; the small one learns from its answers. On the maths competition from stage 15, DeepSeek-R1 scores 79.8%; a 32-billion-parameter student taught by it scores 72.6%. This is distillation.
Trick four: don’t reread what hasn’t changed. Most of a long conversation is the same at every turn, so the work on it can be kept from last time (cached) and read back at a tenth of the price.
In a real Claude Code session, 98.8% of the tokens the model read came from the cache.
Together with better chips, this is why prices fall so fast. The price of reaching GPT-4’s level on PhD-level science questions fell about 40-fold a year; across tasks, Epoch AI measured falls from 9 to 900 times a year.
And energy: published 2025 figures for a typical text question to a chatbot range from about 0.24 to 0.34 watt-hours. Long documents, long thinking, images and video use much more.
Stage 19 · Measured · Act IV
Many models
There isn’t one AI. There are hundreds of models from dozens of labs, big and small, open and closed. Here they all are on one map, and the map shows why none of them is simply the best.
The line along the bottom is time, from early 2023 to now. Each bead that lights up is one model, placed at the month it came out.
Height is how capable it is: a single score from Epoch AI, an independent research group, that combines dozens of tests. Each chime you hear is a new overall record. The faint lines follow each big lab’s best so far.
Solid beads are open-weight models: the maker publishes the numbers inside them, so anyone can download and run them. Hollow rings are closed-weight models: you can only use them through the maker’s app or service. Claude is one of the hollow rings.
Now the open beads grow to their real size, the number of adjustable numbers (parameters) inside them, from about a billion to a few trillion. The hollow rings stay hazy because their makers don’t publish their size.
Look low and to the left: the smallest beads are models small enough for a laptop. The language model built into Apple’s phones has about 3 billion parameters. Small models answer fast, cost little and keep your data on your device, but they score lower.
Now the models with a published price slide sideways: left is cheap, right is expensive, in dollars per million tokens. Two models at almost the same height can sit far apart. Higher scores usually cost more, and older top models cost far more than newer ones that match them.
So the right model depends on the job: a hard research question, a quick reply on a phone, a million cheap summaries, or a model you must run on your own computers. That’s why so many exist.
Scores come from tests called benchmarks. Each line is one test: the best score any model had reached by that date. Watch them rise, then flatten as they near 100%. A full test can no longer tell the best models apart, so harder tests keep appearing.
Read scores with care. Test questions leak onto the web and into training data, and models can be tuned to a test rather than to the skill. On fresh maths questions written to match a famous test, some models did noticeably worse.
Some tests also contain wrong answers, so even a perfect model could not reach 100%.
Stage 20 · Established · Act IV
Seeing and hearing
So far everything in this story has been text. But you can show many models a photo, and some models can listen to sound. Neither is text. Watch how a picture and a sound become the same glowing tiles as words.
Here is a question, already cut into word tiles, and a photo: Earth seen from Apollo 17. The picture is 224 pixels wide and 224 tall, a size many image models are trained on.
Each soft chime is one row being cut. The picture is sliced into small squares of 16 by 16 pixels, called patches: 14 across and 14 down, so 196 patches. Even the empty black corners of space are patches.
Look at one patch up close. Its 256 pixels each have a red, a green and a blue value. Laid out in a line, that is 768 numbers.
A table of numbers learned in training turns those 768 numbers into 768 new ones: the patch’s list of numbers, like a word’s in stage 7, warm bars for positive values, cool for negative. These are the real numbers from an open image model called ViT.
Now each patch is a tile, and the tiles join the stream right after the words, in reading order. From here on, the model treats them like any other tiles: one question, then 196 pieces of picture.
Sound works the same way. What you hear now is drawn as a spectrogram: time runs left to right, pitch runs up, and brightness is loudness. Each note is a bright mark; the falling tone is a swoop.
The picture is cut every 10 milliseconds into 80 pitch bands, the recipe of an open speech model called Whisper. Three seconds make 300 slices; every two slices become one tile, so 150 tiles.
The sound’s tiles join the stream too, after the words that ask about it.
One line of tiles: words, picture and sound. Pictures are expensive: this one small photo took 196 tiles, many more than the words around it. Thirty seconds of speech would take 1,500.
Stage 21 · Established · Act V
Why code
Writing code is one of the things today’s models do best. Here is why: there is a lot of it, it is built from a few shapes that repeat, and a computer can check whether it works.
Watch a short Python function write itself, one token at a time. It is meant to take a list of numbers and give back the average of the even ones.
Each glowing arc joins an opening bracket to the bracket that closes it. Arcs sit inside arcs, like branches of a tree. The bow on the left joins the first line to the indented lines that belong to it. Listen: the tick climbs as the brackets nest.
Now the check. Three small tests call the function with lists whose answers are known. Two go red. The third passes, by luck.
What the tests printed comes back to the model as more tokens: “division by zero” and “2.0 != 3”. It reads them like any other text. The mistake is one token: “== 1” keeps the odd numbers. It should be “== 0”.
Fixed, run again: three green squares. That flash is a reward. In training, a model can be given a reward of 1 only when every test passes, and 0 otherwise. No person has to judge the answer. This is called reinforcement learning with verifiable rewards.
Five more short functions, behind it. Code looks abstract to people, but to a pattern learner it is very regular: the same opening line, brackets closing in the same order, the same words again and again. Of the 163 tokens in these six functions, 97 had already appeared earlier.
And there is a great deal of it to learn from. In Meta’s Llama 3, about 17% of the training text was code.
Stage 22 · Measured · Act VI
Hands: tools and the loop
On its own, a model only writes text. Put it inside a program that can act on what it writes, and that text can be a request: read this file, run these tests. The answers come back to the model as more text.
The glowing ball is the model, here Claude. The line of tiles running into it is everything the model can see: the conversation, files it has opened, results so far. That line is its context window (stage 17). Each step, it reads all of it again.
Then it writes. Sometimes it thinks first (the violet tiles). Here what it wrote is a request: run a command. It leaves as a packet for one of the ports on the ring.
The ring is not the model. It is the harness, an ordinary program on the computer, here Claude Code. It runs the command, and the output comes back as tokens at the end of the context. The model only writes the request and reads the result.
Repeat that, and text becomes action. A file the model has read is not something it remembers; it is something it looked up, there in the context. Looking things up instead of memorising them is what retrieval means, and web search is just another port.
Now a real Claude Code session, replayed: 3.9 hours of building an interactive web page. Claude was the model. Only its shape is kept: timing, token counts, tool names. No words, files or commands.
Each tone is one tool call, 545 of them. Most are commands: running scripts, tests and searches. The smaller balls are helpers: more copies of the model, each started with its own fresh context to do one job.
Look at the lengths. A helper can read hundreds of thousands of tokens, but only its short report comes back to the main context. Over the session that main context grew by about 34,000 tokens while the helpers did most of the work.
A context has a limit. Near the end the main one reached 791,533 tokens, and it was compacted: replaced by a summary of 21,980 tokens, and the work carried on from there.
Stage 23 · Modelled · Act VII
What’s inside, and what isn’t known
You have seen how a model is built and trained. Here is an honest admission: nobody, not even the model, can read most of what happens inside. Each glass layer is full of numbers that change with every token, and none of them comes with a label.
Researchers at Anthropic found a way to pick out patterns in those numbers. One lights up whenever the model reads about the Golden Gate Bridge, or sees a picture of it. A pattern like this is called a feature. It is spread over many numbers at once: this constellation.
Watch the sentence come in from below. The constellation stays dark until the words “Golden Gate Bridge” arrive, then lights. This feature was one of about 34 million they pulled out of a 2024 Claude model.
Then they turned it up by hand, changing the numbers directly instead of the prompt. For a day in 2024, that model, nicknamed Golden Gate Claude, brought the bridge into most of its answers, relevant or not. Changing behaviour this way is good evidence the feature is real.
In 2025 they traced whole chains of features, drawn here as light flowing up from the words, through the layers, to the word written. Asked for a rhyming couplet, the model read “He saw a carrot and had to grab it,”.
Before writing a word of the second line, features for “rhymes with it” and for carrots combined into a plan: end on “rabbit”, with “habit” as a runner-up. Then it wrote the line to land there: “His hunger was like a starving rabbit”. It plans ahead, at least sometimes.
To test it, they switched the “rabbit” plan off. The model wrote a different line and ended it on “habit”. The plan really was steering the words.
Adding 36 + 59 in its head, the model ran two paths side by side. One (orange) worked out the rough size: add something near 57, get a sum near 92. The other (blue) worked out only the last digit: 6 plus 9 ends in 5. Together: 95.
Ask the model how it did that sum, and it describes the carry-the-one method from school. What a model says about its own workings and what it actually did are not always the same thing.
And this is the honest part. These pictures came from short, simple prompts, and even there they capture only part of what happened. The researchers say the method gave a satisfying explanation for about one prompt in four they tried. The rest of the tower is still dark.
Stage 24 · Established · Act VII
What it isn’t
Now the edges of the stream: some honest limits of language models, and what is publicly known about one frontier model, Claude.
Everything the model can see is inside this frame: the context window. Here a conversation fills it, token by token.
When the conversation ends, the tokens are gone. Nothing in the model changes from talking to you: the next conversation starts with an empty window. If an app seems to remember you, it saved notes somewhere and puts them back into the window as text.
Now the stream as a timeline. A model learns from text written up to a point, the glass wall: its knowledge cutoff. After it, the world kept going without it, unless someone tells it what happened or it is given a search tool.
This is GPT-2 asked for the capital of Australia. Its brightest beam, the most likely answer, is “Sydney”: wrong. The right answer, “Canberra”, is only its fifth choice.
That is a hallucination: the model picks a likely-sounding token, not a checked fact, and it reads just as fluent and confident as a right answer. Bigger, better-trained models do this far less, but not never. Check what matters.
Safety training is drawn here as guard rails: after the first training, models are trained further to refuse harmful requests and to say when they are unsure. Most attempts to steer off the road are turned back. Not all: people keep finding gaps.
Here is the whole explanation, one tile per stage, coloured by how sure each one is. Green: measured, real model outputs or published figures. Blue: established, textbook facts. Orange: modelled, simplified drawings of published research. Violet would be metaphor, a picture like the guard rails; no whole stage needed it.
And about Claude: Anthropic has not published its size, its training data or the hardware it runs on, and the model cannot inspect its own numbers. What Claude says about itself comes from its training and what is in its context, not from looking inside. That is why this explanation used open models whenever it showed real numbers.
Stage 25 · Measured · Finale
Your turn: the next-token playground
Your turn. This is the step from the start, and you are driving it. Pick a prompt, watch the ring light up with the odds for the next token, set the temperature, and step.
The presets work straight away: their odds were computed in advance by GPT-2 small. Without the model itself, the die can only follow the few paths GPT-2 actually took.
Press Start to download the model and run it on your own device, on your graphics card if your browser allows it. Then type anything. Nothing you type leaves your browser.
The counter prices your tokens as if a large model like Claude had written them. This one is a tiny 2019-era model: frontier models are vastly larger and trained much further, but this step is the same.