There are about 8.2 billion people alive. If every one of them sat down with a pencil and did one sum per second, in shifts, around the clock, they would need roughly seven and a half months to finish reading the conversation you have open right now.
Then they could start on the reply.
Nothing to install, nothing to sign up for. I just think this is worth knowing.Let me put a real machine on the table, because vague awe is worthless and a specific number is not. Llama 3.1 405B is the largest AI model whose insides are fully published: 405 billion parameters, 126 layers, 128 attention heads. Everything below uses that model, so you can go and check every figure yourself. The commercial ones you actually use do not publish their numbers, and nobody thinks they are smaller.
A parameter is just a number the model learned during training. To produce one token, roughly one word, the machine has to touch every single one of those parameters and do about two sums with each. Multiply it out and one word costs about 810 billion arithmetic operations.
That figure is meaningless written down, so here it is converted into people.
Now scale it up, which is where it stops being a fun fact and starts being slightly unnerving. A reply of about 600 words is roughly 800 tokens. Same crowd, same pencils: every person on Earth doing sums for just under a full day, each, to produce one moderately chatty answer about, say, whether you should repot your fig.
And …
the reply is the cheap half. Before the machine writes a single word, it has to read everything already in the conversation. On a 200,000-token window that comes to 162 quadrillion operations, which is the number I have been quietly building toward:
One person doing that job alone, at one sum per second, would need 5.1 billion years. The Earth is 4.5 billion years old. They would have needed to start before there was a planet to sit on, and they would still be going.
I checked this three times, because I did not believe it either. The arithmetic is deliberately conservative and I have not rounded anything in my favour.
The other number people find hard to picture is the context window, which is the plain-English idea that the model can only hold so much of your conversation in mind at once. It is measured in tokens, and a token is about three quarters of a word.
A 300-page paperback novel runs to roughly 90,000 words, which comes to about 120,000 tokens. So the whole thing converts cleanly into a unit anyone who owns a bookshelf can picture.
| What it is | Tokens | 300-page novels | Mid-size Stephen King |
|---|---|---|---|
| A long email thread | 10,000 | a long chapter | — |
| The chatbots of a few years ago | 8,000 | a short chapter | — |
| A standard window today | 200,000 | 1.7 | 0.9 |
| The large windows | 1,000,000 | 8.3 | 4.6 |
Using The Shining at roughly 165,000 words as the mid-size King. His actual range runs from about 110,000 words for Misery to something over 400,000 for IT, so treat that column as a rough shelf measurement rather than a precise one.
Worth sitting with for a second: when you paste a long document into a chat and ask a question about it, you are asking something to hold a couple of novels in its head, all at once, and then answer a question about page 140.
If you use one of the coding or agent tools, you will eventually see an indicator telling you the conversation is getting full. Different tools show it differently. The polite ones offer to compact the chat.
Here is what that word is covering for. The tool takes the bulk of everything said so far, writes a summary of it, throws away the original text, and carries on with the summary in its place. On a large context window, the amount being squeezed down is somewhere around 800,000 tokens, or three and a half mid-size Stephen King novels, compressed into about two pages.
Your transcript is fine. Nothing is deleted from your end, and you can scroll back through all of it. But the model can no longer see the detail, only the summary of the detail.
Which explains a thing you have probably noticed and blamed on yourself. A very long conversation starts giving vaguer, more generic answers than it was giving an hour earlier. It has not got bored of you. It is working from the abridged edition.
The practical move, and this is the only advice on this page: when a chat has gone very long and the answers start going soft, start a fresh one and paste in just the part that matters. You will usually get a sharper answer in less time than it takes to argue with the tired one. Pushing harder in the old chat is the instinct, and it is the wrong instinct.
You have probably read that these systems work one word at a time — that each word is chosen, then the whole machine runs again to choose the next one. It is the standard explanation, and the results never quite seemed to match it. Something that genuinely only knew the next word should not be able to land a joke.
It turns out the standard explanation is a simplification of one design, it was never the full story even for that design, and it is now flatly wrong for some models. Three separate things are going on.
Anthropic went looking inside their own model and found that when it writes rhyming poetry, it picks the rhyming word for the end of the line before it writes the beginning of the line. It then steers the sentence toward the word it already chose. So it writes one word at a time while thinking several words ahead, which is roughly what you do when you start a sentence you already know the end of.
There is a widely used trick called speculative decoding. A small fast model guesses the next several words, and the big expensive model checks all of those guesses in a single pass instead of generating them one by one. When the guesses are right, which they usually are, several words arrive for the price of one.
Google has shipped text models built on diffusion, the technique behind AI image generation. Instead of writing left to right, they start with a block of noise and sharpen the whole passage into focus over several passes, the way a photograph comes up in a developing tray. Google reports around 1,479 words-worth of text per second from one of them. There is no first word in that process. There is a blurry paragraph that becomes a sharp one.
So the instinct that "it can't just be one at a time" was the right instinct. It just had the wrong reason, and the real reason is considerably stranger.
This is the part where the honest answer is a shrug, and I think the shrug is the most interesting thing on the page.
Inside the model, your words are not words. Each one is converted into a long list of numbers — on Llama 405B, a list of 16,384 of them. You can think of that as a point in a space with 16,384 directions, the way a point on a map has two and a point in a room has three. Nobody can picture 16,384. The mathematics does not care.
Meaning lives in the geometry of that space. Related ideas sit near each other, and directions in the space turn out to correspond to concepts. And because the model needs to represent far more concepts than it has directions available, it does something researchers call superposition: it packs concepts in at angles that are almost, but not quite, separate from one another. Rather like over-filling a wardrobe by not quite closing the door.
Now the part that ought to be better known. These systems are not designed the way a bridge is designed. They are grown by training, and then studied afterwards like a natural object, by the people who grew them.
Anthropic built what amounts to a microscope for this, pointed it at one middle layer of one model, and pulled out 34 million distinct features — individual recognisable concepts, from specific landmarks to abstract things like sarcasm, or a bug in code. Then, in the same paper, they wrote that their set of features is incomplete, and that they have no rigorous way of checking whether those features faithfully describe what the model is actually doing.
That is the state of the art, published by one of the most capable labs on earth. They can build it, run it, sell it, and make it safer. They cannot fully open it up and tell you what it is doing in there, and they are straightforward about that in print, which is more than most industries manage.
Not so you can quote a big number at a dinner party, although the seven-and-a-half-months one does land well.
It is worth knowing because it quietly corrects two things people over 50 get told constantly. The first is that this is just autocomplete, a phrase that has no relationship to 810 billion operations and a machine that picks its rhyme before it starts the line. The second is the opposite error, that it is a mind with intentions. It is neither. It is an enormous amount of arithmetic in a space too large to picture, and that is genuinely a third thing.
And it makes you better at using it. Once you know that the window is measured in novels and that compaction is an abridgement, the tool stops being moody and starts being predictable.
If you think I have got a number wrong here, I would genuinely like to know. Every figure on this page is checkable and I would rather be corrected than quoted. — Scott
The standard approximation is two arithmetic operations per parameter, per token. On Llama 3.1 405B, that is about 810 billion operations for one token, which is roughly one word. Divided across everyone alive, about a hundred sums each. Commercial frontier models do not publish parameter counts, and are not thought to be smaller.
Because it compacted. When a conversation nears the model's limit, the tool summarizes the earlier part and continues with the summary in place of the original. The model stops seeing the detail. Starting a fresh chat and pasting in only what matters usually beats arguing with the old one.
No, or at least not in the way that phrase implies. Anthropic's own research found their model chooses a line's rhyming word before writing the line. Speculative decoding produces several words per pass. And diffusion text models such as Gemini Diffusion sharpen a whole block at once, with no first word at all.
Not completely, and they publish that fact. Anthropic extracted 34 million features from a single layer of one model and stated in the same paper that the set is incomplete and unverified. These systems are grown by training rather than designed part by part, so the insides have to be investigated afterwards.
No. You can use it well knowing none of it. Two bits are practically useful though: the window is finite and measured in novels, and compaction is an abridgement rather than a deletion. Those two facts explain most of the odd behaviour people run into.