IntegrAuth Academy
Identity, security and practical AI — free, from zero. Byte-sized lessons, real diagrams, hands-on labs, zero fluff. Two programs, each with its own exam and certificate.
English is hard? Your browser can translate this page into Tamil, Telugu, Kannada, Malayalam or Hindi. Tech words stay in English so you learn the real names.
How?
- Chrome on phone: ⋮ menu → Translate
- Chrome on computer: right-click → Translate to …
- Edge: the translate icon in the address bar
Inside one answer
Priya's help bot reads a 30-page handbook in a second or two. Then its short reply takes several seconds. Your goal: see what the model does for each token, and why that sets speed and cost.
🧪 Interactive lab — enable JavaScript to play with this one.
- The model makes one token per trip through all its layers.
- It reads your whole prompt in one trip. It writes one trip per token.
- Saved notes on earlier tokens make prompt caching possible.
One pass per token
To guess one token, the text goes through every layer of the model once. This trip is a forward pass.
Your prompt is all there at the start. So it goes through in one pass. But each new token needs the one before it. So writing takes one pass per token. That is why output is slower and costs more.
Looking back, and saving the work
In each pass, the newest token looks back at all earlier tokens. This step is attention.
The model keeps notes on earlier tokens, so it never redoes them. These notes are the KV cache. Prompt caching keeps these notes for a while, from minutes to hours. A new request with the same start can skip that work. But each note depends on every token before it. So one change near the top breaks the match.
Many experts, few awake
Some big models split part of each layer into small "experts." For each token, a router wakes only a few. This design is a mixture of experts (MoE). Speed follows the awake part. But all the experts must still fit in memory.
Now route tokens yourself.
🧪 Interactive lab — enable JavaScript to play with this one.
Go deeper: reading, writing and speed
Engineers call the reading step prefill, and the writing step decode. The reading pass also makes the first token. The wait before the first word is the time to first token (TTFT). It is mostly reading, plus any queue. After that, speed is counted in tokens per second.
Why is writing slow? For each pass, the chip must move the weights from memory to where it does the math. Doing that once per token takes time, even when the math is small. Servers answer many users in one batch. So each move serves many people.
A reasoning model writes its hidden thinking the same way, token by token. Those tokens are billed as output, even when you can't see them. So more thinking means a longer wait before the first word you see. See Choosing a model.
Go deeper: why the whole chat is sent again
The model remembers nothing between requests. So each follow-up sends the whole chat again. Some APIs store the chat for you, but the model still reads all of it again each turn. So on turn 10, the app sends turns 1 to 9 as well. The chat grows every turn, and so does the input you pay for. See Using AI through an API.
Prompt caching helps here. The start of the chat is the same as last time. So the provider can reuse its saved notes. Cached input costs only part of the normal price, often a tenth or less. Apps also cut or summarize old turns. See Context and memory.
Go deeper: what attention costs
Every token compares itself with every earlier token. So twice the text means about four times the comparisons when the model reads it. To save work, some models let certain layers look back only a fixed distance. The saved notes also grow with each token, for every chat a server holds. That is one reason long prompts are slower and can cost more.
Attention also spreads thin. A fact in the middle of a long text is easier to miss. See Tokens and context windows.
Go deeper: experts in numbers
As of October 2026, many large open-weight models use experts. Their makers publish the numbers:
- Mixtral 8x7B picks 2 of 8 experts in each layer. It has about 47 billion weights in total, and about 13 billion work on each token.
- DeepSeek-V3 has about 671 billion weights in total. About 37 billion work on each token.
- Qwen3-235B-A22B has 235 billion in total and 22 billion active. The name says so.
- DeepSeek-V4-Pro (April 2026) has about 1.6 trillion in total, and about 49 billion per token.
Most closed model makers do not say how their models are built.
An expert is not a topic specialist, like a "math expert." The router learns its picks in training. They change from token to token, and from layer to layer.
The size in a model's name is usually the total. For running a model yourself, check two numbers. Total size sets the memory you need. Active size sets the speed. See Open and closed models.
Open Transformer Explainer. Type a sentence and watch one pass. Or see a small model in 3D at LLM Visualization.