Academy

How LLM inference actually works: prefill vs decode

Text generation has two phases with very different behaviour: one compute-bound, one memory-bound. Understanding the difference explains why the KV cache exists and why decode is slow.

The Felesh team 4 min

Text generation in a language model doesn’t work the way it looks from the outside. Behind every request, the work splits into two phases with fundamentally different characteristics: prefill and decode. Understanding the difference between them explains why optimising inference is so difficult.

Why generation is sequential

Language models generate text token by token, with each new token depending on all the tokens before it. You cannot produce the next token before the current one, because each prediction changes the subsequent context. For this reason, generation is inherently sequential and cannot be parallelised.

Each token waits for the one before itThe prompt Why is the sky sits in a row of tokens. The model reads every token so far and adds one: blue, then because, then light. Each new token joins the row only after the one before it, so the steps cannot run in parallel.the context grows one token at a timeWhyistheskybluebecauselightread again on every stepmodelreads every token so far, adds oneone token per step
Each token joins the context before the next one can be predicted, so the steps cannot run in parallel.

Prefill: compute-bound

During the prefill phase, the model processes the entire input prompt at once, in parallel. All positions are computed simultaneously, producing large matrix multiplications that saturate the GPU’s compute cores. This phase is compute-bound: the workload is heavy computation, and core utilisation is very high — close to 95 percent in the source’s example.

Prefill runs the whole prompt in one passSix prompt tokens, Why is the sky blue ?, sit in a row above one matrix multiply that covers all six positions at once. The block lights up, core use rises to 95 percent in the example, and the text says the cores set the pace.prefill: the whole prompt, one passWhyistheskyblue?one matrix multiply covers all six positions at oncecore use95%compute-bound: the cores set the pace in this phase
The whole prompt goes through in one parallel pass, which keeps the cores close to fully busy.

Decode: memory-bandwidth-bound

During the decode phase, the model produces just one token at a time. The input shrinks to a small vector, and the matrix multiplications become tiny. The primary cost shifts elsewhere: at every step, the entire cache must be read from memory. This phase is memory-bandwidth-bound. Gigabytes of data move per step, but very little computation occurs on that data. In the source’s example, core utilisation in this phase drops to around 10 percent, while memory bandwidth nears saturation.

Decode reads the whole cache for one tokenFive cached tokens, Why is the sky blue, sit in memory as keys and values. Each decode step appends one token, because, and a read sweeps across the whole cache. In the example compute runs at 10 percent while memory bandwidth nears saturation.decode: every step reads the whole cachecached keys and values, one pair per tokenbecauseWhyistheskybluecompute10%memorymemory bandwidth nears saturation while the cores wait
Each decode step reads the whole cache from memory but does little arithmetic on it, so memory sets the pace.

Why the cache exists

If there were no cache, every step would have to recompute all past tokens from scratch — a hugely wasteful process. When a new token is added, the data required from past tokens for the attention computation has not changed, because the past does not depend on the future. It is therefore enough to compute those values once and store them, preventing them from being rebuilt at every subsequent step. This is the key-value cache (KV cache).

This cache carries a cost, growing linearly with the length of the text. In the source’s example, for a model with 32 layers and a hidden dimension of 4,096 at half precision, each token requires about 512 KB of cache. That translates to roughly 2 GB for a 4,096-token sequence — and that is for a single request. This is why the cache, rather than the raw size of the model, usually dictates how many concurrent users a system can serve.

Cache size from layers, width and precisionThe cache for one token is 32 layers times a hidden size of 4,096 times 2 for keys and values times 2 bytes per value: 512 KB. A 4,096-token request holds 512 KB times 4,096 tokens, about 2 GB, for one user.cache for one request, example modelper token32layers4,096hidden size2key, value2 bytesper value×××for a 4,096-token request512 KBper token, all layers×4,096tokens=2 GBa requestcache size, not model size, decides how many users fit
One 4,096-token request holds about 2 GB of cache, and that cache, not the model's size, limits how many users fit.

What this means in practice

In a typical request, the decode phase consumes most of the execution time, because each token requires a separate step. Since this phase is bound by memory rather than computation, the most effective way to raise throughput is to batch several requests together. By spreading the cost of reading the weights and the memory across multiple requests, the system’s total throughput increases considerably.

One read of the weights, four tokensTop: one request reads the weights and its own cache and yields one token. Bottom: four requests share one read of the weights, each with its own cache, and each receives a token from the same pass.one request: one token per passweights + this request’s cache1 tokenfour requests: four tokens per passweights once, each cacheuser 1user 2user 3user 4four tokens from one read of the weights
Batching shares one read of the weights across requests, so one memory pass yields a token for each request.