Academy

PagedAttention and continuous batching: how one server answers more users

Two infrastructure tricks multiply the capacity of a language-model server: continuous batching and smart KV-cache management. Both come from one simple idea — don't waste resources.

The Felesh team 5 min

Serving a language model in production raises an immediate question: how many concurrent requests can you handle on the same hardware? The answer depends heavily on two infrastructure techniques, both rooted in a simple principle — never leave resources idle.

The problem with simple batching

The simplest serving method is static batching: waiting for a set of requests to gather, processing them synchronously, and only moving to the next batch when every request in the current one completes. The flaw here is that generation lengths vary. If one request finishes early while another continues, that compute slot sits idle until the longest sequence concludes. Expensive GPU cycles are spent waiting.

Static batching waits for the longest requestFour slots run four requests from one batch: A takes 40 tokens, B 180, C 80 and D 120. A, C and D finish early, and their slots sit idle, in rust, until B finishes at 180 tokens. Only then does the batch refill.slot 1slot 2slot 3slot 4A · 40B · 180C · 80D · 120idleidleidle060120180 tokensthe batch waits for B · 180 before any slot refills
Static batching holds every slot until the longest request finishes, so the short ones leave their slots idle.

Continuous batching

Continuous batching removes this waste. Rather than waiting for the entire batch to finish, the system ejects a request the moment it completes, immediately slotting a new request into its place. The batch remains full, and no compute slot sits idle. The impact on throughput is significant — often several times higher than static batching, particularly when generation lengths vary widely.

Freed slots take the next waiting requestFour slots run requests A, B, C and D as before, while E, F and G wait in a queue. E starts in slot 1 at token 40, when A finishes. F starts in slot 3 at token 80, when C finishes. G starts in slot 4 at token 120, when D finishes. Every slot runs until token 180 and none is idle.waitingE · 140F · 100G · 60slot 1slot 2slot 3slot 4A · 40B · 180C · 80D · 120E · 140F · 100G · 60060120180 tokensno slot sits idle from token 0 to token 180
Each slot takes the next waiting request the moment its own request finishes, so the batch stays full.

The KV-cache problem

Compute is only one bottleneck; the second is the KV cache. Every request requires memory to store its context, and this requirement scales with sequence length. The naive approach is to allocate a single, contiguous block of memory per request, sized for the maximum possible length. Because most requests finish well before reaching that maximum, a large portion of the reserved allocation sits empty. In practice, this method wastes roughly half the available memory.

Each request reserves the longest reply it could needThree requests each reserve a contiguous block of 2,048 token slots. Request 1 uses 412 of its slots, request 2 uses 96 and request 3 uses 1,310. The unused part of the first two reservations sits empty.request 1request 2request 3emptyempty412 used of 2,048 reserved96 used of 2,048 reserved1,310 used of 2,048 reserved
Each request reserves room for the longest possible reply; short replies leave most of their reservation empty.

PagedAttention

The solution borrows a core concept from operating systems: PagedAttention. Just as an OS partitions memory into small, non-contiguous pages, PagedAttention divides the KV cache into fixed-size blocks (typically sixteen tokens per block). A request claims only the blocks it actively needs, and those blocks do not need to be contiguous in memory. Memory fragmentation effectively disappears, and utilisation rises above ninety percent.

Blocks land in scattered slots, and a table maps themPhysical memory has 16 slots, and most are held by other requests. Request 1 has 40 tokens, which take three blocks of 16. Block 0 lands in slot 2, block 1 in slot 9 and block 2 in slot 14, which holds 8 of 16 tokens. Slot 13 is free. The block table records each mapping, so the blocks need not be next to each other.physical slotsrequest 1 · 40 tokens0123456789101112131415block 0 → slot 216 of 16block 1 → slot 916 of 16block 2 → slot 148 of 16any free slot; the table maps each block
Each block of a request goes into any free slot, and the block table records which slot holds each one.

This architecture yields a secondary benefit. If multiple requests share a common prefix — such as an identical system prompt — they can share the same physical blocks rather than maintaining duplicate copies. For workloads with heavy, static prompts, this reduces memory overhead significantly.

Two requests share the blocks of one system promptA system prompt of 48 tokens takes three blocks of 16. Request 1 and request 2 both map their first three logical blocks to physical slots 5, 6 and 7. Request 1 has one block of its own, in slot 10. Request 2 has two of its own, in slots 12 and 13. The three shared blocks are stored once and referenced twice.shared system prompt · 48 tokensrequest 1request 2slot 5slot 6slot 7slot 10slot 5slot 6slot 7slot 12slot 13stored once, referenced by both
Two requests with the same system prompt point at the same three physical blocks, so the shared text is stored once.

Why these two matter together

Continuous batching keeps compute slots saturated, while PagedAttention frees memory to fit more concurrent requests. The two mechanisms reinforce each other: reclaimed memory allows for a larger batch size, and a larger batch drives higher overall throughput. This compound effect is why both techniques form the foundation of modern language-model serving infrastructure.

Putting it together

Crucially, these optimisations do not alter the underlying model; they simply manage physical resources with greater discipline. The identical model, running on the same hardware, can serve significantly more users strictly because it ceases to waste compute and memory. In production, it is often this resource management — rather than the model itself — that dictates the fundamental unit economics of serving.