Academy

Where LLM serving costs actually go

If you open up the bill for serving a model, most of the cost is concentrated in one place. Understanding that concentration also clarifies the eternal 'build or buy' question.

The Felesh team Updated 4 min

When we think about the cost of serving a language model, we often accept a vague number. But opening up the bill reveals a clearer picture: most of the cost is concentrated in one place. Understanding this concentration not only helps control cost, but also clarifies the eternal question — “build or buy.”

Where the money goes

In serving a model, the bulk of the infrastructure cost goes to GPUs — often close to ninety percent. The rest goes to operations and people. This means any decision that raises GPU utilisation directly affects the bill. And this is exactly where the big saving opportunity lies.

One month’s bill, split by lineAn illustrative monthly bill of 33,300 units. GPU instances, on demand, are 30,000, which is 90 percent. Storage, network and logging are 2,300, and on-call and model operations are 1,000, together 10 percent. Any change that lifts GPU utilisation moves the largest line.one month’s bill, illustrative unitsGPU · 90%10%GPU instances, on demand30,000storage, network, logging2,300on-call and model operations1,000total per month33,300
Of this illustrative bill, 90% is GPU time, so any change that lifts GPU utilisation lands on the largest line.

An illustrative example

Here is an illustrative example, with round numbers rather than a measured result. Say the monthly GPU bill starts at 100. Then several optimisations stack on top of one another:

  • Quantisation: compressing the model brings the biggest saving.
  • Response caching: a fraction of requests are answered from a response cache, not by re-running the model.
  • Model routing: a large share of traffic is handed to a smaller model, and only the hard work reaches the large one.
  • Reserved capacity: a long-term commitment instead of on-demand payment brings a discount.

In this example, together they bring the bill from 100 down to about 23 — roughly a seventy-seven percent saving. The important point is that no single trick reaches this number alone; it’s the result of several optimisations stacking together.

Four levers stacked take the bar downAn illustrative GPU bill of 30,000 units a month, with four levers beside it: quantisation, response caching, model routing and reserved capacity. Each lever lights in turn and steps the bar down: to 18,000, then 12,000, 9,000, and finally 7,000. Stacked together the bill falls about 77 percent; no single lever reaches that alone.GPU cost per month, units30,00018,00012,0009,0007,000quantisationfewer bits per weightresponse cachinga repeat answer skips the GPUmodel routingeasy requests go to a small modelreserved capacitya term commitment, not pay-per-useall four stacked: about 77% less
Stack the four levers and the illustrative GPU bill falls from 30,000 to about 7,000; no single lever gets there alone.

The lesson of this example

The message here is not that you will reach exactly the same number. It is that serving cost is largely in your own hands. When you know where the money goes — to the GPU — and you understand which tricks raise its utilisation, you can shrink the bill meaningfully. This is a combined engineering effort, not a magic button.

A cache hit skips the GPU; a miss is routedThree requests enter: opening hours, a reply to rewrite, and a 40-page contract. The cache answers the first from a stored answer, so it never reaches the GPU. The other two miss the cache. The router sends the rewrite to the small model and the contract to the large one; only those misses run a model on the GPU.requestcacheon the GPUopening hours?rewrite a reply40-page contracthitmissmissrouteranswered, no GPUsmalllargea hit never touches the GPU; only misses run a model
A cache hit never reaches the GPU, and routing sends only the hard requests to the large model.

Build or buy

This picture also clarifies the “build or buy” decision. When cost is mostly tied to GPU utilisation, and control of that utilisation belongs to whoever runs the infrastructure, building in-house capability makes sense over the long term — especially at scale, where those few percentage points of saving turn into large numbers.

Buy or build, by who holds each leverA table for four questions, comparing buying from a provider with building in-house. The bill is 33,300 units, and 30,000 of them are GPU time. A provider runs the GPUs, sets utilisation and applies the levers on its terms. Building in-house puts those with your team, on your own traffic, so the saving at scale is yours.the 33,300-unit bill: 30,000 is GPU timebuya provider runs itbuildyour team runs itwho runs the GPUsthe provideryour teamwho sets utilisationthe provideryouwho applies the leversits terms onlyyou, own trafficwho captures the gainthe provideryou, at scale
Build or buy turns on who runs the GPUs, because whoever runs them sets the utilisation that drives the bill.

Putting it together

The cost of serving a language model is not a fixed fate; it is an engineering output. Most of the money goes to the GPU, and any optimisation that raises its utilisation brings direct savings. The key is to understand where the money goes and which tricks compound — and then, rather than accepting the bill, to design it.