Academy

Run it for real

Moving from local prototype to production serving — how to select, adapt, and run models once cost, latency, and reliability actually matter.

The Felesh team 3 min

The prototype always works on your laptop. You pass a curated input to a large managed model, wait a few seconds, and watch a flawless response print to the console. It feels complete. Then it meets a live environment. The latency that felt acceptable locally becomes real friction under traffic. The cost per request, negligible in development, compounds into a serious operational expense. The model starts failing in ways the demo never exposed — dropping context, hallucinating details, or breaking structurally when handed messy, out-of-bounds user data.

One call, multiplied by a day of trafficOn a laptop one call gives one answer. In production, at an illustrative 50 calls a second, a day has 86,400 seconds, so the same call runs 4,320,000 times. A day-long bar fills in four six-hour blocks; under each block a running total reads 1,080,000, then 2,160,000, then 3,240,000, and 4,320,000 by midnight.on your laptopone call, one answerin production, illustrative50 calls every second50 × 86,400 seconds = 4,320,000 calls a day1,080,0002,160,0003,240,0004,320,00000:0006:0012:0018:0024:00every six hours adds 1,080,000 calls
One call on a laptop is nothing; at an illustrative 50 calls a second, the same call runs 4.3 million times a day.

This rung is production engineering. Moving an agentic system across the threshold means replacing generic assumptions with specific constraints. You stop reading benchmark leaderboards and start evaluating models against the exact operational profile of your task. A heavy reasoning step might need a flagship model; a fast routing layer might run better on a specialised small one. You adapt the model to the workload — deciding where to inject context and where to touch the weights. Then you serve that compute economically.

A router sends each request to its laneTwo customer messages reach a small model acting as router. “Where is order #3051?” goes down the lookup lane, which runs on the small model. “Billed twice: refund?” goes down the dispute lane, which runs on the flagship model. The flagship runs only for the dispute.where is order #3051?billed twice: refund?routersmall modelpicks lanelookupsmall modeldisputeflagshipthe flagship runs only where reasoning is needed
A small model reads every request and sends it down its own lane; only the request that needs reasoning reaches the flagship.

Production is an unforgiving filter. Cost, latency, and rigorous evaluation decide whether earlier architectural planning survives contact with reality. An elegant orchestration pattern is useless if running it consumes your margin, or if the time-to-first-token stretches past human patience. Getting to production means defining the exact bounds of acceptable performance and holding the deployment to them.

Candidates scored on the queue they will serveA table of three models against four kinds of input from an illustrative queue: a clean order lookup, typos mixed with Persian, JSON cut off mid-bracket, and a 40-page paste. The flagship holds on all four. The mid-size model breaks on the 40-page paste. The small model breaks on the cut-off JSON and on the 40-page paste.illustrative queueflagshipmid-sizesmallorder 3051, cleantypos + PersianJSON cut mid-bracket40-page pasteholdsholdsholdsholdsholdsholdsholdsholdsbreaksholdsbreaksbreaksa model that breaks on your queue is out, whatever it costs
Score each candidate on your own queue, not on a leaderboard: the cheap model that breaks on cut-off JSON is not the one you ship.