Academy

Working with LLM APIs: first calls, tokens, and structured output

Your first call to a language-model API is simpler than it looks. Once you know a few basics — roles, tokens, temperature, and structured output — the rest falls into place.

The Felesh team 6 min

Using a language model through an API is, in practice, sending a message and getting a response. But there are a few basics that, once you know them, make you fluent from the very first call. Let’s go through them.

The anatomy of a call

Every call has three parts: the endpoint (where to send it), the API key (authentication), and the message payload (what you want). The response has a status code (for example 200 for success, 401 for an auth error, 429 for a rate limit) and a body containing the model’s reply.

Three parts out, a status and a body backA request box holds three parts: the endpoint POST /chat/completions, the API key sent as an Authorization Bearer header, and the payload with model, messages and limits. An arrow leads to a response box: status 200 OK with the reply in the body, 401 when the key is rejected, and 429 when there are too many calls.one call, in and outyou sendendpointPOST /chat/completionsAPI key, sent as AuthorizationBearer ••••payloadmodel · messages · limitsit returns200 OKreply in body401key rejected429too many calls
Every call sends three parts out and reads a status code and a body back.

Roles: system, user, assistant

The most important point is that the model has no memory. Each time, you have to send the whole history it needs yourself. That history is expressed as a list of messages with three roles:

  • system: the overall instruction that sets the personality and rules once, at the start.
  • user: the user’s message.
  • assistant: the model’s previous replies, which you send back to preserve conversational continuity.
Each call resends the whole listFour messages stack in one list: a system instruction, a user question about 2 plus 2, the assistant reply 2 plus 2 is 4, and a user message that says multiply that by 3. The assistant reply is highlighted because that points to it. The list flows into the model, which replies 12. The footer notes that that means 4 only because the assistant turn is in the list.each call resends the whole listsystemmath teacher, be patientuserWhat is 2 plus 2?assistant2 plus 2 is 4.userNow multiply that by 3.modelreads thewhole listreply12“that” means 4 only because the assistant turn is in the list
The model keeps no memory, so each call resends the whole list, and “that” resolves only because the assistant turn is in it.
messages = [
    {"role": "system", "content": "You are a math teacher. Be patient."},
    {"role": "user", "content": "What is 2 plus 2?"},
    {"role": "assistant", "content": "2 plus 2 is 4."},
    {"role": "user", "content": "Now multiply that by 3."},
]

Because the assistant message is in the history, the model understands that “that” means 4.

Tokens and cost

The model breaks text into units called tokens. A rule of thumb: in English each token is about three-quarters of a word; Persian uses more tokens (roughly one token per two characters). Cost is calculated per token, and input and output are priced separately. The context window is the maximum number of tokens the model can see at once; for modern models this ranges from about a hundred thousand to over a million tokens.

One request in two languages, billed in tokensTwo stacked bars per language. The English request is 8 input tokens and 12 output tokens, 20 in all. The same request in Persian is 18 input tokens and 14 output tokens, 32 in all. Input and output are counted separately, and the Persian bar is longer.one request, billed in tokens (illustrative)input: what you sendoutput: what it returnsEnglish8 in + 12 out = 20Persian18 in + 14 out = 32same request, more tokens in Persian
Billing counts the tokens you send and the tokens it returns, and the same request takes more tokens in Persian.

For long conversations that exceed the context window, you have two options: a sliding window (keep only the last few messages) or summarisation (compress the old messages into a summary).

Two ways to fit a long chat in the windowA table with three columns: option, what it keeps and what it loses. A sliding window keeps the last six messages and loses the early instructions unless they are sent again. A summary keeps a short summary of the first 40 messages and loses the exact wording and numbers. The summary is cheaper per call, but exact numbers are gone for good.two ways to fit a long chat in the windowoptionkeepslosessliding windowlast 6 messagesthe newest turnsearly instructions,unless re-sentsummarya short summary ofthe first 40 messagesexact wordingand numberscheaper per call; exact numbers are gone for good
Once the list outgrows the window, a sliding window drops the early instructions and a summary keeps the gist but not the exact words.

Temperature and a few key parameters

Temperature controls how random the output is, usually from 0 to 2. A value of 0 always picks the most likely option (good for coding and data extraction); 1 is balanced; and higher values are for creative writing. One rule: change either temperature or top_p, not both at once.

Another important parameter is the max output tokens. This caps the length of the reply — and if the model reaches it, the reply is cut off mid-sentence rather than summarised. So set it to match the task.

The output cap cuts the reply mid-sentenceA meter counts output tokens toward a cap of 12, marked illustrative. The reply about a yearly Pro plan fills three lines, one per step. On the third line it stops after the words prorated at the, at the cap, with a rust break mark and the note: cap reached, the reply stops here.the cap stops the reply mid-sentencemax output tokens = 12 (illustrative)12 of 12The Pro plan is billed yearly, so the firstpayment covers twelve months, and each seatadded mid-cycle is prorated at thecap reached: the reply stops here
Max output tokens is a hard cap, so when the model reaches it the reply stops mid-sentence rather than wrapping up.

Structured output

Often you want to process the reply as data, not free text. There are two ways to do this.

First, JSON mode: you ask the model to return output as JSON only. The catch is that you must mention the word JSON in the instruction.

response = client.chat.completions.create(
    model="...",
    messages=[
        {"role": "system", "content": "Return output as JSON only."},
        {"role": "user", "content": "Extract: iPhone 15, price $999"},
    ],
    response_format={"type": "json_object"},
)
data = json.loads(response.choices[0].message.content)

This guarantees valid JSON, but not that the structure is correct (such as all fields being present). Second, function calling: you define a function signature and the model is forced to return output matching that structure exactly — more precise and more reliable for structured data.

JSON mode checks the syntax, not the fieldsThe prompt Extract iPhone 15, price $999 goes to the model. Its output is valid JSON with a product field set to iPhone 15 and no price field. The first check, valid JSON, passes; the second, the price field, fails. Function calling is forced to match a schema you define.JSON mode checks the syntax, not the fieldsExtract: iPhone 15, price $999model outputproduct: “iPhone 15”price: not in outputvalid JSONprice fieldfunction calling is forced to match a schema you define
JSON mode guarantees parseable JSON, not that the fields you need are in it, such as a price.

Putting it together

A first API call is just these few pieces: messages with roles, an awareness of tokens and cost, a few parameters to control behaviour, and a way to get structured output. Once you have these basics in hand, the rest of the capabilities are just add-ons.