Academy

How agents remember: memory and knowledge representation

A language model has no memory of its own and begins each conversation from scratch. What gives an agent memory is the layers built around the model.

The Felesh team 6 min

A language model remembers nothing on its own. Each time you ask it something, it sees only the text you give it in that moment; yesterday’s conversation does not exist for it. So when an agent appears to “remember,” that memory does not come from the model itself — it is an artefact of the layers built around it. Here is how those layers work.

Each call starts from zeroTwo calls to a model. Monday: the user says she prefers aisle seats and the model replies noted. Tuesday: she asks which seat she likes, but nothing was carried over, so the model says it does not know. Then the app copies Monday’s line into Tuesday’s prompt and the model answers: the aisle seat. The app has to resend the history on every call.call 1 · MondayyouI prefer aisle seats.modelreplyNoted, aisle.nothing carried overcall 2 · TuesdayMon: I prefer aisle seats.youWhich seat do I like?modelreplyI don’t know.The aisle seat.the app resends the history on every call
The model sees only what the call contains, so Tuesday's question is answered correctly only when the app sends Monday's line along with it.

Three kinds of memory

Agents usually need three kinds of memory, each with a different role.

Working memory is the shortest: the context of the immediate task at hand. It answers the question “what am I doing right now?” Its capacity is small and it holds only a few items at once.

Short-term memory is the history of the current conversation — what was said a few messages ago. This is what lets an agent understand what “multiply that by three” refers to. But this history is not endless; as the conversation grows, usually only its recent part is kept.

Long-term memory is persistent: preferences, learned facts, and things that should survive between sessions. It answers “what did the user prefer last month?”

Three kinds of memory side by sideA table of three memory kinds. Working memory answers what am I doing now, holds a seat for Laleh on Sunday, a few items at once, and lasts for this task. Short-term memory answers what was just said, holds that equals four seats, keeps the last messages only, and lasts for this conversation. Long-term memory answers what she prefers, holds Laleh likes aisle seats, is kept between sessions, and lasts across sessions.kindholds right nowlastsworkingwhat am I doing?seat for Laleh, Sundaya few items at oncethis taskshort-termwhat was said?‘that’ = 4 seatslast messages onlythis conversationlong-termher preference?Laleh: aisle seatskept between sessionsacross sessionsthe app puts all three back into the prompt on every call
Working memory holds the task, short-term holds this conversation, long-term holds what lasts between sessions; the app puts all three back into the prompt on each call.

The problem of remembering across long conversations

Because the model does not hold the history itself, you have to give it back on every call. But the history keeps getting longer and the context window is limited. There are two common ways to handle this. First, the sliding window: keep only the last few messages and drop the old ones — simple and cheap, but you lose the distant context. Second, summarisation: compress the old messages into a summary and keep that summary alongside the recent messages — more expensive, but it preserves the important context.

Sliding window versus summaryThe same six messages handled two ways. With a sliding window only the last three are kept and the first three, which said Laleh flies Sunday and takes only aisle seats, are dropped. With summarisation the first three become one short summary. Asked which seat she gets, the window version does not know because the aisle line was dropped, and the summary version answers aisle.sliding windowsummarisationI’m Laleh, flying SundayI’m Laleh, flying SundayI only take aisle seatsI only take aisle seatsCheck the baggage rulesCheck the baggage rulesTwo bags, 23 kg eachTwo bags, 23 kg eachAdd a vegetarian mealAdd a vegetarian mealWhich seat do I get?Which seat do I get?dropped, never resentsummaryLaleh, flies Sunday,aisle seats only,asked about baggage.Q: which seat do I get?unknown: aisle was droppedaisle: kept in the summarycheap, loses the distant pastcosts one more call, keeps gist
A sliding window drops the early messages for good, while a summary keeps their gist at the price of one more model call.

Knowledge representation: what long-term memory looks like

Long-term memory is not just a pile of text; knowledge has to be stored in a form you can retrieve. There are a few classical methods.

Semantic networks hold knowledge as nodes (concepts) and labelled edges (relations): a “penguin” is a “bird”, and a “bird” has “wings”. This structure allows inheritance; a penguin inherits a bird’s traits, except where it overrides them (a penguin does not fly).

Frames are templates with slots: the “bird” frame has slots like {wings: yes, fly: yes}, and the “penguin” frame inherits from it but changes the fly slot to “no”.

A frame inherits from its parentTwo frames. The penguin frame says is-a bird, wings not set, fly no. The bird frame says wings yes, fly yes. Asked whether it flies, the penguin answers no from its own slot. Asked about wings, its own slot is empty, so the lookup follows the is-a link to the bird frame and finds yes.frame: penguinframe: birdis ais-a: birdwings: (not set)fly: nowings: yesfly: yesask the penguin:fly? no, its own slot winswings? empty here, ask bird: yes
A penguin answers from its own slot when it has one and otherwise asks its parent, so it inherits wings and overrides flying.

First-order logic holds knowledge as facts and rules and allows inference: if “Socrates is human” and “every human is mortal”, then “Socrates is mortal”. It is precise, but it does not cope well with uncertainty.

Semantic search and the vector store

But for an agent’s memory, you often need something softer. Word-by-word search is brittle: if knowledge is stored under “machine learning”, a search for “AI” will not find it. The solution is semantic search. Each piece of text is turned into a vector that represents its meaning, and these vectors are stored in a vector store. At query time, the query itself is turned into a vector and the nearest vectors are retrieved — that is, the nearest meanings, not the nearest words. This is the very mechanism that lets an agent find similar past conversations, even when they were phrased differently.

A query lands next to its meaningFive stored notes sit as points in a meaning space: machine learning, neural networks, pasta recipe, flight rules, metro map. The query AI becomes a point between machine learning and neural networks. A keyword search finds 0 hits because no word is shared, while the nearest vectors are machine learning and neural networks.meaning spacemachine learningneural networkspasta recipeflight rulesmetro mapAIyou askAIkeyword search0 hits, words differnearest by meaningmachine learningneural networksfound by meaning, not by shared words
Both the stored notes and the query become vectors, so a search for AI lands next to machine learning even though they share no word.

Knowledge graphs: when relationships matter

Another method is the knowledge graph, which holds facts as triples: (subject, relation, object) — like (Milad Tower, located in, Tehran). The advantage of this structure is chain inference: from “the tower is in Tehran” and “Tehran is in Iran”, you can reach a new conclusion. Combining a knowledge graph with a language model gives the best of both worlds: precise facts from the graph, and natural language from the model.

Chain inference over a knowledge graphTwo stored facts, Milad Tower located in Tehran and Tehran located in Iran, form a chain of three nodes. Following both edges yields a derived fact, Milad Tower located in Iran, which the graph never stored. The language model then writes it as a sentence: the Milad Tower stands in Tehran, Iran.stored facts(Milad Tower, located in, Tehran)(Tehran, located in, Iran)Milad TowerTehranIranlocated inlocated inlocated in: derived, never storedmodel writes: The Milad Tower stands in Tehran, Iran.
Two stored triples chain into a third fact the graph never stored, and the model turns it into a sentence.

Putting it together

An agent’s memory is not one thing; it is a set of layers, each doing a different job: working memory for the immediate task, short-term for the current conversation, and long-term for lasting knowledge. And because the model itself holds nothing, designing these layers well is what turns an agent from a forgetful responder into a colleague that remembers.