AI agent memory: what to keep and what to throw away

AI agent memory is the problem of deciding what an agent still needs to know, given that it cannot keep everything. A context window is working space, not memory — everything in it is re-sent and re-paid for on every single call, and it fills up fastest exactly when a run is going well. The engineering is entirely about what to discard.

Four tiers, with different lifetimes

Treating memory as one bucket is the root of most agent failures. It’s four distinct things.

The instruction set. Your system prompt, the approval policy, the output contract. Constant across every call and every run. Put it at the very front so provider prompt caching can serve it cheaply.

Working state. The current goal, the plan, and what’s happened so far this run. Lives for one run, changes on every step, and is the tier that grows without bound.

Retrieved knowledge. Documents, rows, search results pulled in for a specific step. Should be ejected the moment the step that needed it is done — this is where most context bloat hides.

Durable facts. Things true across runs: the user’s timezone, their writing style, that they always want invoices in EUR. Small, slow-changing, and worth storing properly rather than re-deriving.

Conflating tiers two and three is the classic mistake — a 40-page document read at step 2 gets carried to step 20 for no reason.

The compaction trap

The obvious response to a filling window is to summarise older turns. It works, and it has a specific, nasty failure mode.

Summarisation is lossy in a way that isn’t uniform: it preserves narrative and drops specifics. Exactly the detail an agent needs later — the id it was given at step 3, the precise error string at step 7, the fact the user said “not the Berlin office” — is what a summariser treats as noise.

Three mitigations:

Never summarise tool results — evict them. A tool result is either still needed in full or not needed at all. A summary of a JSON payload is the worst of both.

Keep a verbatim tail. The last few turns stay exact. Compaction applies only to older material.

Pin facts out of the narrative. When the agent learns something durable mid-run — an id, a constraint, a correction — extract it into a small structured block that never gets compacted. That block survives; the prose around it doesn’t need to.

Context rot is real

Beyond a certain fill, quality degrades before you hit any hard limit. Models attend less reliably to the middle of a long context, and irrelevant material actively distracts.

The practical consequence: a fuller window is not a better-informed agent. We treat available context as a budget to stay well under, not a target to fill. If a step doesn’t need the document, the document shouldn’t be there.

This is also why aggressive retrieval beats generous stuffing. Fetching three relevant paragraphs outperforms including the whole file, on both cost and quality.

Durable memory needs a write policy

The tempting design is to let the agent decide what to remember. In practice that produces a store full of transient nonsense — “the user seems frustrated today” — that later runs treat as fact.

What works better is narrow and boring:

• Write on explicit correction. The user says “no, always use the EU region” — that’s a durable fact.
• Write on stable preference, observed repeatedly rather than once.
• Never write inferences about the person. Mood, intent and personality guesses age badly and skew every future run.
• Make it inspectable and editable. If a user can’t see and delete what’s been stored about them, you have a privacy problem as well as a correctness one.

Scope every durable fact to the tenant, and treat the store with the same isolation discipline as any other tenant data — it’s exactly the kind of table that needs deny-by-default policies.

What to do at each step

A compact policy that covers most agents:

1. Instruction set — always present, always first, cached.
2. Durable facts for this user — small, retrieved once at run start.
3. Working state — plan plus a verbatim tail of recent steps.
4. Retrieved knowledge — fetched for the current step, dropped after it.
5. Pinned facts — extracted, never compacted.

Then measure the thing that matters: not how much context you can carry, but how far into a long run quality holds. If step 15 is noticeably worse than step 3, the problem is almost always tier 3 never being evicted.

See also: how tool definitions stay out of the window entirely, and cutting the costs this creates.