Agent Memory & State: Why Your Agent Keeps Forgetting
A supervisor and its specialists can route perfectly and still fail, if none of them remember what happened five minutes or five sessions ago.
Rohan’s multi-agent support system, built on Priya’s supervisor pattern from Week 9, had been live for two weeks when the same customer wrote in twice on the same ticket, four minutes apart. The first message asked about a delayed order. The refund specialist checked it, confirmed the delay, and replied. The second message just said, “okay so what do I do now.”
The supervisor routed it fresh, as a brand-new request. No order number. No prior context. The refund specialist, seeing an empty question with nothing to work with, asked the customer to repeat their order number, the same order number they had given four minutes earlier in the same conversation.
Rohan pulled the trace and felt the specific annoyance of watching a system fail at something a human would never fail at. Every agent in his system, the supervisor and all four specialists, was stateless by construction. Each request built a fresh prompt, ran its loop, returned an answer, and threw everything away. There was no thread connecting message one to message two, because nothing in the architecture was holding onto it.
He had built agents that could reason and act and even hand work off to each other. He had never built one that could remember.
“I kept treating every request like it was the agent’s first conversation ever,” Rohan thought. “Of course it looked confused. I never gave it anywhere to put what it already knew.”
He started listing out what actually needed remembering, and it split cleanly into two very different problems. Within a single conversation, the agent needed to hold onto the last few turns so a follow-up question like “what do I do now” resolved against the order they had just discussed. That was short-term memory, and it only needed to live as long as the conversation did.
But there was a second kind of forgetting that bothered him more. A returning customer who had complained about the same defect three times in the past two months looked, to his system, like a total stranger every single time. That was not a conversation-length problem. That needed something that survived across sessions entirely, a place to store facts about a customer that got retrieved when relevant, not replayed in full every time.
Two different memory problems, two different solutions. Rohan opened a blank file, and for the first time in this series, the first thing he built was not a loop or a tool. It was a place for the agent to keep things.
Two Kinds of Forgetting
Every agent built so far in this series, the Week 8 ReAct loop and the Week 9 supervisor, has been stateless by default. A request comes in, the loop or the routing tree runs, an answer goes out, and the conversation history that built up along the way disappears the moment the request finishes. That is fine for a single self-contained task. It falls apart the moment a real conversation spans more than one turn, or a real customer comes back more than once.
There are two genuinely different problems hiding under the word “memory,” and conflating them is where most agent memory implementations go wrong:
- Short-term memory: what was said earlier in this conversation. It needs to survive a few turns and then can safely be discarded.
- Long-term memory: facts worth keeping across conversations entirely, a customer's past complaints, a user's stated preferences, decisions made in prior sessions.
They need different storage, different retrieval strategies, and different lifetimes. Building one system to handle both usually means building something that does neither well.
Short-Term Memory: Holding the Current Conversation
The simplest version of short-term memory is a buffer: keep the last N turns and prepend them to every new prompt. It works, and it is also the first thing that breaks, because conversations that run long enough eventually blow past the context window, and every turn you keep costs tokens on every subsequent call.
Sliding window buffer
The cheapest fix is a fixed-size window: keep the most recent K turns, drop the rest. It is simple to reason about and cheap to run, but it means anything said more than K turns ago is gone completely, which is exactly what happened to Rohan’s returning customer if their first message falls outside the window.
public class SlidingWindowMemory {
private final Deque<Message> turns = new ArrayDeque<>();
private final int maxTurns;
public void add(Message turn) {
turns.addLast(turn);
while (turns.size() > maxTurns) {
turns.removeFirst();
}
}
public List<Message> asContext() {
return new ArrayList<>(turns);
}
}
Summarization memory
A better middle ground: once the buffer gets full, instead of dropping the oldest turn, summarize it into a running summary and keep that summary instead. Older detail gets compressed rather than deleted outright, so the agent still knows an order number was mentioned even after the exact original wording is gone.
public class SummarizingMemory {
private String runningSummary = "";
private final Deque<Message> recentTurns = new ArrayDeque<>();
private final int maxRecent;
public void add(Message turn) {
recentTurns.addLast(turn);
if (recentTurns.size() > maxRecent) {
Message oldest = recentTurns.removeFirst();
runningSummary = summarizerModel.updateSummary(runningSummary, oldest);
}
}
public String contextPrefix() {
return "Summary of earlier conversation: " + runningSummary;
}
}
The tradeoff is an extra model call every time a turn ages out of the window, which adds latency and cost on a slow drip. For most support and assistant use cases that cost is worth it, since losing a customer’s order number mid-conversation is worse than a small summarization bill.
Long-Term Memory: Remembering Across Sessions
Short-term memory answers “what did we just talk about.” Long-term memory answers a completely different question: “what do I already know about this person, and is any of it relevant right now.” The architecture looks a lot more like the RAG pipeline from Week 4 and Week 5 than like a conversation buffer, because retrieval, not replay, is the right tool here.
The pattern: extract, store, retrieve
- Extract: after a conversation ends, pull out durable facts worth remembering, not the full transcript. "Customer reported a cracked screen on order 48213" is worth keeping. Small talk is not.
- Store: embed each fact and store it in a vector store, keyed to the customer, the same Pinecone setup from Week 5 but storing facts about people instead of product documentation.
- Retrieve: at the start of a new conversation, run a similarity search against that customer's stored facts using the current message as the query, and only pull in what is actually relevant.
That last point is what keeps long-term memory from becoming a second context bloat problem. You are not replaying every past interaction. You are retrieving the two or three facts that matter for the question in front of you right now.
public List<String> retrieveRelevantMemories(String customerId, String currentQuery) {
float[] queryEmbedding = embeddingModel.embed(currentQuery);
return vectorStore.similaritySearch(
SearchRequest.builder()
.query(currentQuery)
.filterExpression("customerId == '" + customerId + "'")
.topK(3)
.build()
).stream().map(Document::getContent).toList();
}
Wiring Both Into an Agent
Rohan’s fix combined both layers ahead of the supervisor from Week 9. Before routing a request anywhere, the system now builds a context block from two sources: the sliding window or summary for the current conversation, and a small set of retrieved facts from the customer’s long-term memory. Only after that context is assembled does the supervisor decide where to route.
public String handleMessage(String customerId, String conversationId, String message) {
ConversationMemory shortTerm = memoryStore.getConversation(conversationId);
shortTerm.add(new UserMessage(message));
List<String> longTermFacts = longTermMemory
.retrieveRelevantMemories(customerId, message);
String context = shortTerm.contextPrefix()
+ "\nKnown history: " + String.join("; ", longTermFacts);
String answer = supervisor.runSupervisor(message, context);
shortTerm.add(new AssistantMessage(answer));
return answer;
}
Failure Modes
Unbounded growth
A sliding window with no cap, or a long-term store with no expiry policy, grows forever. Set explicit limits on both: a max turn count for short-term memory, and either a time-based expiry or a periodic consolidation pass for long-term facts, so old, superseded information does not sit indefinitely.
Stale or contradictory facts
Long-term memory can retrieve a fact that used to be true and no longer is, a shipping address from two moves ago, a preference the customer has since changed. Store a timestamp with every fact and prefer more recent entries when two retrieved facts conflict, rather than presenting both as equally current.
Retrieval mismatch
Similarity search can pull in a fact that is topically related but not actually useful, the same failure mode RAG has always had. Keep topK small, three to five facts, and let the specialist agent’s own reasoning decide whether a retrieved fact is relevant rather than blindly injecting everything retrieved.
Cross-customer leakage
The single most damaging failure mode in this category: a retrieval filter that is missing or misconfigured can surface one customer’s facts inside another customer’s conversation. Always filter retrieval by a hard identity boundary, never rely on similarity ranking alone to keep customers’ data separated.
Cost of maintaining memory
Summarization calls, embedding calls, and extraction calls all add up on top of the agent calls already discussed in Weeks 8 and 9. Batch the extract-and-store step to run after a conversation ends rather than on every turn, since long-term memory does not need to update in real time the way short-term memory does.
Implementing With Spring AI and LangChain
Spring AI ships a ChatMemory abstraction that covers the short-term case directly, with a message window implementation close to the sliding window pattern above, plus the ability to plug in a custom repository for persistence across requests.
@Bean
ChatMemory chatMemory(ChatMemoryRepository repository) {
return MessageWindowChatMemory.builder()
.chatMemoryRepository(repository)
.maxMessages(20)
.build();
}
ChatClient.builder(model)
.defaultAdvisors(MessageChatMemoryAdvisor.builder(chatMemory).build())
.build();
For long-term memory, Spring AI does not provide a dedicated abstraction yet, so the extract-store-retrieve pattern above, built on top of the same VectorStore interface already used for RAG, is the natural fit within a Spring Boot codebase. LangChain’s Python ecosystem has both layers built in directly, ConversationSummaryMemory for the short-term case and a VectorStoreRetrieverMemory for the long-term case.
from langchain.memory import ConversationSummaryMemory, VectorStoreRetrieverMemory
short_term = ConversationSummaryMemory(llm=llm, max_token_limit=500)
long_term = VectorStoreRetrieverMemory(
retriever=vectorstore.as_retriever(search_kwargs={"k": 3}),
)
long_term.save_context(
{"input": "Customer reported cracked screen"},
{"output": "Logged under order 48213"},
)
Where This Leaves Rohan
Rohan added a sliding window for in-conversation context and a small vector store of extracted facts per customer, retrieved only when relevant, with a strict customer ID filter on every query. The same customer wrote in a third time two weeks later about the same order, and this time the refund specialist opened with the order number already in hand instead of asking for it again. The fix was not a smarter model. It was giving the system somewhere to put what it already knew, and being disciplined about what actually belonged in short-term memory versus what belonged in long-term storage.
Building agents that need to remember things?
Join the CodeKerdos Spring Boot + AI Bootcamp. Weekends, hands-on, built for working Java developers who want to ship real agentic systems, not just toy demos.
codekerdos.in | Follow along with Week 11 next weekend