{"id":79144,"date":"2026-04-20T00:29:38","date_gmt":"2026-04-19T18:59:38","guid":{"rendered":"https:\/\/www.tothenew.com\/blog\/?p=79144"},"modified":"2026-04-22T11:23:12","modified_gmt":"2026-04-22T05:53:12","slug":"memory-in-ai-why-your-agent-forgets-everything-and-how-to-fix-it","status":"publish","type":"post","link":"https:\/\/www.tothenew.com\/blog\/memory-in-ai-why-your-agent-forgets-everything-and-how-to-fix-it\/","title":{"rendered":"Memory in AI: Why Your Agent Forgets Everything \u2014 And How to Fix It"},"content":{"rendered":"

Three months into building our DevOps AI agent, I gave a demo of it to the team. Checked pods, read logs, suggested fixes. Everyone was impressed. Then one engineer asked it: “Remember that ingress issue we sorted on Tuesday?”<\/p>\n

The agent had no idea what she was talking about.<\/p>\n

I had spent weeks on tool integration, prompt engineering, safety guardrails. I had not spent a single hour thinking about memory. And that gap made the whole thing feel like a toy instead of a tool.<\/p>\n

This post is what I figured out over the month that followed. Not a framework tutorial. Not a list of concepts. Just the actual problems I ran into, what I tried, and what ended up working.<\/p>\n

Why LLMs Forget \u2014 And Why It Actually Matters<\/h2>\n

The short version: LLMs have no state. Zero. Every time you call the API, the model starts completely blank \u2014 no knowledge of what you discussed a minute ago, no awareness of your infrastructure, nothing. The only “memory” it has is whatever you put in the prompt you send right now.<\/p>\n

That sounds abstract, so here is what it looks like in real life.<\/p>\n

You ask the agent to check your pods. It finds one crashing \u2014 api-deployment-xyz, OOMKilled. You then ask “what is causing this?” Without memory, the agent says: “I’d need more context \u2014 which pod are you referring to?”<\/p>\n

The exact conversation that triggered this post<\/p>\n

\u00a0 \u00a0 \u00a0 Me<\/strong>: Check all pods in the default namespace.<\/p>\n

\u00a0 \u00a0 \u00a0 Agent<\/strong>: [runs kubectl] Found api-xyz in CrashLoopBackOff.<\/p>\n

\u00a0 \u00a0 \u00a0 Me<\/strong>: Why is it crashing?<\/p>\n

\u00a0 \u00a0 \u00a0Agent<\/strong>: Could you clarify which pod you are referring to?<\/p>\n

The frustrating part is that this is not the model being bad at its job. It is doing exactly what it is designed to do. Each API call is isolated. The question “why is it crashing?” arrives with no context whatsoever, so the model asks for clarification. Makes sense from its perspective. From yours, it is maddening.<\/p>\n

And it gets worse the more you rely on the agent. Every session you re-explain your setup. Every incident you re-describe the symptoms. Every fix you re-establish the context. The agent never gets better at knowing your system because it starts from scratch every single time.<\/p>\n

The smarter the model, the more you notice how much context it is missing. A dumber model you expect to be useless. A smart one feels like it should know better.
\nI spent a week with a colleague trying to figure out why our agent kept asking basic questions about our infrastructure \u2014 questions it had already “answered” in previous sessions. The problem was not intelligence. It was memory. Two very different things.<\/p>\n

That second response is when I realised we had a problem<\/p>\n

The Different Kinds of Memory \u2014 How I Think About Them<\/h2>\n

When I started digging into this, I kept finding articles that listed memory types in a clean structured way: short-term, long-term, semantic, entity. Four types, four bullets, move on. That never quite clicked for me until I stopped thinking about memory types and started thinking about questions.<\/p>\n

Each type of memory answers a different question your agent might need to ask itself:<\/p>\n

 <\/p>\n

    \n
  • Short-term (In-context) –<\/strong> What did we just talk about? Lives in the prompt as a growing message list. Works instantly, zero setup. Vanishes the moment the session closes. Good enough for a demo, nowhere near good enough for production.<\/li>\n<\/ul>\n
      \n
    • Long-term (Persistent) –<\/strong> What did we talk about last week? Stored in a real database \u2014 SQLite, Postgres, Redis. Survives restarts. Same interface as short-term, just one line different. The first upgrade any serious agent needs.<\/li>\n<\/ul>\n
        \n
      • Semantic (Vector store) –<\/strong> What past conversations are relevant to this question? Messages stored as vector embeddings. Instead of dumping the entire history into every prompt, only the 5 most relevant past exchanges get retrieved. Handles months of history without exploding the context window.<\/li>\n<\/ul>\n
          \n
        • Entity \/ Knowledge graph <\/strong>-What are the fixed facts about our environment? Prod cluster IP, deployment schedule, service owners. Structured data that does not change turn-to-turn \u2014 stored in JSON or a knowledge graph, queried when needed.<\/li>\n<\/ul>\n

           <\/p>\n

          We went through all four stages over about six weeks. The jump from short-term to SQLite persistence was the biggest single improvement in agent usefulness we saw. Took maybe twenty minutes to implement.<\/p>\n

          How the Plumbing Actually Works<\/h2>\n

          Here is the thing about AI memory that took me embarrassingly long to understand: the model is not remembering anything. It physically cannot. What you are doing is feeding it the past conversation as text at the start of every new request.<\/p>\n

          The architecture, stripped back to its bones:<\/p>\n

          \"The

          The memory cycle on every .invoke() call<\/p><\/div>\n

           <\/p>\n

          That slot in the middle \u2014 where history gets injected into the prompt \u2014 is the MessagesPlaceholder in your LangChain prompt template. Every single LLM call, the full conversation history is dropped in there. The model reads it fresh, as if it just happened.<\/p>\n

          This is also why context window limits are not just a theoretical concern. Llama 3.1 supports around 128k tokens. A long conversation generates a lot of history. If you never trim or summarise it, you will eventually hit that ceiling, and older messages get quietly cut off. The semantic and summary memory types exist specifically because of this.<\/p>\n

           <\/p>\n

          The string that has to match \u2014 and nobody warns you<\/p>\n

          MessagesPlaceholder(variable_name=”chat_history”) \u2190 in your prompt<\/p>\n

          history_messages_key=”chat_history” \u2190 in RunnableWithMessageHistory<\/p>\n

           <\/p>\n

           <\/p>\n

          The Code \u2014 Three Stages of Memory<\/h2>\n

          Here is what the progression looks like in actual LangChain code. I am using Ollama because running models locally is free and I am not made of GPU budget.<\/p>\n

           <\/p>\n

          Stage 1 \u2014 Short-term (what you probably have)<\/strong><\/p>\n

          If you have followed any LangChain getting-started guide, this is likely where you are. It works within a session, and that is it.<\/p>\n

          <\/p>\n

          \"ShortTermMemory\"

          Short Term Memory<\/p><\/div>\n

           <\/p>\n

          Stage 2 \u2014 Persistent (the upgrade I should have done week one)<\/strong><\/p>\n

          One import. One changed function. Your agent now remembers across restarts.<\/p>\n

          \"LongTermMemory\"

          Long Term Memory<\/p><\/div>\n

           <\/p>\n

          Stage 3 \u2014 Semantic retrieval (when history gets long)<\/strong><\/p>\n

          After a few weeks of real usage, injecting the entire history into every prompt started causing problems. Prompts got long, responses got slower, context windows started straining. The fix: stop injecting everything, start retrieving only what is relevant.<\/p>\n

          \"SemanticMemory\"

          Semantic Memory<\/p><\/div>\n

           <\/p>\n

          What This Looks Like in a Real Conversation<\/h2>\n

          The demo that finally sold my team on memory was a simple three-turn conversation. No fancy setup, no long explanation \u2014 just this:<\/p>\n

          \"Three

          Three turns, no re-explaining<\/p><\/div>\n

          Turn 3 is two words. The agent patches the right pod with the right fix because it has read the full context from the previous two turns. No re-explaining, no re-specifying, no copy-pasting the pod name again.<\/p>\n

          Run that same conversation without memory and Turn 2 falls apart. The agent has no idea which pod you mean. You have to re-explain everything. The whole thing stops feeling like a conversation and starts feeling like filling out a form.<\/p>\n

           <\/p>\n

          Which Memory Strategy Should You Actually Use<\/h2>\n

          Honest answer: start simple and add complexity only when you feel the pain of not having it. Here is how I would sequence it:<\/p>\n

          \"Memory

          Memory strategy comparison<\/p><\/div>\n

           <\/p>\n

          What I would do if starting over today<\/strong><\/p>\n

          Day 1: SQLite. One line of code, zero thought required.<\/p>\n

          Week 3: ConversationSummaryBufferMemory when prompts get too long.<\/p>\n

          Month 2: Chroma vector store when you have real incident history to search.<\/p>\n

           <\/p>\n

          What worked for us: start with SQLite, deal with context length when it actually becomes a problem, and only touch vector stores once you have real history worth searching. We skipped straight to Chroma on one project and spent a week setting it up before we had enough data to make it useful.<\/p>\n

           <\/p>\n

          Where This Is All Going \u2014 My Honest Take<\/h2>\n

          Right now, AI memory is mostly a clever workaround. We are taking models with no persistent state and faking continuity by stuffing history into prompts. It works, but it is a bit like duct-taping a notepad to someone’s forehead every time they walk into a meeting.<\/p>\n

           <\/p>\n

          What worked for us: SQLite for persistence, summary buffer for context control, Chroma for semantic recall. Layer them in that order. Get the basics solid first.<\/p>\n

          The biggest shift in how the agent feels is not about which memory system you use. It is just about having any persistence at all. The jump from zero to SQLite is enormous. Everything after that is refinement.<\/p>\n

           <\/p>\n

          One Change, Right Now<\/h2>\n

          If your LangChain agent is using ChatMessageHistory() \u2014 and it probably is if you started from any tutorial \u2014 swap it out:<\/p>\n

          # Before<\/strong><\/p>\n

          session_history[session_id] = ChatMessageHistory()<\/p>\n

          # After<\/strong><\/p>\n

          SQLChatMessageHistory(session_id=session_id, connection=”sqlite:\/\/\/memory.db”)<\/p>\n

           <\/p>\n

          That is it. Ninety seconds of work. Your agent wakes up tomorrow knowing what happened today.<\/p>\n

          I genuinely wish someone had told me to do this on day one. Would have saved a fair amount of frustration and one particularly embarrassing demo.<\/p>\n

           <\/p>\n","protected":false},"excerpt":{"rendered":"

          Three months into building our DevOps AI agent, I gave a demo of it to the team. Checked pods, read logs, suggested fixes. Everyone was impressed. Then one engineer asked it: “Remember that ingress issue we sorted on Tuesday?” The agent had no idea what she was talking about. I had spent weeks on tool […]<\/p>\n","protected":false},"author":1833,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"iawp_total_views":8},"categories":[2348],"tags":[8540,1892,6263,8541,6408],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/79144"}],"collection":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/users\/1833"}],"replies":[{"embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/comments?post=79144"}],"version-history":[{"count":18,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/79144\/revisions"}],"predecessor-version":[{"id":79665,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/posts\/79144\/revisions\/79665"}],"wp:attachment":[{"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/media?parent=79144"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/categories?post=79144"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.tothenew.com\/blog\/wp-json\/wp\/v2\/tags?post=79144"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}