Agent Memory: How AI Agents Remember, Forget, and Learn on the Job

Nish · July 7, 2026

Agents   Fundamentals
⏱️ 41 min read

Every morning, your coding agent shows up brilliant and amnesiac. It can refactor a module, chase a bug through five files, and quote the standard library from memory, and tomorrow it will remember none of it: not the bug, not the fix, not the thing you told it three times about how deploys work here. Andrej Karpathy put the complaint bluntly in October 2025: “They don’t have continual learning. You can’t just tell them something and they’ll remember it.” This post is a field guide to the two connected problems hiding in that sentence, remembering (keeping state across sessions) and learning (getting better with experience), and to the machinery being built for both: retrieval scores and memory pipelines, RL-trained librarians, a catastrophic-forgetting experiment you can run in numpy, the elastic weight consolidation derivation in full, and the new architectures that let weights learn at inference time.

Table of Contents

TL;DR

  • An agent’s memory problem is really two problems: state (what happened, what was decided, what the user prefers) and skill (getting better at the job). Neither can currently be written into the model’s weights, which are frozen the moment training ends.
  • The context window is working memory: real learning happens there (in-context learning is provably gradient-descent-like), but it is rented by the token, evaporates at session end, and degrades as it fills. On the NoLiMa benchmark, GPT-4o falls from 99.3% to 69.7% well within its advertised window; practitioners now call this context rot.
  • Explicit memory systems converged on one shape: extract facts from the conversation, score stored memories by recency, importance, and relevance (a formula essentially unchanged since Generative Agents in 2023, with its 0.995-per-hour decay), and consolidate new information with ADD / UPDATE / DELETE / NOOP operations. We build a working one in about sixty lines of Python.
  • The write policy is becoming learned rather than prompted: Memory-R1 trains its memory manager with PPO/GRPO from only 152 QA pairs and beats Mem0’s pipeline on the LOCOMO benchmark (F1 27.3 → 35.7 on the same backbone).
  • Writing memories into the weights collides with catastrophic forgetting. In this post’s numpy experiment, a network at 90% on task A drops to 30% after learning task B. EWC (derived in full below) and replay both help, but only along a stability-plasticity frontier; a 200-example replay buffer competes with clever curvature penalties, echoing the brain’s own replay-based consolidation.
  • The research frontier is melting memory into the architecture: TTT, Titans, and ATLAS make the weights themselves learn during inference via surprise-gated gradient updates, and Meta’s sparse memory finetuning cuts forgetting from an 89% score drop to 11%.
  • Whether continual learning is the bottleneck to broadly capable agents is genuinely contested: Dwarkesh Patel and Karpathy say yes, Dario Amodei says maybe not. Either way, the thing that works today is unglamorous: curated notes, scored retrieval, and scheduled consolidation.

The amnesiac colleague

The strangest property of LLM agents is not what they know but what they cannot retain. A model that read most of the internet cannot remember that your team switched to uv last month unless someone pastes that fact into every single session. Karpathy’s framing is that this is not a bug in an otherwise human-like mind but a structural difference in kind: “We’re not building animals. We’re building ghosts or spirits,” creatures summoned fully formed from pretraining data rather than raised by experience.

Dwarkesh Patel built the strongest version of the complaint in his June 2025 essay “Why I don’t think AGI is right around the corner”: “The lack of continual learning is a huge huge problem.” His argument is worth internalizing because it relocates the value of human colleagues away from raw intelligence: “The reason humans are so useful is not mainly their raw intelligence. It’s their ability to build up context, interrogate their own failures, and pick up small improvements and efficiencies as they practice a task.” A new hire is not impressive on day one; they are impressive on day ninety, and everything between day one and day ninety is memory doing its work.

It helps to split the complaint into its two distinct halves, because they have different solutions:

  1. The state problem. The agent forgets facts: what you decided yesterday, which approaches already failed, how this repo’s deploy actually works. This is a storage-and-retrieval problem, and the industry has made real progress on it.
  2. The skill problem. The agent doesn’t get better: the thousandth code review is performed with exactly the competence of the first. This is a learning problem, and it is much harder, because the natural place to put skill (the weights) fights back when you write to it.

The rest of this post walks the machinery for both, from the context window outward to external stores, and then down into the weights.

Four memories, borrowed from psychology

Cognitive science spent a century developing vocabulary that maps almost embarrassingly well onto agent design, and the current wave of systems borrows it explicitly. Atkinson and Shiffrin’s 1968 “modal model” separated short-term from long-term stores and, crucially, emphasized the control processes (rehearsal, encoding, retrieval strategies) that move information between them. Tulving then split long-term memory itself: episodic memory for personally experienced events situated in time (1972), joined by semantic memory for general knowledge, and later procedural memory for skills, in his 1985 three-system taxonomy. An agent has natural analogues of each: session transcripts and reflections are episodic, extracted facts and notes are semantic, and saved skills, playbooks, and code snippets are procedural, all orbiting a working memory that is just the context window.

Anatomy of an agent memory system. New experience flows into a blue context window box (working memory, holding instructions and retrieved memories), which exchanges with three green external stores on the right: an episodic store of session logs, a semantic store of extracted facts, and a procedural store of skills. An orange consolidate arrow (extract, then ADD/UPDATE/DELETE/NOOP) runs from context to stores; a retrieve arrow scored by recency, importance, and relevance runs back. A dashed orange sleep-time loop rewrites the stores. Below, a purple slab of parametric memory (the weights) supports everything, written only by slow gradient descent. A side annotation maps the split onto McClelland et al.'s complementary learning systems: hippocampus as the explicit stores, neocortex as the weights.
Map 1: the anatomy of an agent's memory. Working memory is the context window; episodic, semantic, and procedural stores live outside it and trade information through retrieval and consolidation; the weights underneath change only through slow gradient descent. The layout mirrors the brain's complementary learning systems: a fast learner for new episodes and a slow statistical learner for structure.

The most load-bearing borrowing is the last one in that caption. In 1995, McClelland, McNaughton, and O’Reilly asked why the brain bothers with two learning systems, a hippocampus that memorizes episodes quickly and a neocortex that learns structure slowly, and their answer came straight from connectionist networks: a single distributed learner that absorbs new information quickly overwrites what it already knows. The hippocampus exists, in their account, precisely so that new memories can be stored fast and then “interleaved” into the neocortex gradually, without catastrophic interference. Keep that in mind for the second half of this post; the entire modern agent-memory stack is an engineering rediscovery of it.

Working memory: rented, tiny, and rotting

The context window is genuinely a form of memory, and a remarkable one. In-context learning is not a metaphor for learning: von Oswald et al. (2023) showed that a single linear self-attention layer can implement exactly one step of gradient descent on a regression loss over the examples in context, and that transformers actually trained on such tasks converge to weights matching that construction. When your agent picks up a codebase convention from three examples in its context, learning is happening in a precise mathematical sense. It just happens in activations instead of weights, so it evaporates when the session ends.

Worse, this memory degrades while you use it. The clean escalation of evidence:

  • RULER (NVIDIA, 2024) tested 17 models with synthetic tasks harder than needle-in-a-haystack and found that among models claiming 32K+ contexts, “only half of them can maintain satisfactory performance” at 32K. Claimed context is not effective context.
  • NoLiMa (ICML 2025) removed the lexical overlap between question and needle, forcing associative rather than string-matching retrieval. Across 13 models claiming at least 128K, performance at 32K dropped below 50% of the short-context baseline for 11 of them; GPT-4o fell from a 99.3% baseline to 69.7%.
  • Chroma’s “Context Rot” report (July 2025) ran 18 then-current models, including GPT-4.1, Claude 4, and Gemini 2.5, and found the same shape everywhere, even on trivially simple tasks: models “do not use their context uniformly; performance grows increasingly unreliable as input length grows.” On LongMemEval, focused ~300-token prompts beat full ~113K-token prompts across every model tested.

By late 2025 the phenomenon had a name and an official acknowledgment: Anthropic’s context engineering guide states plainly that “as the number of tokens in the context window increases, the model’s ability to accurately recall information from that context decreases,” and prescribes the mitigations that now define agent practice: compaction (summarize and restart), structured note-taking (write memory outside the window and pull it back on demand), and sub-agent architectures (let workers burn their own contexts and return condensed summaries). There is also a straightforwardly economic reason not to hoard context: every token you keep resident is re-attended on every forward pass and enlarges the KV cache that dominates serving memory, a bill computed line by line in the earlier post on the KV cache.

So “just make the context longer” is not a memory strategy, a conclusion the working-with-coding-agents post reached from the practitioner side: context is a scarce resource to be spent, not a warehouse to be filled. Which raises the real question: if memory should live outside the window, how does the right memory get back in at the right moment?

The memory stream: a scoring rule from 2023

The answer that stuck was invented for, of all things, a village of simulated people. Stanford’s Generative Agents (Park et al., 2023) populated a Sims-like town with 25 LLM-driven characters who needed to behave consistently over days: remember conversations, hold grudges, plan parties. The architecture stored every observation as a natural-language entry in an append-only memory stream, and at each step retrieved the entries scoring highest on a three-part rule. Each part can carry a coefficient, but with all three set to 1 in their implementation, the score of memory \(m\) for query \(q\) is

\[s(m, q) \;=\; \textcolor{#c2410c}{\underbrace{\gamma^{\,\Delta t}}_{\text{recency}}} \;+\; \textcolor{#199e70}{\underbrace{I(m)}_{\text{importance}}} \;+\; \textcolor{#2a78d6}{\underbrace{\cos\!\big(\mathbf{e}_m, \mathbf{e}_q\big)}_{\text{relevance}}},\]

with each component min-max normalized to \([0,1]\) across the store. The details are worth savoring because every production system since has some version of them:

  • Recency is exponential decay with \(\gamma = 0.995\) per hour, but the clock runs from the memory’s last retrieval, not its creation. Memories you keep using stay hot; this is spaced repetition emerging from a systems decision.
  • Importance is scored once, at write time, by the LLM itself, on a 1-to-10 scale anchored by the delightfully specific prompt: 1 is “purely mundane (e.g., brushing teeth, making bed)” and 10 is “extremely poignant (e.g., a break up, college acceptance).”
  • Relevance is cosine similarity between embeddings, the same mechanism that powers retrieval-augmented generation (RAG) and the semantic search case study on this blog, pointed here at the agent’s own past instead of a document corpus.
Line chart of retrieval score versus hours since three memories were written, over one week. A blue line (memory m1, high relevance 0.9, importance 6) starts highest at 2.5 and decays. A green line (m2, relevance 0.45, importance 8) starts second at 2.25, decays in parallel, then jumps back up at hour 96 when it is retrieved by another query, overtaking m1 for the rest of the week. An orange line (m3, relevance 0.3, importance 5) sinks in parallel below both. Annotations note that the recency clock resets on retrieval and that untouched memories all decay at the same 0.995-per-hour rate.
The Generative Agents score in motion, computed with the paper's decay of 0.995 per hour. All untouched memories sink in parallel, so ranks are stable until usage intervenes: when m2 is retrieved at hour 96 its recency clock resets and it overtakes m1 not by content but by use. Component values for the three memories are illustrative; the dynamics are the real formula.

Retrieval, though, is the easy half. The hard half is deciding what to write, and the field converged on a small vocabulary for it.

Building the store: from lessons to pipelines

A 2026 ACL Findings survey describes agent memory as evolving through three stages: storage (keep trajectories), reflection (refine them), and experience (abstract reusable knowledge from them). The 2023 systems already sketched all three:

  • Reflexion (Shinn et al., NeurIPS 2023) made the episodic store self-authored: after a failed attempt, the agent writes a verbal reflection on what went wrong into a memory buffer that persists across retries. That single loop (fail, write the lesson, retry with the lesson in context) hit 91% pass@1 on HumanEval (the fraction of coding problems solved on the first returned attempt) when GPT-4 alone managed 80%, and it is the same mechanism whose descendants power the goal loops discussed in the previous post.
  • Voyager (Wang et al., 2023) did the equivalent for procedural memory in Minecraft: every behavior the agent verified gets stored as executable code in an ever-growing skill library, indexed by an embedding of its description. The compounding was dramatic (3.3x more unique items discovered, tech-tree milestones up to 15.3x faster than prior state of the art), and the paper’s framing is exactly ours: a skill library is knowledge kept where gradient descent cannot smudge it, which “alleviates catastrophic forgetting.”
  • MemGPT (Packer et al., 2023) contributed the systems metaphor: treat the context window like RAM and let the LLM page information between it and disk, through function calls the model issues itself. Main context (system instructions, a writable working block, a FIFO queue of recent messages) backs onto recall storage (the full message history) and archival storage (arbitrary documents). On their deep memory retrieval benchmark, GPT-4 with MemGPT answered 92.5% of questions about earlier sessions correctly versus 32.1% for the same model with a fixed window. The project grew into Letta, and its “the LLM edits its own memory” design became the default.

The 2025 systems industrialized the write path. Mem0 runs a two-phase pipeline on every conversation turn: an extraction phase pulls candidate facts, and an update phase compares each candidate against similar existing memories and chooses an operation: ADD a new memory, UPDATE an existing one, DELETE a contradicted one, or NOOP. On the LOCOMO benchmark (multi-session conversations averaging 300 turns across up to 35 sessions), Mem0 reports a 26% relative improvement in LLM-judged accuracy over OpenAI’s memory feature, with 91% lower 95th-percentile latency and over 90% token savings compared to stuffing the full history into context. Those last two numbers are really the point of the whole enterprise: a memory system is a bet that a curated store beats a transcript dump on both quality and cost. (Honesty requires a hedge: these are vendor-reported numbers against baselines of the vendor’s choosing, and independent re-implementations, including one we will meet shortly, measure Mem0 well below its self-reported scores.)

A-MEM (Xu et al., 2025) pushed the store’s structure further with an explicitly Zettelkasten-inspired design: each memory becomes a note carrying LLM-generated keywords, tags, and a contextual description; new notes trigger link generation (an LLM decides which existing notes to connect, from embedding-nominated candidates) and memory evolution, where old notes’ descriptions and tags are rewritten in light of new experience. The store stops being an append-only log and becomes a living document.

A minimal memory layer you can run

Sixty lines capture the shared skeleton: scored retrieval with a resettable recency clock, plus a consolidation step choosing among the standard operations. A rule stub stands in for the LLM (in Mem0) or the trained policy (in Memory-R1) that makes the ADD/UPDATE/NOOP call in real systems, and a bag-of-words stands in for the embedding model, so the whole thing runs with no dependencies.

import math
from collections import Counter


def embed(text: str) -> Counter:
    """Toy embedding: a bag of words. Swap in a real embedding model."""
    return Counter(text.lower().replace(",", " ").split())


def cosine(a: Counter, b: Counter) -> float:
    dot = sum(a[w] * b[w] for w in a)
    norm = math.sqrt(sum(v * v for v in a.values()))
    norm *= math.sqrt(sum(v * v for v in b.values()))
    return dot / norm if norm else 0.0


class MemoryStore:
    """A miniature Generative-Agents-style memory with consolidation."""

    decay = 0.995  # recency decay per hour since last retrieval

    def __init__(self):
        self.memories = []  # dicts: text, importance, embedding, last_used

    def score(self, memory, query_vec, now: float) -> float:
        recency = self.decay ** (now - memory["last_used"])
        relevance = cosine(memory["embedding"], query_vec)
        return recency + memory["importance"] + relevance

    def retrieve(self, query: str, now: float, k: int = 2) -> list:
        query_vec = embed(query)
        scored = sorted(((self.score(m, query_vec, now), m)
                         for m in self.memories), reverse=True,
                        key=lambda pair: pair[0])
        for _, m in scored[:k]:
            m["last_used"] = now  # retrieval resets the recency clock
        return scored[:k]

    def consolidate(self, text: str, importance: float, now: float) -> str:
        """Decide ADD / UPDATE / NOOP against the closest existing memory.

        A rule stub stands in for the LLM (Mem0) or RL policy (Memory-R1)
        that makes this call in production systems.
        """
        vec = embed(text)
        best = max(self.memories, default=None,
                   key=lambda m: cosine(m["embedding"], vec))
        sim = cosine(best["embedding"], vec) if best else 0.0
        if sim > 0.9:
            return f"NOOP   (duplicate of: {best['text']!r})"
        if sim > 0.45:
            old = best["text"]
            best.update(text=text, embedding=vec,
                        importance=importance, last_used=now)
            return f"UPDATE ({old!r} -> {text!r})"
        self.memories.append(dict(text=text, importance=importance,
                                  embedding=vec, last_used=now))
        return f"ADD    ({text!r})"


store = MemoryStore()
for hour, text, imp in [
    (0, "user prefers pytest for the test suite", 0.6),
    (1, "the CI runner migrated to arm64 in June", 0.5),
    (30, "user prefers pytest for the test suite", 0.6),
    (48, "user prefers unittest for the test suite now, not pytest", 0.7),
]:
    print(f"h={hour:<3} {store.consolidate(text, imp, hour)}")

print("\nquery at h=72: 'which test framework does the user want?'")
for score, m in store.retrieve("which test framework does the user want?", now=72):
    print(f"  score={score:.3f}  {m['text']}")

h=0   ADD    ('user prefers pytest for the test suite')
h=1   ADD    ('the CI runner migrated to arm64 in June')
h=30  NOOP   (duplicate of: 'user prefers pytest for the test suite')
h=48  UPDATE ('user prefers pytest for the test suite' -> 'user prefers unittest for the test suite now, not pytest')

query at h=72: 'which test framework does the user want?'
  score=1.945  user prefers unittest for the test suite now, not pytest
  score=1.334  the CI runner migrated to arm64 in June

The trace shows the two moments that separate a memory system from a log. At hour 30 the duplicate is recognized and dropped rather than stored again, and at hour 48 the preference changes, so the right operation is not to append a contradicting fact next to the stale one but to rewrite it in place. Everything hard about production memory lives in that second decision: recognizing that “user prefers unittest now” supersedes “user prefers pytest” requires understanding, not string matching, which is why the deciding component in real systems is an LLM and, increasingly, a trained policy.

When the librarian learns

Prompted consolidation has an obvious weakness: nobody optimized it. The LLM deciding ADD versus UPDATE was never trained on the consequences of those decisions; it is doing zero-shot judgment with a prompt. Memory-R1 (Yan et al., 2025) closed that loop with reinforcement learning: a Memory Manager agent learns the ADD/UPDATE/DELETE/NOOP policy and an Answer Agent learns to select and reason over retrieved entries, both fine-tuned with the reinforcement-learning algorithms PPO and GRPO, using a reward that is nothing more than downstream answer correctness. No per-operation labels exist; an operation was good if the agent could answer questions correctly later. The efficiency is striking, “with only 152 training QA pairs” in their words, and on LOCOMO with a LLaMA-3.1-8B backbone the trained manager lifts F1 (a blend of answer precision and recall) from Mem0’s 27.3 to 35.7 and LLM-judged accuracy from 43.9 to 59.8. The same summer produced Memento (Zhou et al., 2025), which frames the whole agent-improvement problem as case-based reasoning: store episodic cases, train only a lightweight case-selection policy, never touch the LLM’s weights, good for an 87.9% Pass@3 on the GAIA validation set and a 4.7-to-9.6 point gain out of distribution from the case memory alone.

MemGen (Zhang et al., 2025) points at something stranger: maybe the store should not hold text at all. A trained trigger watches the reasoning stream and decides when memory should fire; a trained weaver then generates a sequence of latent tokens, machine-native memory rather than prose, that gets spliced into the ongoing reasoning. Across eight benchmarks it reports beating external-store systems like ExpeL and AWM by up to 38.2%. Whether or not MemGen’s specific recipe wins, it marks the direction of travel visible across this whole section: memory operations migrating from hand-designed, to prompted, to learned.

Meanwhile the production systems shipped the boring, effective version of all this:

  • Claude Code’s memory splits exactly along the lines this post has been drawing: CLAUDE.md files are procedural memory you write (instructions loaded every session), while auto memory is a per-project directory of notes the agent writes itself, with an index file whose first 200 lines load at session start and whose topic files are read on demand. That index-plus-lazy-load design is a direct answer to context rot.
  • Anthropic’s memory tool (September 2025) gives API developers the same primitive: a client-side directory the model can create, read, update, and delete across conversations. Their internal agentic-search evaluation credits memory plus context editing with a 39% improvement over baseline (context editing alone: 29%), and context editing cut token consumption 84% in a 100-turn evaluation.
  • Sleep-time compute (Lin et al., 2025, from the MemGPT lineage) moves consolidation offline: let the model think about its memory between sessions, pre-computing inferences so they are ready before the next query arrives. Matched-accuracy test-time compute drops about 5x on their stateful benchmarks, with accuracy gains up to 13-18% when sleep-time thinking scales up.
  • Anthropic’s “dreaming” (announced May 2026 for Claude Managed Agents) productizes that idea under the best name in the industry: per their announcement, “a scheduled process that reviews your agent sessions and memory stores, extracts patterns, and curates memories so your agents improve over time,” with the tagline that “together, memory and dreaming form a robust memory system for self-improving agents.”

Notice what every one of these does not do: change the weights. The entire commercial memory stack, as of mid-2026, is context-level machinery. To see why, we have to visit the second half of the problem, and its decades-old curse.

The other place to keep memory: in the weights

Karpathy’s sharpest observation on the Dwarkesh podcast was not the complaint but the diagnosis of what is missing: “When I go to sleep, something magical happens where I don’t think that context window stays around. There’s some process of distillation into the weights of my brain.” Humans do not keep their skills in a notes file. Practice becomes parameter change. The obvious move for agents, then, is fine-tuning on experience: as covered in the LoRA fine-tuning post, the mechanics are cheap and well understood. So why does no major agent product learn this way?

Because sequential gradient descent on new data overwrites old competence, a failure mode named catastrophic interference by McCloskey and Cohen in 1989 and given its definitive review by Robert French in 1999, whose abstract contains the cruelest sentence in the field: “the very features that give these networks their much-touted abilities to generalize, to function in the presence of degraded input, etc., are the root cause of catastrophic forgetting.” Distributed representations share weights across everything the network knows; gradients for the new task pull on weights the old tasks were relying on. This is not a quaint small-network pathology, and it does not vanish at LLM scale. Luo et al. (2023) measured forgetting of domain knowledge, reasoning, and reading comprehension during continual instruction tuning of 1B-to-7B models and found it “generally observed,” with severity increasing with scale in that range. Meta’s continual-learning team supplies the most concrete recent number: fully fine-tuning an LLM on a batch of new facts collapsed its NaturalQuestions F1 by 89%; even LoRA, which freezes the base weights and trains a small sidecar, lost 71%.

Numbers like that are best felt by producing them yourself.

Watching a network forget

The classic demonstration protocol comes from the EWC paper: take one task, derive a second task of identical difficulty by fixing a random permutation of the input features, and train on them sequentially. The permutation guarantees task B needs the network’s capacity organized differently while being exactly as learnable as task A. This post’s committed figure script runs the whole experiment in pure numpy: an 8-class Gaussian classification problem in 128 dimensions with the class signal confined to 16 of them (mirroring MNIST’s sparse informative pixels), a two-hidden-layer MLP of width 512 with dropout, and a permuted-features task B. Every number below is printed by that script.

Training on task A for 40 epochs reaches 90.0% test accuracy. Then task B arrives, and plain stochastic gradient descent (SGD) on it is a demolition: task A accuracy falls from 0.900 to 0.872 after one epoch of B, 0.680 after five, and 0.304 after thirty (chance for eight classes is 12.5%), while task B climbs to 0.880. Nothing about task B required destroying A; a network trained on both tasks together ends at 0.849 and 0.851. Sequential presentation alone did this.

Two-panel figure. Left panel: test accuracy versus epoch over 70 epochs. An orange task-A curve rises to 0.90 during 40 epochs of training on A, then collapses toward 0.30 once training switches to task B, while a dashed blue task-B curve rises from chance (0.125) to 0.88. A red annotation marks the collapse from 0.90 to 0.30. Right panel: task A accuracy during the 30 epochs of task B training under three strategies: plain SGD falls to 0.30, a green replay curve (200 stored examples) holds around 0.54, and a purple EWC curve with lambda 10^5 holds around 0.64, all below a dashed line at 0.90 marking accuracy before B arrived.
Catastrophic forgetting, computed live by the post's figure script. Left: plain SGD on task B erases task A (0.90 to 0.30) even though a jointly trained network handles both at about 0.85 each. Right: what survives of task A under each mitigation; EWC and replay slow the bleed but neither stops it.

Two standard mitigations are overlaid on the right panel:

  • Replay (the idea dates to Robins, 1995): keep a small buffer of raw task A examples and mix a few into every task B batch. Two hundred stored examples, 5% of task A’s training set, hold task A at 0.536 instead of 0.304.
  • EWC with \(\lambda = 10^5\) holds task A at 0.643 while still reaching 0.766 on B, and unlike replay it stores no data at all, only a per-weight importance estimate. Where does that come from? It falls out of a genuinely pretty piece of Bayesian reasoning, which is the next section.

EWC: a quadratic leash on the weights that matter

Elastic Weight Consolidation (Kirkpatrick et al., 2017, DeepMind) starts from the Bayesian view of what training on task A even produced: not just a point estimate but a posterior, the distribution over weights that are plausible given task A’s data. By Bayes’ rule on the combined data of two independent tasks, the log-posterior splits as

\[\log p(\theta \mid \mathcal{D}_A, \mathcal{D}_B) \;=\; \log p(\mathcal{D}_B \mid \theta) \;+\; \log p(\theta \mid \mathcal{D}_A) \;-\; \log p(\mathcal{D}_B),\]

so everything task A has to say about the weights is carried by \(\log p(\theta \mid \mathcal{D}_A)\), the posterior after task A. That posterior is intractable, but near the trained optimum \(\theta^*_A\) it can be Laplace-approximated (tool 1), meaning replaced by the best-fitting Gaussian at its peak: centered at \(\theta^*_A\), with precision given by the curvature of the log-posterior there. Estimating that curvature with the Fisher information, which equals the expected negative Hessian of the log-likelihood (tool 2) while being computable from first derivatives alone and guaranteed nonnegative, and keeping only its diagonal \(F_i\) (one importance number per weight), turns “stay probable under task A’s posterior” into a quadratic penalty. Training on task B then minimizes exactly the paper’s Equation 3:

\[\mathcal{L}(\theta) \;=\; \mathcal{L}_B(\theta) \;+\; \sum_i \frac{\lambda}{2}\, F_i \,\big(\theta_i - \theta^*_{A,i}\big)^2 .\]

The abstract’s one-line gloss is the right mental model: EWC “remembers old tasks by selectively slowing down learning on the weights important for those tasks.” Weights with high Fisher information (moving them changes task A’s predictions a lot) are held on a stiff spring anchored at \(\theta^*_A\); weights task A never cared about are free to serve task B. A small implementation note that matters in practice: applying this penalty as a naive gradient step with learning rate \(\eta\) diverges whenever \(\eta \lambda F_i > 2\) (the per-coordinate recursion overshoots), so the script integrates the quadratic pull exactly, \(\theta \leftarrow \theta^* + (\theta - \theta^*)\,e^{-\eta \lambda F}\) (tool 3), which is stable for every \(\lambda\) and lets us sweep the whole regime.

And sweeping the whole regime reveals the honest picture, which a single cherry-picked \(\lambda\) would hide:

Scatter plot of task A accuracy (stability, y-axis) versus task B accuracy (plasticity, x-axis) after sequential training. A purple EWC curve sweeps lambda from 30 (bottom right, near plain SGD) to 10^6 (upper left, task A 0.69 but task B only 0.66). A green replay curve sweeps buffer sizes from 25 to 1600 examples and sits above the EWC curve through the middle of the range. A black dot marks plain SGD at (0.88, 0.30) and a gold star marks joint training at (0.85, 0.85), far above both frontiers.
The stability-plasticity frontier, computed by the figure script. Every mitigation trades new-task accuracy for old-task retention along a curve; more lambda or a bigger buffer slides you along it, and nothing on either frontier approaches the gold star of joint training. Replay's frontier dominates EWC's through the middle: storing a little raw experience beats a clever curvature penalty here.

Three lessons condense out of this plot, and they generalize far beyond a toy MLP:

  1. There is no free knob. Raising \(\lambda\) from 30 to \(10^6\) walks task A retention from 0.35 up to 0.69 while task B falls from 0.87 to 0.66. Stability is purchased with plasticity; the dilemma is structural, not a tuning failure.
  2. Cheap raw experience is embarrassingly competitive. The replay frontier sits above EWC’s through the operating range. This echoes the broader continual-learning literature and lands close to the CLS story from earlier: the brain’s own solution involves replaying hippocampal episodes into the neocortex during sleep, not a curvature penalty. When memory is cheap, store the data.
  3. Joint training is the ceiling, and nothing sequential touches it. The gold star at (0.851, 0.849) is what “just retrain on everything, interleaved” buys. That gap is a large part of why frontier labs periodically retrain models from scratch on refreshed corpora instead of continually fine-tuning the deployed one, and why your agent’s vendor ships it memory files instead of nightly gradient updates.

For completeness, the classical toolbox has more tools than these two, each with the same tradeoff wearing different clothes: Learning without Forgetting distills the old model’s outputs into the new one; progressive networks freeze old columns and grow new ones (immune to forgetting, parameters grow forever); PackNet prunes and packs tasks into disjoint weight subsets; synaptic intelligence computes EWC-like importances online. And the modern LLM-era workaround inherits from all of them: one LoRA adapter per task or customer, swapped at inference, keeps the base weights pristine at the cost of never integrating anything into them.

Weights that learn at inference time

If writing to all the weights is dangerous and writing to none of them caps skill, the frontier’s answer is to build models where some designated weights are meant to be written, continuously, by the input stream itself.

The lineage starts from an observation in the attention post: linear attention maintains a running state matrix \(S_t = S_{t-1} + \phi(k_t)v_t^{\top}\), a fixed-size associative memory accumulated by plain summation. Summation is a terrible librarian, though; it never overwrites and never forgets. TTT layers (Sun et al., 2024) replaced the sum with actual learning: the recurrent state is a small model (linear, or a two-layer MLP), and the “update rule” of the RNN is a step of self-supervised gradient descent on the incoming token. The hidden state literally trains at test time, and unlike Mamba, whose fixed-size state stops helping past 16K tokens in their evaluation, TTT layers keep improving as context grows.

Google’s Titans (Behrouz et al., December 2024) scaled the idea into a full memory module and made the psychology explicit. Its neural memory \(M\) is trained, at inference, on an associative objective \(\ell(M; x_t) = \lVert M(k_t) - v_t \rVert^2\): store the mapping from the current token’s key \(k_t\) to its value \(v_t\), both projections of the token \(x_t\). Two twists are borrowed from how organisms memorize. First, updates are gated by surprise: the update direction is a momentum-smoothed history of gradients,

\[S_t \;=\; \eta_t\, S_{t-1} \;-\; \theta_t\, \nabla \ell\big(M_{t-1}; x_t\big),\]

where \(\theta_t\) is a data-dependent learning rate and this \(S_t\) is the paper’s surprise state, a different object from the linear-attention state above. Tokens the memory already predicts well (small gradient, no surprise) barely write, violated expectations write hard, and the momentum term \(\eta_t S_{t-1}\) lets a surprising moment keep writing for a while after it happens. Second, memory decays: the state update \(M_t = (1 - \alpha_t)\,M_{t-1} + S_t\) carries a learned forgetting gate \(\alpha_t \in [0,1]\) that lets the module erase itself when the context moves on. For a linear memory, one gradient step on \(\ell\) expands (tool 4) to

\[M_t \;=\; M_{t-1}\big(I - 2\theta_t\, k_t k_t^{\top}\big) \;+\; 2\theta_t\, v_t k_t^{\top},\]

which is exactly linear attention’s write (\(v_t k_t^{\top}\)) plus an erase term that removes what the memory currently predicts for \(k_t\) before writing the correction: the delta rule, rediscovered as the fix for the summation librarian. Paired with ordinary attention as accurate short-term memory, Titans handles needle-in-a-haystack recall tasks past 2M tokens; ATLAS (May 2025) upgraded the per-token update to optimize over a sliding window of recent tokens, reporting “+80% accuracy” at 10M-token context on the BABILong benchmark; and the group’s NeurIPS 2025 Nested Learning paper reframes an entire model as a hierarchy of such memories, each level updating at its own frequency, with a self-modifying architecture (HOPE) as the demonstration. The pitch, in their words, is a route around catastrophic forgetting itself.

A complementary Meta line asks where facts should live so that updating them is safe. Memory layers replace some FFN capacity with a vast sparsely-activated key-value lookup (scaled to 128B memory parameters), which outperforms dense models given more than twice the compute on factual tasks. Because each fact touches only a few slots, sparse memory finetuning (October 2025) can teach the model new facts by updating only the slots that new knowledge activates unusually strongly relative to pretraining traffic. The result is the cleanest forgetting number in recent literature: after learning the new facts, held-out NaturalQuestions F1 drops 89% under full fine-tuning, 71% under LoRA, and 11% under sparse memory finetuning, at equal new-knowledge acquisition. Add early-2026 work like SDFT, which has the model generate its own on-policy training signal from demonstrations (an idea with clear kinship to knowledge distillation, with the model as its own teacher) and sequentially accumulates skills with substantially reduced forgetting, and in-weights learning starts to look less cursed than its reputation.

The map of the territory

One picture now organizes everything this post has visited. Ask two questions of any memory mechanism: where does the memory live (in the context, in an external store, in the weights) and who writes it (a human or hand-built rule, the model via prompted decisions, or a learning algorithm).

A map with three tinted columns (in the context, blue; in an external store, green; in the weights, purple) and a vertical axis running from written by hand, through written by the model, to written by learning. In the context column, system prompts and CLAUDE.md sit at the hand-written bottom, compaction and self-curated instructions in the middle, MemGen latent memory at the learned top. In the store column, classic RAG and the Generative Agents stream sit low, Reflexion, Voyager, MemGPT, and Mem0/A-MEM pipelines in the prompted middle, sleep-time compute and dreaming above them, and Memory-R1/Memento at the RL-trained top. In the weights column, hand-picked fine-tune sets sit at the bottom and LoRA adapters, EWC and replay, sparse memory finetuning, and TTT/Titans/ATLAS/HOPE stack toward the top.
Map 2: the territory of agent memory, organized by where the memory lives (columns) and who writes it (vertical axis). Each column's top is where its 2024-26 research energy is concentrated: within every substrate, the write policy is migrating from hand-designed rules through prompted model decisions to learned policies and inference-time gradients.

Read column by column, the map compresses the post: context memory is precise, universal, and evaporating; store memory is unbounded, inspectable, and only as good as its retrieval; weight memory is always-on and compressed but hostile to writes. Read bottom to top, it shows the field’s actual direction of motion. In every column, the hand-designed layer came first, the prompted layer shipped as products, and the learned layer is where the papers are: Memory-R1’s RL librarian in the store column, MemGen’s trained weaver in the context column, TTT and its descendants in the weights column. The write policy, not the storage substrate, is where the field believes the remaining performance lives.

The brain, for what it is worth, occupies the top row of all three columns at once: its stores are written by learned consolidation policies, its “weights” learn continually under complementary-systems scheduling, and nothing about it is hand-designed. That is one way to state how far there is to go.

Is continual learning the bottleneck?

Here the engineering ends and the argument begins, so it is worth being explicit about who claims what.

The “yes” camp. Dwarkesh Patel’s essay stakes the strong claim: without continual learning, agents stay permanent day-one hires, and with it, things move fast: “An AI that is capable of online learning might functionally become a superintelligence quite rapidly without any further algorithmic progress.” His concrete bets: 50/50 on an AI that can do small-business taxes end-to-end by 2028, and 50/50 on an AI that “learns on the job as seamlessly as humans” for white-collar work by 2032. He also predicts, plausibly, that we will first get “a broken early version of continual learning” because labs ship innovations early. Karpathy lands in the same camp with a longer clock, “the problems are tractable, they’re surmountable, but they’re still difficult. If I just average it out, it just feels like a decade to me,” and his system prompt learning post sketches the near-term shape: a lot of human learning feels less like weight updates and more like editing your own instructions, so let models write and revise their own system prompts. (Squint at Claude Code’s self-written memory directory and you are looking at exactly that.)

The “maybe not” camp. Dario Amodei, pressed on continual learning by Dwarkesh in their February 2026 conversation, offered the counterposition: “I think with the coding agents, I don’t think people would say that learning on the job is what is preventing the coding agents from doing everything end to end.” His view is that pre-training plus long-context in-context learning “may just be enough,” with continual learning as an accelerant rather than a prerequisite, though he added: “there’s a good chance that in the next year or two, we also solve that.”

A reasonable synthesis, held with explicit uncertainty: for state, the memory stack described in this post already works and is compounding fast; for skill, in-context learning over ever-longer horizons is carrying more weight than the pessimists expected, and true in-weights continual learning remains unsolved in deployment, with TTT-style architectures and sparse-update fine-tuning as the live candidates. Whether the gap between “carrying more weight” and “sufficient” closes from the context side or the weights side is, right now, arguably the most consequential open question in the field, and this post declines to pretend it knows the answer.

What to do today

None of the open questions excuse skipping the parts that already work. If you run agents daily, the practical translation of this post is short:

  • Write the procedural store yourself, and keep it small. Your CLAUDE.md or AGENTS.md is procedural memory loaded every session, and it pays context-rot rent on every request; the working-with-coding-agents post covers keeping it an index rather than an encyclopedia.
  • Let the agent write the episodic and semantic stores, but read them. Auto-memory files are your agent’s beliefs about your project. Stale ones actively poison future sessions; the UPDATE and DELETE operations exist for a reason, and occasionally you should be the one issuing them.
  • Prefer restarting with notes over marinating in a long session. Compaction plus a good store beats a 200K-token context that has begun to rot. When a session goes sideways, the lesson belongs in a memory file, and the session belongs in the bin.
  • Schedule consolidation. The single most underrated pattern from this whole literature is sleep-time processing: a recurring job that reviews recent sessions, extracts durable lessons, merges duplicates, and prunes stale facts. Every serious memory paper and product of 2025-26, from A-MEM’s memory evolution to Anthropic’s dreaming, is a variation on it.
  • Save fine-tuning for skills, not facts. Facts change and forgetting is real; stores handle facts gracefully. If you do push experience into weights, the literature’s advice is consistent: isolate the update (adapters, sparse slots) and keep something to replay.

The agent you work with next year will very likely still be a ghost in Karpathy’s sense. But a ghost with a well-run memory system is a surprisingly capable colleague, and the distance between the two is, today, mostly an engineering choice you get to make.

Appendix: the derivation toolbox

Tool 1: the Laplace approximation

Claim. Let \(f(\theta) = \log p(\theta \mid \mathcal{D})\) be twice differentiable with a maximum at \(\theta^*\). Then near \(\theta^*\),

\[p(\theta \mid \mathcal{D}) \;\approx\; \mathcal{N}\!\big(\theta^*,\, H^{-1}\big), \qquad H \;=\; -\nabla^2 f(\theta)\big\rvert_{\theta^*}.\]

Proof. Taylor-expand \(f\) around \(\theta^*\). The zeroth-order term \(f(\theta^*)\) is a constant. The first-order term vanishes because \(\nabla f(\theta^*) = 0\) at an interior maximum. Keeping the quadratic term,

\[f(\theta) \;\approx\; f(\theta^*) \;-\; \tfrac{1}{2}\,(\theta - \theta^*)^{\top} H \,(\theta - \theta^*),\]

with \(H\) positive semi-definite because \(\theta^*\) is a maximum. Exponentiating gives \(p(\theta \mid \mathcal{D}) \propto \exp\!\big(-\tfrac{1}{2}(\theta - \theta^*)^{\top} H (\theta - \theta^*)\big)\), which is the kernel of a Gaussian with mean \(\theta^*\) and covariance \(H^{-1}\); normalizing constants do not depend on \(\theta\), so the approximation is exactly the claimed Gaussian. \(\blacksquare\)

Taking the log of this Gaussian and keeping only \(\theta\)-dependent terms yields the quadratic penalty in the EWC loss; EWC additionally replaces \(H\) by the diagonal of the Fisher information, justified next.

Tool 2: the information matrix equality

Claim. For a model \(p(y \mid \theta)\) satisfying standard regularity conditions (support independent of \(\theta\), differentiation and integration interchangeable),

\[F \;=\; \mathbb{E}_{y \sim p(\cdot \mid \theta)}\!\Big[\nabla_\theta \log p(y \mid \theta)\, \nabla_\theta \log p(y \mid \theta)^{\top}\Big] \;=\; -\,\mathbb{E}_{y \sim p(\cdot \mid \theta)}\!\Big[\nabla^2_\theta \log p(y \mid \theta)\Big].\]

Proof. Start from normalization: \(\int p(y \mid \theta)\, dy = 1\) for all \(\theta\). Differentiating once and interchanging,

\[0 \;=\; \int \nabla_\theta p(y \mid \theta)\, dy \;=\; \int p(y \mid \theta)\, \nabla_\theta \log p(y \mid \theta)\, dy,\]

using \(\nabla_\theta p = p \, \nabla_\theta \log p\). So the score has mean zero. Differentiate that identity once more:

\[0 \;=\; \int \Big[ p \,\nabla^2_\theta \log p \;+\; \big(\nabla_\theta p\big) \nabla_\theta \log p^{\top} \Big] dy \;=\; \mathbb{E}\big[\nabla^2_\theta \log p\big] \;+\; \mathbb{E}\big[\nabla_\theta \log p \, \nabla_\theta \log p^{\top}\big],\]

where the second term again used \(\nabla_\theta p = p\, \nabla_\theta \log p\). Rearranging gives the equality. \(\blacksquare\)

This is why EWC can use the Fisher as its curvature: at a maximum-likelihood solution the expected Hessian of the negative log-likelihood is the Fisher, and the outer-product form on the left is computable with first derivatives only and is nonnegative by construction (it is an expectation of squares). One subtlety the paper’s construction respects: the expectation is over labels drawn from the model’s own predictive distribution, which is why the experiment’s Fisher routine samples \(y_i\) from the network’s softmax rather than using the dataset labels.

Tool 3: integrating the quadratic pull exactly

Claim. The gradient-flow of the EWC penalty alone, \(\dot\theta = -\lambda F (\theta - \theta^*)\) with diagonal \(F\), has the exact per-coordinate solution over a step of size \(\eta\):

\[\theta_i \;\leftarrow\; \theta^*_i \;+\; \big(\theta_i - \theta^*_i\big)\, e^{-\eta \lambda F_i},\]

which is stable for every \(\lambda \ge 0\), whereas the explicit Euler step \(\theta_i \leftarrow \theta_i - \eta \lambda F_i (\theta_i - \theta^*_i)\) diverges when \(\eta \lambda F_i > 2\).

Proof. Write \(u = \theta_i - \theta^*_i\). The flow is the scalar linear ODE \(\dot u = -\lambda F_i u\) with solution \(u(t) = u(0)\, e^{-\lambda F_i t}\); evaluating at \(t = \eta\) gives the update, and since \(0 < e^{-\eta \lambda F_i} \le 1\) the displacement can only shrink. The Euler step instead maps \(u \leftarrow (1 - \eta \lambda F_i)\, u\), whose magnitude contracts only if \(\lvert 1 - \eta \lambda F_i \rvert < 1\), i.e. \(0 < \eta \lambda F_i < 2\); beyond 2 the iterate oscillates with growing amplitude. \(\blacksquare\)

The experiment alternates one SGD step on task B’s loss with one exact penalty step, a first-order splitting of the full dynamics that stays stable across the six orders of magnitude of \(\lambda\) in the frontier sweep.

Tool 4: the gradient of the associative memory loss

Claim. For a linear memory \(M\) (a matrix), key \(k\), and value \(v\), the loss \(\ell(M) = \lVert Mk - v \rVert^2\) has gradient

\[\nabla_M\, \ell \;=\; 2\,(Mk - v)\,k^{\top},\]

so a single gradient step with rate \(\theta_t\) gives \(M_t = M_{t-1}(I - 2\theta_t k_t k_t^{\top}) + 2\theta_t v_t k_t^{\top}\), the delta rule quoted in the Titans section.

Proof. Write \(\ell = \sum_i \big(\sum_j M_{ij} k_j - v_i\big)^2\) and differentiate with respect to a single entry \(M_{ab}\): only the \(i = a\) term depends on it, giving \(\partial \ell / \partial M_{ab} = 2\big(\sum_j M_{aj}k_j - v_a\big) k_b = 2\,(Mk - v)_a\, k_b\), which is entry \((a,b)\) of \(2(Mk - v)k^{\top}\). The update follows by substitution:

\[M_t \;=\; M_{t-1} - 2\theta_t (M_{t-1}k_t - v_t)k_t^{\top} \;=\; M_{t-1}\big(I - 2\theta_t\, k_t k_t^{\top}\big) + 2\theta_t\, v_t k_t^{\top}. \;\blacksquare\]

Setting the “erase” factor to \(I\) (no correction term) recovers linear attention’s pure-summation write \(M_t = M_{t-1} + v_t k_t^{\top}\) up to scale, which locates Titans precisely: it is the linear-attention memory upgraded from accumulating associations to correcting them, with surprise (the residual \(Mk - v\)) deciding how hard to write.

Sources & further reading

Memory systems

Context limits and products

Forgetting and continual learning

The debate

The figures in this post are generated by figures/2026-07-07-agent-memory.py; the forgetting experiment, frontier sweep, and every number quoted from them can be reproduced with one command.

Citation Information

If you find this content useful, please cite this work as:

Bhana, Nish. "Agent Memory: How AI Agents Remember, Forget, and Learn on the Job". Nish Blog (July 2026). https://www.nishbhana.com/Agent-Memory/

Or use the BibTeX citation:

@article{bhana2026agentmemory,
  title   = {Agent Memory: How AI Agents Remember, Forget, and Learn on the Job},
  author  = {Bhana, Nish},
  journal = {nishbhana.com},
  year    = {2026},
  month   = {July},
  url     = {https://www.nishbhana.com/Agent-Memory/}
}

Related Posts 🔥

Goals and Loops: Telling Your Coding Agent When It's Done
Agents Jul 2026
Working With Coding Agents: Principles for Reliable AI-Built Software
Agents Jul 2026
How LLMs Scale: FLOPs, Bytes, and the Price of a Token
NLP Jul 2026
The Evolution of Attention: One Mechanism, Four Escape Routes
NLP Jul 2026
Diffusion Models: Learning to Create by Learning to Destroy
ML Jul 2026
The KV Cache: Why LLMs Never Do the Same Work Twice
NLP Jul 2026

x.com, Facebook