💬 NLP

From Recall to Forgetting: Benchmarking Long-Term Memory for Personalized Agents

This paper introduces Memora, a long-term memory benchmark spanning weeks to months that evaluates personalized agents on remembering, reasoning, and recommending tasks using a novel Forgetting-Aware Memory Accuracy (FAMA) metric, revealing that current models frequently rely on obsolete information and struggle to reconcile evolving memories despite marginal improvements from dedicated memory agents.

Original authors: Md Nayem Uddin, Kumar Shubham, Eduardo Blanco, Chitta Baral, Gengyu Wang

Published 2026-04-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Md Nayem Uddin, Kumar Shubham, Eduardo Blanco, Chitta Baral, Gengyu Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a digital assistant, like a super-smart personal secretary. You've been working with this secretary for months. You've told them your favorite movies, your grocery budget, your fitness goals, and even that you used to hate a certain type of music but now love it.

Ideally, this secretary should remember all of that, update their notes when you change your mind, and forget the stuff you no longer care about. But in reality, most current AI assistants are like goldfish with a very short attention span. They remember what you said five minutes ago, but if you ask them about something you discussed three weeks ago, they often act like they've never met you. Or worse, they remember the old you, not the current you.

This paper, titled "From Recall to Forgetting," introduces a new way to test these assistants and a new metric to see if they are actually getting smarter or just getting confused.

Here is the breakdown in simple terms:

1. The Problem: The "Goldfish" vs. The "Human"

Current AI benchmarks (tests) are like asking a student to memorize a single fact from a textbook and recite it back.

The Old Way: "I told you yesterday I like pizza. Do you remember?" -> AI: "Yes."
The Real World: You tell the AI you like pizza. Two weeks later, you tell them you're on a diet and hate pizza. A month later, you tell them you're celebrating and want pizza again.
The Failure: Most AI assistants get confused. They might still recommend pizza because they forgot the diet, or they might say you hate pizza because they forgot the celebration. They struggle to update their memory and forget the old stuff.

2. The Solution: Introducing "Memora"

The authors created a new test called Memora. Think of Memora not as a pop quiz, but as a long-term relationship simulator.

Instead of a quick chat, Memora simulates interactions over weeks, months, and even quarters. It creates a "life" for a digital persona (like a busy software engineer or a creative designer) and tracks how their life changes.

The test checks the AI on three specific skills, using a "Memory Grounding" approach:

Remembering: Can you recall that I need to buy milk? (Basic recall)
Reasoning: Can you look at my grocery receipts from the last month and tell me if I'm over my budget? (Connecting the dots)
Recommending: Can you suggest a movie that fits my current taste, knowing I used to love horror movies but now prefer comedies? (Adapting to change)

3. The New Scorecard: "FAMA" (The "Forgetting" Test)

This is the most creative part of the paper. Previous tests only checked: "Did the AI get the answer right?"
But what if the AI got the right answer for the wrong reason? What if it recommended a horror movie because it remembered your old taste, even though you said you hate them now?

The authors introduced FAMA (Forgetting-Aware Memory Accuracy).

The Analogy: Imagine a detective solving a case.
- Old Score: Did the detective catch the criminal? (Yes/No)
- FAMA Score: Did the detective catch the criminal using the right clues?
- If the detective catches the criminal but uses a clue that was proven false yesterday, FAMA gives them a penalty. It rewards the AI for forgetting outdated info and remembering the new truth.

4. What They Found (The Plot Twist)

The researchers tested 4 big AI models and 6 different "memory agents" (specialized tools designed to help AI remember). The results were surprising:

The Longer the Time, the Worse the Memory: As the conversation stretched from a week to a quarter (3 months), the AI's performance tanked. It's like trying to remember a story told over 3 months; the details get fuzzy.
The "Forgetting" Failure: The biggest issue wasn't that the AI couldn't remember; it was that it couldn't forget. It kept holding onto old preferences and outdated facts, leading to bad recommendations.
Specialized Agents vs. Raw AI: The specialized "memory agents" were better at remembering facts (like a phone number), but they were still terrible at reasoning (connecting the dots) and adapting to changes.
The "Reasoning" Gap: Even the smartest AI models struggled to do math or logic based on old conversations. They could tell you "I spent $50," but they couldn't figure out "I have $50 left in my budget" if that info was buried in a chat from two weeks ago.

5. The Takeaway

The paper concludes that building a truly "personal" AI isn't just about giving it a bigger brain or a longer memory bank. It's about teaching it how to manage a library.

A good AI needs to:

Shelve new information correctly.
Cross-reference old information with new information.
Throw away (forget) the old, invalid books so they don't clutter the shelf.

Currently, our AI assistants are great at reading the book in front of them, but they are terrible at managing the library of their entire life with you. Memora is the first rigorous test to see if they can finally learn to be good librarians.

1. Problem Statement

Current Large Language Models (LLMs) and personalized agents lack robust persistent long-term memory. While they excel at short-term context retention within a single session, they fail to maintain coherent, evolving user states over weeks or months. Existing benchmarks for long-term memory suffer from two critical limitations:

Retrieval-Centric Focus: Most benchmarks (e.g., LoCoMo, LongMemEval) treat memory as a static fact-retrieval task, requiring models to recall isolated facts from previous sessions. They fail to test memory consolidation (integrating information across many sessions) or memory mutation (handling updates, corrections, and deletions).
Lack of Forgetting Evaluation: Standard metrics reward the inclusion of relevant information but do not penalize the reuse of obsolete or invalidated memory. In real-world scenarios, user preferences and goals change; an agent that recalls an old preference after it has been updated is functionally incorrect, yet standard metrics often overlook this.

2. Methodology: The Memora Benchmark

The authors introduce Memora, a comprehensive benchmark designed to stress-test long-term memory under realistic conditions of time, consolidation, and mutation.

A. Benchmark Construction

Simulation Pipeline: Memora uses a simulation-driven pipeline to generate long-horizon conversations (spanning weeks to months) based on 10 professional personas (e.g., researchers, engineers).
Memory Types: The simulation tracks three dynamic memory categories:
1. Preference Memory: Evolving likes/dislikes (movies, music, travel).
2. Activity Memory: Recurring personal and professional tasks (expenses, fitness, document drafting).
3. Goal Memory: Long-term objectives (budgets, fitness targets).
Temporal Scales: The benchmark is structured into three durations: Weekly (~~155 sessions), Monthly (~~615 sessions), and Quarterly (~1,991 sessions).
Memory Operations: The simulator explicitly introduces Add, Update, and Delete operations. Crucially, it enforces memory mutation, where information is revised or invalidated over time, requiring the agent to "forget" outdated data.
Quality Control: An automated memory-grounding loop verifies that generated conversations strictly adhere to the simulated memory traces. This is supplemented by human evaluation to ensure consistency.

B. Evaluation Tasks

Memora evaluates agents on three memory-grounded tasks:

Remembering: Direct recall of stored facts (e.g., "What is on my to-do list?").
Reasoning: Synthesizing multiple pieces of information to derive an answer (e.g., "Am I within my budget given these recent expenses?").
Recommending: Suggesting items based on current preferences, requiring the agent to ignore outdated ones (e.g., "Recommend a movie" after the user changed their taste).

C. New Metric: Forgetting-Aware Memory Accuracy (FAMA)

The paper proposes FAMA to replace standard accuracy metrics. FAMA aggregates two distinct criteria:

Memory Presence Accuracy (MPA): Rewards the inclusion of valid, current information.
Forgetting Absence Accuracy (FAA): Penalizes the inclusion of invalidated or deleted information.

The formula is defined as:
$\text{FAMA} = \max(0, \text{MPA} - \lambda \cdot (1 - \text{FAA}))$
Where $\lambda$ is a weighting factor based on the number of forgetting criteria. This metric explicitly penalizes agents that rely on obsolete memories, even if the final answer appears superficially correct.

3. Key Contributions

Memora Benchmark: A rigorous dataset spanning weekly to quarterly timelines with high demands on memory consolidation and mutation, significantly exceeding the complexity of prior benchmarks (Table 1 in the paper shows Memora requires up to 309 memory operations vs. ~15 in previous works).
FAMA Metric: A novel evaluation framework that quantifies the ability of agents to "forget" invalid data, addressing a critical blind spot in current LLM evaluation.
Comprehensive Empirical Analysis: Evaluation of 4 state-of-the-art LLMs (GPT-5.2, Claude Sonnet 4.5, etc.) and 6 specialized memory agents (A-Mem, LangMem, Mem-0, etc.) reveals systemic failures in maintaining consistent belief states.

4. Experimental Results

The evaluation of LLMs and memory agents yielded three primary findings:

Performance Degradation Over Time: Performance consistently declines from weekly to quarterly settings. As the interaction history grows and mutation pressure increases, agents struggle to maintain temporally correct memory states.
- Example: Memory agents like MemoBase saw their "Remembering" scores drop from 43.6 (Weekly) to 15.18 (Quarterly).
Task-Dependent Capabilities:
- Remembering: Specialized memory agents significantly outperform raw LLMs (Avg FAMA ~119 vs. ~66), proving the value of external memory stores for factual recall.
- Recommending: Raw LLMs perform competitively with or better than memory agents. This is likely because recommendation allows for plausible inference based on persona cues, whereas strict recall requires exact retrieval.
- Reasoning: This is the weakest area for all systems. Even the best agents score very low (~27.55), indicating that integrating multiple memory elements to solve complex problems remains a major challenge.
The "Forgetting" Gap: Standard metrics (Memory Presence Accuracy) vastly overestimate performance. When applying the FAMA penalty:
- Memory Agents suffer large score reductions as timelines lengthen (e.g., a 29.5 point drop in quarterly settings), indicating they increasingly rely on obsolete data that they fail to discard.
- LLMs show smaller reductions in longer timelines, not because they forget better, but because long histories exceed context windows, causing relevant information to be omitted entirely rather than misused.

5. Error Analysis

Manual analysis of 75 errors revealed task-specific failure modes:

Recommendation Errors (64%): Caused by failing to forget outdated preferences (e.g., recommending a movie genre the user explicitly stopped liking).
Remembering Errors (72%): Caused by partial retrieval, where agents retrieve some but not all required facts (e.g., missing a newly added task).
Reasoning Errors (100%): Caused by incomplete retrieval preventing the consolidation of facts needed for calculation or deduction.

6. Significance and Conclusion

The paper concludes that long-term memory is not a monolithic capability. Current agents can retrieve facts but fail to manage the lifecycle of memory (updating and forgetting).

Implication: Simply increasing context window size or adding external vector stores is insufficient. Future systems must treat forgetting, consolidation, and mutation as first-class design principles.
Benchmark Value: Memora provides a "stress test" that exposes brittleness in current systems, moving the field beyond simple fact-retrieval toward evaluating the dynamic, evolving nature of human-agent interaction.

In summary, Memora demonstrates that while agents can remember, they struggle to unlearn, a critical requirement for truly personalized, long-term AI companions.