🤖 AI

MedMemoryBench: Benchmarking Agent Memory in Personalized Healthcare

MedMemoryBench introduces a novel, large-scale benchmarking framework for personalized healthcare agents that utilizes a human-agent collaborative pipeline to generate realistic medical trajectories and a streaming assessment protocol to rigorously evaluate memory performance, revealing critical bottlenecks like memory saturation in existing architectures.

Original authors: Yihao Wang, Haoran Xu, Renjie Gu, Yixuan Ye, Xinyi Chen, Xinyu Mu, Yuan Gao, Chunxiao Guo, Peng Wei, Jinjie Gu, Huan Li, Ke Chen, Lidan Shou

Published 2026-05-13

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Yihao Wang, Haoran Xu, Renjie Gu, Yixuan Ye, Xinyi Chen, Xinyu Mu, Yuan Gao, Chunxiao Guo, Peng Wei, Jinjie Gu, Huan Li, Ke Chen, Lidan Shou

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Super-Doctor" with a Bad Memory

Imagine you have a personal AI health assistant. Its job is to be your doctor for life, tracking your health from childhood to old age. It needs to remember everything: your allergies, your blood pressure trends, the medications you took last year, and even that you hate the taste of strawberries.

The problem? Current AI assistants are like students who study for a test, take it, and then immediately forget everything. They are great at chatting about the weather, but when it comes to your complex, long-term health history, they get confused, mix up facts, or forget critical details like "I'm allergic to penicillin."

The authors of this paper built a giant stress-test called MedMemoryBench to see how well these AI "doctors" actually remember things in a realistic medical setting. They found that most current AI memory systems are failing the test, especially when the amount of information gets huge.

Why Existing Tests Didn't Work

Think of previous memory tests like a game of "Simon Says" with simple instructions: "Remember the color red." or "Remember the word 'apple'."

But real life isn't that simple. Real life is like a 10-year-long mystery novel where:

Details matter: "Right lower abdominal pain" is very different from "diffuse abdominal pain." One means appendicitis; the other might be gas.
Time matters: Your blood sugar wasn't 7.0 last month; it was 9.8 today. The AI needs to know the trend, not just the number.
Noise matters: In real life, you don't just talk about your diabetes. You also ask about your dad's flu, your favorite football team, and how to lose weight. The AI gets flooded with "junk" data that distracts it from the important stuff.

The paper argues that old tests didn't check if the AI could handle this messy, long-term, noisy reality.

How They Built the Test (The "Medical Simulator")

To create a fair test, the researchers didn't just ask random questions. They built a virtual hospital using a team of humans and AI agents.

The Patient Avatar: They created 20 detailed "virtual patients" with chronic diseases (like diabetes or sleep apnea). Each patient has a full year of medical history, including fake but realistic doctor visits, lab results, and lifestyle changes.
The "Trap" Events: They hid "landmines" in the data. For example, they made sure a patient had a severe allergy mentioned in Session 1. If the AI forgets this by Session 50 and suggests a drug that kills the patient, it fails.
The "Noise" Injection: They added extra conversations that had nothing to do with the main disease (like asking about a spouse's cold). This simulates the real world where an AI has to filter out the noise to find the signal.
The "Streaming" Rule: This is the most important part. They didn't give the AI the whole book at once. They fed the AI the story page by page. After every 10 pages, they stopped and asked a question. The AI had to answer using only what it had read so far. This mimics how a real AI works in production.

The Result: A massive dataset of 2,000 simulated doctor visits and 16,000 conversation turns, all checked by real medical experts to ensure the medicine was accurate.

The Big Discovery: "Memory Saturation"

The paper discovered a phenomenon they call Memory Saturation.

The Analogy: Imagine a librarian (the AI) trying to find a specific book in a library.

Early on: The library has 100 books. The librarian finds the right one easily.
Later: The library grows to 10,000 books. But here's the catch: 9,000 of them are almost identical copies of the same book, or they are about completely different topics.
The Problem: The librarian gets overwhelmed. They start grabbing the wrong books because there are too many similar-looking ones on the shelf. The more books they have, the worse they get at finding the right one.

The paper found that as the AI's memory grew larger and filled with "noise" (irrelevant info), its ability to reason and answer correctly dropped significantly. It wasn't just that it forgot; it was that the extra memory confused it.

What They Tested and What They Found

They tested many different types of AI memory systems (some that use graphs, some that just list things, some that use reinforcement learning).

The "Reasoning" Gap: The AI was okay at simple facts (e.g., "What is the patient's name?"). But when asked to connect the dots (e.g., "Based on the patient's weight gain last month and their current blood sugar, what should we do?"), almost all of them failed.
Retrieval is the Bottleneck: The main reason they failed wasn't that the AI couldn't "think"; it was that they couldn't find the right memory. They were drowning in their own notes.
The "Letta" Winner: One system called Letta performed the best. Why? Because it keeps a "Core Memory" (like a sticky note on the AI's forehead) that always shows the most critical info, so it doesn't have to search through the whole library every time.
Cost vs. Benefit: Some complex systems were very expensive to run (using a lot of computer power) but didn't perform much better than simpler ones.

The Conclusion

The paper concludes that we cannot just "plug and play" current AI memory systems into healthcare. They are too fragile. If you feed them too much data, they get confused and make dangerous mistakes.

MedMemoryBench is a new tool that forces developers to build AI that can:

Remember critical safety details (like allergies) perfectly.
Track changes over time without getting lost.
Ignore the "junk" conversations to focus on the patient's actual health.

It's a wake-up call: Before we let AI manage our health, we need to teach them how to be better librarians in a chaotic, noisy library.

Technical Summary: MedMemoryBench

Problem Statement

The large-scale deployment of personalized healthcare agents necessitates memory mechanisms that are exceptionally precise, safe, and capable of long-term clinical tracking. However, existing benchmarks (e.g., LoCoMo, LongMemEval) primarily focus on daily open-domain conversations or static long-context processing. They fail to capture the high-stakes complexity of real-world medical applications, which impose unique constraints:

Safety-Critical Personalization: Clinical data is risk-heterogeneous; life-threatening details (e.g., severe allergies, contraindications) require zero-fault retention, unlike general conversational facts.
Longitudinal Clinical Progression: Agents must track temporal evolution (e.g., shifting blood glucose levels) over months or years, rather than simply overwriting short-term states.
Streaming Memory Integration: In production, memory accumulates incrementally, with each session immediately shaping the next. Traditional offline, one-shot evaluations fail to capture this continuous dynamic.
Noisy Memory Accumulation: Real-world interactions involve redundant queries and proxy consultations, leading to "memory saturation," where sustained information influx degrades retrieval and reasoning robustness.

Current methods, validated mostly in general domains, remain insufficiently examined in large-scale, high-risk medical dialogue scenarios.

Methodology

Data Construction Pipeline

MedMemoryBench introduces a human-agent collaborative pipeline to synthesize highly realistic, long-horizon medical trajectories based on clinically grounded, synthetic patient archetypes. The process involves four stages:

Patient Profile Construction: 20 representative chronic-disease profiles (e.g., diabetes, cardiovascular, respiratory) are created from de-identified real cases, expanded with LLM assistance for diversity while preserving privacy.
Disease Progression Event Generation: Expert-verified, year-long diagnostic trajectories are generated, organized into temporally and causally linked event graphs. Crucially, "trap events" (high-priority facts like allergies or economic constraints) are injected to test safety-critical retention.
Multi-turn Sessions Simulation: Patient and physician agents simulate interactions following the event graph. Memory summaries are recorded after each session to maintain long-term coherence.
Memory Extraction and Query Construction: A multi-agent framework (Trap Agent, Query Agent, Critique Agent) constructs query-answer pairs. Medical experts verify trajectories, dialogues, and QA pairs for clinical plausibility and terminology accuracy.

The dataset comprises approximately 2,000 sessions and 16,000 interaction turns (expanding to ~42,000 turns in the "Mixed" noise setting), totaling ~20.74 million tokens.

Evaluation Framework

MedMemoryBench departs from static evaluation by pioneering an evaluate-while-constructing streaming assessment protocol.

Protocol: Memory methods receive sessions strictly in chronological order. Checkpoints are triggered every $\Delta=10$ sessions. Agents must answer queries using only the memory accumulated up to that point, preventing future information leakage.
Settings:
- Efficient Mode: Interactions focus solely on the patient's own health.
- Mixed Mode: Injects "noise" via general health consultations and proxy consultations for family members to simulate memory saturation.
Query Types: Six types spanning three capability bands:
- Precise Extraction: Entity Exact Match (EEM), Temporal Location Accuracy (TLA).
- Temporal Resolution: State Update Accuracy (SUA).
- Reasoning: Multiple Choice (MQ), Inference Generation (IG), Multi-hop Clinical Deduction (MCD).
Metrics: Binary scoring for accuracy. Additional metrics include Recall, F1, ROUGE-L, and Hallucination Rate (measuring confident but unsupported answers).

Key Contributions

MedMemoryBench Benchmark: A specialized benchmark for evaluating agent memory in personalized healthcare, built through a four-stage human-agent pipeline with expert medical verification.
Streaming Evaluation Protocol: A novel "evaluate-while-constructing" approach that assesses memory as it accumulates, mirroring production dynamics, alongside the first systematic study of memory saturation in healthcare agents.
Comprehensive Benchmarking: An evaluation of mainstream memory architectures (Unstructured, Graph-Structured, Flat-Extracted, Multi-Store) revealing fundamental limitations in complex medical reasoning and noise resilience.

Experimental Results

Experiments were conducted using GPT-5.1 as the backbone and Claude-4-Sonnet as the judge. Key findings include:

Performance Gaps: All methods perform poorly on complex medical reasoning tasks (IG and MCD) that require combining multiple memories. Retrieval remains the primary bottleneck.
Memory Granularity: Methods with medium-grained memories (e.g., A-Mem, MemOS, Letta) generally outperform those with highly fragmented (Mem0) or overly dense (MIRIX) entries.
Retrieval Bottleneck: Even methods with rich memory stores fail to reliably retrieve relevant evidence from large pools, particularly for multi-hop reasoning.
Memory Saturation: Under the Mixed (noisy) setting, all methods show significant performance degradation. Long-term memory without adequate filtering often harms overall performance, as redundant information floods the retrieval space.
Safety Risks: High hallucination rates were observed; when memory support is insufficient, models tend to provide confident but wrong answers rather than abstaining.

Significance

MedMemoryBench establishes a vital foundation for developing robust, production-ready medical agents. By exposing fundamental flaws in current architectures—specifically regarding complex medical reasoning, noise resilience, and the phenomenon of memory saturation—the benchmark provides a critical tool for future research. It highlights that general-purpose memory systems are insufficient for the stringent, safety-critical, and longitudinal demands of personalized healthcare, urging the development of specialized memory mechanisms that prioritize precision and safety over general conversational fluency.