💬 NLP

MemUse: Moving Memory Evaluation from Direct QA to Natural Integration in Long-Term Human-AI Conversation

This paper challenges the validity of traditional Direct QA benchmarks for evaluating conversational LLM memory by demonstrating that while such metrics fail to correlate with user satisfaction, a new framework called MemUse, which assesses the natural integration of recalled facts into dialogue, successfully predicts user experience in long-term human-AI interactions.

Original authors: Ryuichi Sumida, Koji Inoue, Tatsuya Kawahara

Published 2026-08-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ryuichi Sumida, Koji Inoue, Tatsuya Kawahara

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the quiet hours of a long conversation, a human listener does more than just store facts; they weave the past into the present. When a friend mentions a trip to a city they visited years ago, a good listener does not simply retrieve a file labeled "Vienna" and recite it back. Instead, they might recall the specific café you liked there, or the reason you were traveling, and weave that detail naturally into their reply. This ability to integrate memory into the flow of conversation is what makes a relationship feel real. For years, developers of artificial intelligence have tried to build this same capability into their machines. They have created systems designed to remember everything a user has ever said, hoping that a machine with a perfect memory would feel more like a true companion. To test if these systems work, researchers have traditionally asked them direct questions, much like a teacher quizzing a student: "What color was the car you mentioned last week?" If the machine answers correctly, the test is passed, and the system is deemed successful. But this method assumes that the ability to answer a direct question is the same as the ability to remember something naturally when it matters.

A team of researchers at Kyoto University decided to see if this assumption held true in the real world. They spent four months observing how forty real people interacted with an AI diary companion named Luke. The users wrote about their daily lives, their worries, and their dreams, treating the AI as a trusted confidant. During this time, the researchers tested seven different ways the AI could remember things. Some versions only gave the AI a brief summary of past chats, while others allowed the AI to read through thousands of pages of past conversation or search a database of old memories. The goal was to see if giving the AI more memory capacity made the users happier. The results were surprising. The AI became much better at answering direct questions about the past as its memory grew, yet the users' satisfaction with the conversation did not change at all. The machine could recall facts when asked, but it failed to use those facts to make the conversation feel more personal or connected.

To understand why this happened, the researchers looked closer at the specific moments when users actually tried to use the AI's memory. They found that these moments were rare, occurring in only about one out of every seventy user messages. When a user did bring up a past topic, the researchers measured two things: could the AI answer a direct question about that topic, and did the AI naturally bring that topic up in its response? They discovered a massive gap between the two. In the best-performing memory condition, the AI could answer direct questions correctly nearly eighty percent of the time. However, when the user mentioned a past event in a normal sentence, the AI only wove that memory into its reply about eight percent of the time. The machine was holding the information, but it was not using it to build a connection. It was like a librarian who could find any book on a shelf when asked, but never recommended a book to a reader unless they specifically asked for it.

The study showed that the ability to retrieve a fact and the ability to integrate it into a conversation are two different skills. The researchers found that when the AI successfully integrated a memory into a natural response, the user's satisfaction with that session went up. But when the AI simply answered a direct question correctly, it did not make the user feel any better. In fact, the most advanced memory systems, which could recall the most facts, were often the ones that failed to use those facts naturally. The AI would give a generic, polite response that ignored the user's hint, even though it had the answer right in front of it. This suggests that the problem is not about how much the machine can remember, but how it chooses to speak. The bottleneck is not in finding the memory, but in the act of conversation itself. The machine struggles to decide when to bring up a past detail and how to do so without sounding robotic or forced.

The researchers also looked at times when the AI tried to remember things on its own, without the user asking. They found that when the AI brought up a past topic at the wrong time, it actually made the user less happy. If the machine interrupted a user's current thought to mention an old memory, the conversation felt awkward and the user's satisfaction dropped. This suggests that for an AI to be a good companion, it needs to learn not just to remember, but to listen. It needs to understand the rhythm of a conversation and know when a memory is relevant to the moment. The study concludes that simply making AI smarter at answering questions is not enough to make it a better friend. To truly improve the experience, we need to measure how well the AI weaves the past into the present, rather than just how well it can recite the past when tested. The future of human-AI relationships depends on this shift from simple recall to natural, thoughtful integration.

Technical Summary: MemUse – Moving Memory Evaluation from Direct QA to Natural Integration

Problem Statement

Current evaluation paradigms for memory systems in conversational Large Language Models (LLMs) rely heavily on Direct QA: asking the model specific, fact-seeking questions about prior dialogue to measure recall accuracy. Benchmarks such as LoCoMo, LUFY, and RealTalk operate under the structural assumption that elicited retrieval (recalling facts when explicitly asked) correlates with natural integration (detecting relevance and weaving prior context into a conversational response).

The authors challenge this assumption, hypothesizing that high Direct QA accuracy does not necessarily translate to higher user satisfaction in real-world, long-term human-AI interactions. They posit that existing benchmarks measure a capability (recall-on-demand) that is distinct from the capability required for satisfying conversation (contextual integration).

Methodology

1. Longitudinal Deployment

The authors conducted a 4-month longitudinal study involving 40 users interacting daily with an AI diary companion ("Luke," built on GPT-4.1-mini). The study generated 1,872 sessions (21,575 turns) across 7 distinct memory conditions:

Summary-only: Baseline providing only a summary of prior conversations.
LC-k% (Long Context): Prepending the top $k\%$ (10%, 50%, 100%) of most important prior turns to the context.
RAG-k% (Retrieval-Augmented Generation): Retrieving the top-10 turns from a vector index built over the top $k\%$ pool.
Note: Importance scoring was performed by a RoBERTa model fine-tuned on human annotations from the LUFY dataset.

Users rated each session on a 1–7 satisfaction scale. The study filtered for engagement quality, resulting in a final analysis set of 1,270 sessions from 29 users.

2. The MEMUSE Benchmark

To investigate the gap between retrieval and integration, the authors introduced MEMUSE, a benchmark derived directly from the deployment corpus rather than synthetically authored.

Detection: The system identified 147 "memory moments" (1.4% of user turns) where users explicitly cued memory via probes ("Do you remember X?"), re-provisions ("As I mentioned..."), or reactions to system recalls.
Composition: The benchmark consists of 72 reactive instances (user-cued moments).
Scoring Metrics: For each instance, the system is evaluated on:
1. Direct QA: Can the model answer fact-seeking questions about the referenced topic when explicitly asked? (Standard benchmark format).
2. Reference: Do the facts retrieved in Direct QA actually appear in the model's natural conversational response?
3. Natural Integration (Primary Metric): A binary judgment (via LLM judge) of whether the model's natural response to the user's cue successfully demonstrates memory of the referenced topic.

Key Results

1. The Retrieval–Integration Gap

The study reveals a stark dissociation between retrieval capability and conversational integration:

Direct QA Accuracy: Increased significantly with memory capacity, rising from 19.7% (Summary-only) to 70.1% (LC-100%).
Natural Integration: Remained flat across all conditions, ranging only between 22.2% and 27.8%.
The Gap: In the LC-100% condition, the model answered 78.8% of MEMUSE Direct QA questions correctly but referenced those facts in its natural response only 7.9% of the time. This 71-point gap persisted across different model architectures (GPT-4.1-mini, GPT-5.5, Gemini 3.1 Pro) and memory strategies (Mem0, Letta).

2. User Satisfaction Correlations

Capacity vs. Satisfaction: Increasing memory capacity (LC/RAG) did not improve user satisfaction. The mean difference in satisfaction between any condition and the Summary-only baseline was less than 0.06 within-user standard deviations.
Integration vs. Satisfaction: Within the subset of sessions containing user-cued memory moments, Natural Integration was positively associated with satisfaction ( $\rho = +0.29, p = .046$ ). Successful integration correlated with a +0.56 SD increase in satisfaction.
Direct QA vs. Satisfaction: Direct QA accuracy showed no correlation with satisfaction ( $\rho = +0.03$ ).
Proactive Recall: System-initiated (proactive) recalls showed no positive satisfaction signal. However, mistimed proactive recalls (where the system interrupts the user's current agenda) resulted in a significant negative signal ($-0.48$ SD).

3. Failure Modes

Analysis of the LC-100% condition revealed that when the model failed to integrate:

38.9% of responses were generic/empathetic, ignoring the context.
23.6% hallucinated details.
26.4% showed partial (gist-only) recall.
Only 8.3% fully integrated prior details.
The authors identify the bottleneck as conversational generation, not retrieval. Even when a "Two-step" prompt intervention successfully extracted ground-truth details, the downstream generation failed to reference them 77% of the time.

Contributions and Significance

The paper makes three primary contributions:

Empirical Evidence of Dissociation: It provides data from a real-world deployment showing that Direct QA accuracy and user satisfaction are uncorrelated, challenging the prevailing assumption that benchmark recall scores predict user value.
The MEMUSE Benchmark: It introduces a new evaluation standard based on real user-cued moments scored by Natural Integration, offering a metric that aligns with user experience rather than just fact-checking.
Identification of the Bottleneck: It demonstrates that the gap between retrieval and integration is a generation-level failure, not a retrieval limitation, and that current capacity-scaling strategies (longer contexts, RAG) do not solve this integration problem.

Significance:
The authors argue that memory evaluation must shift from "elicited retrieval" to "natural integration." They conclude that benchmarks must be redesigned to score how well a model weaves retrieved information into a natural response under user-cued conditions, as this is the capability that actually drives user satisfaction in long-term companions. The paper releases the full deployment corpus, the MEMUSE benchmark, and all scoring prompts to facilitate this realignment in the field.