💬 NLP

Beyond RAG for Agent Memory: Retrieval by Decoupling and Aggregation

This paper introduces xMemory, a novel agent memory framework that improves upon standard RAG by decoupling interaction histories into reusable components and aggregating them hierarchically to enable top-down retrieval that reduces redundancy while preserving critical details, thereby enhancing answer quality and inference efficiency.

Original authors: Zhanghao Hu, Qinglin Zhu, Runcong Zhao, Di Liang, Hanqi Yan, Yulan He, Lin Gui

Published 2026-05-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhanghao Hu, Qinglin Zhu, Runcong Zhao, Di Liang, Hanqi Yan, Yulan He, Lin Gui

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Problem: The "Noisy Library" vs. The "Smart Librarian"

Imagine you are an AI assistant (an "Agent") talking to a human over many months. You have a massive memory of every conversation you've ever had.

The Old Way (Standard RAG):
Think of your memory as a giant, chaotic library where every book is just a raw transcript of a conversation. When the user asks a question, the "Librarian" (the AI) grabs the top 5 books that have the most similar words to the question.

The Flaw: In a long conversation, people often repeat themselves or talk about the same topic in slightly different ways. If you ask, "When did I lose my job?", the old system might grab five different pages from five different days where you mentioned losing your job. It gives you a pile of redundant, confusing information. It's like asking for a specific date, and the librarian hands you five different calendars that all say "January" but don't tell you which January.

The New Way (xMemory):
The authors argue that agent memory isn't like a library of random books; it's more like a continuous stream of events where details are tightly packed together. To fix this, they propose a new system called xMemory.

The Core Idea: "Decoupling Before Aggregation"

The paper suggests a two-step process: Decouple (separate the important bits) and then Aggregate (organize them).

1. Decoupling: The "Fact Extractor"

Instead of treating a whole conversation as one big block, xMemory acts like a skilled editor. It reads a conversation and pulls out the specific, reusable facts, updates, and details, separating them from the "fluff" (like "Hello," "How are you," or repeated greetings).

Analogy: Imagine you have a messy desk covered in sticky notes. Some notes say "Meeting at 2 PM," others say "Meeting at 2 PM (confirmed)," and others say "Meeting at 2 PM (cancelled)."
- The Old Way grabs the whole pile of sticky notes.
- xMemory peels off just the single, most accurate note: "Meeting at 2 PM." It isolates the truth from the noise.

2. Aggregation: The "Smart Filing System"

Once the facts are isolated, xMemory doesn't just throw them in a bucket. It organizes them into a hierarchy:

Segments: Small groups of related events (like a specific conversation topic).
Components: The specific facts extracted from those segments (e.g., "User lost job at DoorDash in Jan").
Groups: High-level folders that hold related components (e.g., "User's Employment History").

Crucially, this system is revisable. If you have a new conversation that clarifies an old fact, xMemory can go back, split a messy folder, or merge two small ones to keep the organization perfect. It's like a filing cabinet that reorganizes itself automatically when you add a new document.

How It Finds Answers: The "Top-Down Detective"

When you ask a question, xMemory doesn't just search for keywords. It uses a Top-Down approach:

Stage 1: The Skeleton Search. It first looks at the high-level "Groups" and "Components" (the facts). It picks the most relevant "backbone" of evidence. It asks: "Do I have the right facts to answer this?"
Stage 2: The Uncertainty Check. Only if the facts aren't clear enough does it dig deeper. It expands to the original conversation segments and raw messages only if it needs more detail to reduce its "uncertainty."
- Analogy: Imagine a detective solving a crime.
  - Old Way: The detective grabs 20 witness statements that all sound vaguely similar and reads them all.
  - xMemory: The detective first checks the "Case File Summary" (the Group). If the summary says "Suspect was at the bank," the detective stops there. But if the summary is vague, the detective then pulls the specific transcript of the witness (the Segment) to get the exact time.

Why This Matters (The Results)

The paper tested this system on two datasets (LoCoMo and PerLTQA) using different AI models. Here is what they found:

Better Answers: Because xMemory separates the "decisive evidence" from the "redundant noise," the AI gives more accurate answers, especially for tricky questions about time or specific changes in a user's life.
Cheaper & Faster: The system uses fewer "tokens" (the currency of AI processing). Instead of feeding the AI a huge, messy block of text, it feeds it a tight, concentrated package of facts.
Efficiency: It retrieves "denser" evidence. Instead of finding 10 pages that say "I like pizza," it finds the one specific note that says "I like pepperoni pizza."

Summary in One Sentence

xMemory fixes the problem of AI getting confused by its own long conversations by first separating the important facts from the noise, organizing them into a self-updating hierarchy, and then retrieving only the exact amount of detail needed to answer a question, saving time and money while getting the answer right.

Technical Summary: Beyond RAG for Agent Memory: Retrieval by Decoupling and Aggregation

Problem Statement

Standard Retrieval Augmented Generation (RAG) systems are fundamentally mismatched with the requirements of agent memory. While standard RAG is designed for large, heterogeneous document corpora, agent memory consists of bounded, coherent interaction streams where many spans are highly correlated or near-duplicates. In this context, flat top- $k$ similarity retrieval often returns redundant context, failing to distinguish decisive evidence hidden among similar histories. Conversely, existing hierarchical memory strategies that rely on progressive compression (e.g., summaries) often blur the subtle details necessary to differentiate between candidates, such as small updates, constraints, or factual differences. The core challenge is not merely locating relevant text, but isolating specific, answer-critical evidence from highly similar interaction logs.

Methodology: xMemory

The authors propose xMemory, a framework that couples memory structuring with adaptive retrieval based on the principle of decoupling before aggregation. Unlike traditional approaches that treat long interaction logs as monolithic units, xMemory decomposes history into fine-grained evidence units before organizing them.

1. Hierarchical Memory Construction

xMemory constructs a revisable hierarchical structure through three stages:

Segmentation: Raw messages are partitioned into contiguous segments, each capturing a coherent local event. This preserves intact local context which is often lost in isolated message retrieval.
Decoupling: Each segment is decomposed into memory components. These components are concise, structured units representing reusable facts, constraints, attributes, relations, or state updates. This step isolates distinctive evidence from the surrounding narrative.
Aggregation: Related components are grouped into high-level groups. This aggregation is guided by an objective function balancing sparsity (favoring balanced group sizes to minimize scanning costs) and semantic faithfulness (ensuring intra-group coherence and navigable inter-group geometry).

2. Dynamic Maintenance

The memory structure is not static; it evolves incrementally as new interactions arrive.

Attachment: New components are attached to the nearest compatible group based on embedding similarity.
Revisability: The system supports split and merge operations. Overly large or incoherent groups are split, while small or isolated groups are merged. These operations are triggered when they improve the guidance objective, allowing the system to retroactively reorganize previously stored components based on new evidence.

3. Top-Down Adaptive Retrieval

Retrieval proceeds in two stages to minimize redundancy while ensuring sufficient context:

Stage I: High-Level Backbone Selection: The system first selects a compact set of complementary groups and components. It utilizes a greedy selection strategy that trades off query relevance against structural coverage, leveraging pre-maintained $k$ -nearest neighbor (kNN) links among groups to avoid selecting near-duplicate evidence.
Stage II: Adaptive Text Expansion: Only after the high-level backbone is established does the system expand to lower-level text (segments and raw messages). This expansion is uncertainty-driven: a segment or message is included only if it provides a positive marginal reduction in the reader's uncertainty (measured via predictive entropy). Retrieval stops when no further uncertainty reduction is possible.

Key Contributions

Problem Identification: The paper identifies the mismatch between standard RAG assumptions and the nature of agent memory, arguing that agent memory requires a "decoupling before aggregation" principle.
xMemory Framework: The proposal of a framework that constructs a revisable hierarchical memory structure by segmenting history, decoupling segments into memory components, and aggregating components into high-level groups.
Adaptive Retrieval: The development of a top-down retrieval method that selects a compact evidence backbone before expanding to raw text based on uncertainty reduction, demonstrated to improve both answer quality and inference token efficiency.

Experimental Results

The authors evaluated xMemory on two benchmarks: LoCoMo (multi-session dialogue) and PerLTQA (personalized long-term memory), using diverse open-source (Qwen3-8B, Llama-3.1-8B) and closed-source (GPT-5 nano) LLMs.

Answer Quality: xMemory consistently achieved the highest average BLEU and F1 scores across all backbone models and datasets. For instance, on LoCoMo with Qwen3-8B, it improved average BLEU/F1 from 31.44/40.88 (Nemori baseline) to 34.48/43.98.
Efficiency: xMemory demonstrated superior token efficiency. On LoCoMo with Qwen3-8B, it reduced inference token usage from 7,755 (Nemori) to 4,711 while simultaneously improving answer quality.
Evidence Density: Analysis showed that xMemory retrieves denser evidence, with higher proportions of "multi-hit" blocks (containing multiple distinct answer evidence units) compared to Naive RAG and pruning-based baselines.
Cost-Performance Trade-off: When accounting for both memory construction and inference costs, xMemory achieved the most favorable balance, outperforming structured memory baselines in quality while maintaining lower total token costs.

Significance and Claims

The paper claims that effective agent memory should not merely retrieve or compress past interactions but must reorganize them into evidence-oriented structures that make subtle distinctions accessible. By disentangling highly correlated memories into semantic components and retrieving top-down, xMemory reduces redundancy while preserving temporally linked evidence. The authors posit that their work provides a step toward more adaptive and efficient memory systems for long-horizon LLM agents, addressing the specific retrieval challenges posed by coherent, overlapping interaction streams that standard RAG fails to handle.

The authors remain modest regarding limitations, noting that the evaluation is confined to specific benchmarks (LoCoMo and PerLTQA) and that the reliance on LLM-generated components introduces potential extraction errors. They suggest future work should explore broader agentic settings and more robust extraction mechanisms.