🤖 AI

The Retriever Should Remember: Experience-Amortized Reranking for Long-Term Agent Memory

This paper introduces EARM, an experience-amortized reranking framework that learns to reuse previously acquired LLM relevance scores via causal matrix completion to significantly improve long-term agent memory retrieval accuracy while substantially reducing inference overhead.

Original authors: Qi Feng, Chris Ding, Jicong Fan

Published 2026-08-25

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Qi Feng, Chris Ding, Jicong Fan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a conversation that stretches over months or years, where a person asks questions about events that happened long ago, or pieces together facts scattered across dozens of previous exchanges. For a computer program designed to act as a helpful assistant, keeping track of this vast history is a monumental task. These programs, often called agents, store their past interactions in an external library of memories. However, having a library is not the same as knowing how to find the right book. When a new question arrives, the system must quickly sift through thousands of stored memories to find the few that are actually useful. Traditionally, the system uses a quick, broad search based on word similarity, and if it needs to be more precise, it asks a powerful language model to read the question and the candidate memories to judge their relevance. But here lies a frustrating inefficiency: every time a new question is asked, the system starts from scratch. It forgets the lessons it learned from the last time it searched, discarding the scores it calculated and the patterns it noticed, as if it were a student who studies for a test, forgets everything immediately after, and then has to relearn the same material for the next exam.

Researchers at The Chinese University of Hong Kong, Shenzhen, have proposed a way to break this cycle of forgetting. They introduced a new framework called EARM, which stands for Experience-Amortized Reranking for Memory. The core idea is simple yet profound: the system should not just remember the content of past conversations, but also remember how it successfully found information in the past. Instead of treating the judgment of relevance as a one-time cost, the researchers designed a system that treats these judgments as a form of experience that accumulates over the agent's lifetime. By keeping a record of which memories were useful for which types of questions, the system learns a map of its own memory. This map allows it to predict the relevance of new memories without having to ask the powerful language model to read every single one.

In their experiments, the researchers tested this approach on a dataset of long conversations containing complex questions that required the agent to connect information from different times. They compared their new method against two standard approaches: a basic system that relies only on word similarity, and a more powerful system that asks the language model to score every single candidate memory. The results showed that the new method, which uses a mix of direct scoring and learned predictions, significantly outperformed the basic system. It improved the accuracy of the answers by up to 6.62 percent, a substantial gain in a field where small improvements are hard to come by. Perhaps more importantly, the system remained highly effective even when it was only allowed to directly score a tiny fraction of the candidates—specifically, just 17.5 percent. In this sparse setting, the system used its accumulated experience to fill in the gaps, estimating the relevance of the remaining memories based on the patterns it had learned from previous interactions.

The mechanism behind this success relies on the idea that relevance is not random. Certain memories tend to be useful for specific kinds of questions, and certain questions tend to trigger similar patterns of relevance across different memories. The researchers built a system that records these connections in a structured way. When a new question arrives, the system selects a small, strategic sample of memories to have the language model score directly. It then uses the history of past scores to estimate the relevance of the rest. This process is similar to how a seasoned librarian might know that a patron asking about a specific historical event likely needs a particular set of documents, even if the patron hasn't mentioned them yet. The system does not guess blindly; it uses the "experience" of past retrievals to guide its search, effectively learning how to access its own knowledge base.

One of the most striking findings was that the system could recover most of the benefits of a full, exhaustive search while using only a quarter of the computational effort. By the time the system had processed enough questions to build a robust history, it could make highly accurate decisions with very little new input. The researchers found that the memories selected through these learned estimates were not just filler; they added a measurable amount of value, improving the final answer quality by an additional 0.78 to 2.79 percent beyond what the directly scored memories provided on their own. This suggests that the system was successfully identifying useful information that the initial, quick search had missed, simply by recognizing the underlying structure of its own memory.

The study also highlighted a shift in how we might think about artificial intelligence memory. Traditionally, the focus has been on storing more information or representing it better. This work suggests that a truly long-lived agent must also learn how to retrieve that information efficiently. The researchers argue that an agent should not just accumulate content, but also accumulate experience about how that content has proven useful. By treating retrieval scores as a persistent state rather than a disposable byproduct, the system transforms the act of searching from a repetitive expense into a learned capability. This approach allows the agent to become smarter over time, not just by knowing more, but by getting better at finding what it already knows.

The researchers acknowledge that this method relies on certain conditions to work effectively. It assumes that the questions asked over time share some common themes or structures, and that the memories themselves remain stable. If every question were completely unrelated to the last, or if the meaning of the stored memories changed rapidly, the system would struggle to find patterns to learn from. However, in the context of long-term conversations where users often return to similar topics or build on previous ideas, the method proved robust. The system was able to maintain its advantage even as the number of direct scores dropped, showing that the accumulated experience was a reliable guide.

Ultimately, this research offers a new perspective on the efficiency of artificial intelligence. It demonstrates that the cost of making a system smarter does not have to grow linearly with the amount of information it holds. By allowing the system to remember its own search history, the researchers have shown that it is possible to build agents that are both knowledgeable and efficient. The system does not need to re-evaluate everything from scratch every time; it can stand on the shoulders of its past decisions. This approach moves the field closer to creating agents that can truly operate over long periods, adapting and refining their behavior as they interact with the world, much like a human who learns not just from what they experience, but from how they navigate their own history.

Technical Summary: Experience-Amortized Reranking for Long-Term Agent Memory (EARM)

Problem Statement

Long-lived language model (LLM) agents accumulate vast amounts of memory across interactions, yet their retrieval mechanisms often fail to learn from past retrieval experiences. Current systems typically rely on semantic retrieval (embedding similarity) for efficiency, followed by optional LLM reranking to assess query-conditioned relevance. However, this reranking process is stateless: for every new query, the system scores a large candidate pool and discards the relevance scores immediately after use.

This creates a fundamental asymmetry: while the memory store evolves, the retrieval procedure resets to an initial state for every query. Semantic similarity alone is insufficient because representational closeness does not guarantee evidentiary relevance; a memory can be topically similar yet irrelevant, or distant yet critical for temporal or causal reasoning. Furthermore, performing full LLM reranking on all candidates for every query incurs a prohibitive inference cost ( $T \times N$ operations), which scales linearly with the number of interactions and the size of the memory pool.

Methodology: EARM Framework

The authors propose EARM (Experience-Amortized Reranking for Memory), a framework that treats LLM relevance scores not as disposable inference traces, but as persistent retrieval experience. EARM learns a "latent memory accessibility map" to estimate relevance for unobserved query-memory pairs, thereby amortizing the scoring cost over the agent's lifetime.

The framework operates through five key components:

Semantic Candidate Generation: A standard semantic retriever generates a high-recall candidate set $C_t$ of size $N$ for each incoming query $q_t$ .
Retrieval Experience Matrix: EARM maintains a sparse matrix $R(t)$ where rows represent persistent memory identities and columns represent queries. Observed entries contain LLM relevance scores; unobserved entries (gray or white cells) represent pairs that were either retrieved but not scored, or available in the store but omitted from the candidate set.
Stage-Wise Experience Acquisition: To manage the inference budget, EARM employs a stage-wise schedule where the observation ratio (the fraction of candidates directly scored by the LLM) decreases as the agent gains experience (e.g., from 100% in early stages to 17.5% in later stages). For each query, a subset $O_t$ of size $B_t$ is selected for direct scoring using a stratified strategy: half the budget samples from high-similarity candidates (exploitation), and half samples from the remaining pool (exploration) to ensure matrix coverage.
Causal Relevance Completion: The missing entries in the matrix are estimated using a bias-aware low-rank model. The model decomposes relevance into:
- A global relevance level ( $\mu$ ).
- Memory-specific bias ( $\alpha_i$ ) and query-specific bias ( $\beta_\tau$ ).
- Latent interaction factors ( $p_i^\top z_\tau$ ) capturing shared relevance patterns.
  The model is optimized via Alternating Least Squares (ALS) with warm starts, minimizing masked reconstruction error on observed entries. Crucially, the model is causal: it uses only historical data and current anchors to predict future scores, avoiding data leakage from future queries.
Mixed Reranking: The final ranking combines directly observed LLM scores for the sampled subset with estimated scores for the remaining candidates (and potentially memories outside the initial semantic pool). The top $K$ memories are selected, restored to chronological order, and passed to the answer generator.

Key Contributions

The paper makes two primary contributions:

The EARM Framework: A novel approach that maintains LLM relevance scores as persistent retrieval state. It integrates stratified anchor scoring, causal low-rank matrix completion, and mixed observed-estimated ranking to handle sequential query streams.
Empirical Validation: Demonstration that EARM significantly improves answer accuracy over standard semantic retrieval while drastically reducing LLM inference overhead. The framework remains effective even when the direct-scoring ratio drops to 17.5%, proving that accumulated retrieval experience can substitute for direct scoring.

Experimental Results

The framework was evaluated on the LoCoMo dataset (long-term conversational memory) using a pipeline where a semantic retriever selects candidates, EARM reranks them, and GPT-4o-mini generates and evaluates answers.

Accuracy Gains: Compared to semantic retrieval, EARM improved overall answer accuracy by up to 6.62% (Top-10 retrieval) and 2.99% (Top-20 retrieval).
Proximity to Full Reranking: EARM configurations recovered approximately 70% of the performance gain offered by full LLM reranking (which scores all 200 candidates) while using only 25.57% of the LLM calls (a 74.43% reduction in cost).
Robustness to Sparse Scoring: Even when the direct-scoring ratio fell to 17.5% (only 35 of 200 candidates scored), EARM maintained a significant accuracy advantage over semantic retrieval (e.g., +5.53% for Top-10).
Component Ablation: The study showed that the improvement is not solely due to the small set of directly scored memories. The "completed" scores (estimated via matrix completion) contributed an additional 0.78% to 2.79% in accuracy over the "observed-only" baseline, confirming that the learned structure adds value beyond the sampled anchors.

Significance and Claims

The paper argues for a paradigm shift in how agent memory is conceptualized. Traditionally, memory systems focus on content memory (what the agent has experienced). EARM introduces the concept of retrieval memory (what the agent has learned about accessing that experience).

The authors claim that a long-lived agent should not only remember past content but also remember how that content has proved useful for retrieval. By treating relevance scores as reusable experience rather than transient inference outputs, EARM transforms reranking from a repeated per-query expense into a capability that is learned and refined over the agent's lifetime. This approach addresses the scalability bottleneck of LLM reranking in long-term agent systems, enabling high-quality retrieval with substantially reduced computational overhead.