💬 NLP

Evo-Memory: Benchmarking LLM Agent Test-time Learning with Self-Evolving Memory

This paper introduces Evo-Memory, a comprehensive streaming benchmark and framework designed to evaluate and enhance the self-evolving memory capabilities of LLM agents in continuous, dynamic task environments, featuring the proposed ReMem pipeline that integrates reasoning, actions, and memory updates for continual improvement.

Original authors: Tianxin Wei, Noveen Sachdeva, Benjamin Coleman, Zhankui He, Yuanchen Bei, Xuying Ning, Mengting Ai, Yunzhe Li, Jingrui He, Ed H. Chi, Chi Wang, Shuo Chen, Fernando Pereira, Wang-Cheng Kang, Derek Zhiy

Published 2026-05-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Tianxin Wei, Noveen Sachdeva, Benjamin Coleman, Zhankui He, Yuanchen Bei, Xuying Ning, Mengting Ai, Yunzhe Li, Jingrui He, Ed H. Chi, Chi Wang, Shuo Chen, Fernando Pereira, Wang-Cheng Kang, Derek Zhiyuan Cheng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant assistant who can solve complex puzzles, write code, or navigate a virtual house. Right now, most of these AI assistants are like amnesiac geniuses. They are incredibly smart in the moment, but as soon as a conversation ends or a task is finished, they forget everything they just learned. If you ask them to solve a similar problem the next day, they have to start from scratch, making the same mistakes and rediscovering the same solutions.

The paper "Evo-Memory" proposes a way to fix this by giving these assistants a living, breathing memory that evolves over time.

Here is a breakdown of the paper's core ideas using simple analogies:

1. The Problem: The "Goldfish" vs. The "Apprentice"

Currently, most AI memory works like a library archive. If you ask a question, the AI looks up a specific fact it stored earlier (like "What is the capital of France?") and reads it back. This is called Conversational Recall.

But the authors argue this isn't enough. Real intelligence requires Experience Reuse.

The Goldfish: Remembers what happened (e.g., "I tried to open the door and it was locked").
The Apprentice: Remembers how to handle it next time (e.g., "Next time I see a locked door, I should check for a key or try the handle first").

The paper says current AI systems are stuck being Goldfish. They remember facts but don't learn strategies from their past failures and successes.

2. The Solution: A "Self-Evolving" Notebook

The authors introduce Evo-Memory, a new way to test and build AI agents that act like an Apprentice.

Imagine an AI agent has a notebook.

Old Way: After every task, the AI just pastes the whole conversation into the notebook. The notebook gets huge, messy, and hard to read.
Evo-Memory Way: After every task, the AI does three things:
1. Think: "What did I just do? Did it work?"
2. Act: Finish the current task.
3. Refine: "Okay, that solution worked. I'm going to write a short, clear rule in my notebook about how I solved this, and throw away the messy parts that didn't matter."

Over time, this notebook doesn't just get bigger; it gets smarter and more organized. It turns raw experiences into reusable strategies.

3. The Test: The "Streaming" Exam

To prove this works, the researchers created a new benchmark called Evo-Memory.

Think of this like a marathon instead of a sprint.

Old Tests: You give the AI one math problem, then a new one, then another. They are unrelated. The AI just solves them one by one.
Evo-Memory Test: You give the AI a continuous stream of tasks, like a video game level that keeps getting harder.
- Task 1: Solve a simple equation.
- Task 2: Solve a slightly harder equation using the same logic.
- Task 3: Navigate a robot to find a tomato, then put it in a microwave.
- Task 4: Navigate a robot to find a cup, then put it on a counter.

The test checks: Does the AI get faster and better at Task 4 because it learned from Task 3? If the AI is truly "evolving," it should use its memory to skip steps and solve problems more efficiently as the stream continues.

4. The Results: The "ReMem" Agent

The paper tested many different types of memory systems. They found that:

Static Memory: Agents that just store past chats (like a standard chatbot) didn't improve much over time. They kept making the same mistakes.
The "ReMem" Agent: This is the paper's new method. It actively thinks about its memory. It asks, "Is this old memory useful? Should I delete the bad stuff? How can I use this to solve the new problem?"

The Analogy:

Old Agents are like a student who takes a test, gets a bad grade, and then immediately forgets the test, taking the next one with the same confusion.
ReMem is like a student who takes a test, reviews their mistakes, writes a "cheat sheet" of the right formulas, and uses that cheat sheet to ace the next test.

5. Key Findings

Learning from Failure: The best agents didn't just remember success; they learned from failure. If an agent tried to open a door and it was locked, the "ReMem" agent updated its memory to remember "Check for keys first," rather than just remembering "I tried to open the door."
Efficiency: Agents with this evolving memory took fewer steps to solve problems. They didn't waste time re-inventing the wheel.
It Works Everywhere: This improvement happened whether the AI was doing math, writing code, or navigating a virtual house.

Summary

The paper argues that for AI to be truly helpful in the real world (where tasks are continuous and complex), they need more than just a hard drive to store facts. They need a dynamic memory system that constantly reviews, organizes, and upgrades its own knowledge, turning "what happened" into "how to do it better next time."

They built a new test (Evo-Memory) to measure this and showed that their new method (ReMem) is the first to successfully make AI agents learn and improve while they are working, rather than just waiting to be retrained by humans later.

Technical Summary: Evo-Memory

Problem Statement

Current Large Language Model (LLM) agents possess significant capabilities in reasoning, planning, and tool use, yet their ability to maintain state and evolve through interaction remains underexplored. Existing memory systems primarily function as passive storage for static conversational recall, retrieving past facts to answer queries without actively learning from accumulated experiences to improve future decision-making. While recent benchmarks have addressed sequential learning or long-term consistency, they often fail to evaluate an agent's capacity to reuse reasoning strategies and adapt memory structures across evolving task streams in real-world deployment scenarios. There is a critical gap in unified frameworks that test whether LLMs can continuously retrieve, integrate, and update memory during deployment—a process termed test-time evolution.

Methodology

The Evo-Memory Benchmark

To address this gap, the authors introduce Evo-Memory, a comprehensive streaming benchmark and framework designed to evaluate self-evolving memory in LLM agents.

Task Stream Restructuring: Unlike static benchmarks, Evo-Memory restructures datasets into sequential task streams. Agents process inputs $\{x_1, \dots, x_T\}$ where the memory state $M_t$ evolves after each interaction.
Unified Formulation: The framework models a memory-augmented agent as a tuple $(F, U, R, C)$ $(F, U, R, C)$ , where $F$ $F$ is the base LLM, $U$ $U$ is the update pipeline, $R$ $R$ is the retrieval module, and $C$ $C$ constructs the context. The agent operates in a cycle:
1. Search: Retrieve relevant memory entries $R_t$ based on current input $x_t$ .
2. Synthesis: Construct a working context $\tilde{C}_t$ from retrieved items.
3. Evolve: Generate output $\hat{y}_t$ , receive feedback $f_t$ , and update the memory state $M_{t+1} = U(M_t, m_t)$ , where $m_t$ captures the new experience.
Dataset Scope: The benchmark spans diverse domains including factual knowledge (MMLU-Pro, GPQA-Diamond), mathematical reasoning (AIME-24/25), tool use (ToolBench), and multi-turn embodied interactions (AlfWorld, BabyAI, ScienceWorld, PDDL).

Proposed Frameworks

The paper implements and evaluates several memory architectures, introducing two specific methods to bridge the gap between static recall and active evolution:

ExpRAG (Experience Retrieval and Aggregation): A baseline method that treats memory as a retrieval-augmented context. It encodes past tasks as structured experiences and retrieves top- $k$ similar examples to condition the model, effectively performing one-shot experience reuse via in-context learning.
ReMem (Refine Memory): An advanced framework that unifies reasoning, action, and memory refinement into a single decision loop. Unlike standard ReAct agents, ReMem introduces a "Refine" operation alongside "Think" and "Act." This allows the agent to perform meta-reasoning over its own memory—actively evaluating, pruning noise, and reorganizing stored experiences to better support future tasks. This creates a Markov decision process where the state includes the current input, memory state, and reasoning traces.

Key Results

The authors evaluated over ten representative memory modules (including SelfRAG, Mem0, Dynamic Cheatsheets, and Agent Workflow Memory) across Gemini-2.5 and Claude model families.

Performance Gains: Self-evolving memory architectures consistently outperformed static baselines. In single-turn reasoning and QA tasks, ReMem achieved an average exact match of 0.65 and API accuracy of 0.85/0.71 on Gemini-2.5 Flash. In multi-turn embodied environments, ReMem reached success rates of 0.92/0.96 on BabyAI and 0.78/0.91 on ScienceWorld (Claude 3.7 Sonnet), significantly outperforming non-evolving baselines.
Efficiency: Evolving-memory methods demonstrated improved step efficiency. ReMem reduced the average number of steps required to complete tasks (e.g., from 22.6 to 11.5 steps on AlfWorld), indicating that memory refinement leads to more focused reasoning.
Task Similarity and Transfer: Performance gains correlated strongly with within-dataset task similarity (Pearson $r=0.717$ ). Agents benefited most in environments with high structural coherence (e.g., PDDL, AlfWorld), suggesting that recurring task structures facilitate effective memory reuse.
Robustness to Difficulty and Failure:
- Sequence Order: ReMem maintained strong performance regardless of task difficulty order (Easy→Hard vs. Hard→Easy), whereas baseline methods showed significant variance.
- Noise Handling: When exposed to both successful and failed experiences, ReMem remained robust by actively refining stored memories, whereas naive accumulation of failures degraded baseline performance.
Model Agnosticism: The benefits of evolving memory were observed across both large (Sonnet, Pro) and smaller (Haiku, Flash-Lite) models, with smaller models showing particularly significant relative improvements.

Significance and Claims

The paper claims that Evo-Memory establishes a unified platform for evaluating the "lifelong intelligence" of LLM agents, shifting the focus from passive conversational recall to active experience reuse.

Bridging the Gap: The work highlights that while agents can remember what was said, they often fail to learn what was learned. Evo-Memory provides the first systematic framework to measure how agents accumulate and refine strategies across task streams.
Design Insights: The results suggest that explicit task-level utilization and continual reorganization of memory are critical for adaptability. The success of ReMem indicates that interleaving memory refinement with reasoning and action is a promising paradigm for creating self-improving agents.
Future Direction: The authors position this work as a foundation for developing LLMs with reliable, continually improving memory, noting that while current methods show promise, stability and procedural reuse remain areas requiring further investigation.

The paper concludes by releasing code and configurations to foster reproducibility, aiming to standardize the evaluation of test-time learning and self-evolving memory in the broader research community.