💬 NLP

Auto-Dreamer: Learning Offline Memory Consolidation for Language Agents

The paper introduces Auto-Dreamer, a learned offline memory consolidator inspired by complementary learning systems theory that decouples fast online experience acquisition from slow cross-session consolidation, enabling language agents to abstract recurring patterns and prune redundancy to achieve superior performance with significantly smaller active memory banks across multiple environments.

Original authors: Chongrui Ye, Yuxiang Liu, Yu Wang, Haofei Yu, Yining Zhao, Ge Liu, Julian McAuley, Jiaxuan You

Published 2026-05-21

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Chongrui Ye, Yuxiang Liu, Yu Wang, Haofei Yu, Yining Zhao, Ge Liu, Julian McAuley, Jiaxuan You

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are an AI agent, like a super-smart robot assistant, trying to learn how to do chores, solve science puzzles, or navigate the web. Every time it tries a task, it learns something new. But here's the problem: if you just keep dumping every single thought, mistake, and success into a giant notebook, that notebook eventually becomes so huge and messy that the robot can't find the good stuff when it needs it.

This paper introduces Auto-Dreamer, a new way for AI agents to manage their memories. Think of it as a system that separates taking notes from writing a textbook.

The Two-Part System

The authors realized that current AI memory systems try to do two things at once:

Fast Learning: Immediately writing down what just happened.
Slow Learning: Figuring out the big patterns and rules from all those notes later.

They found that doing both at the same time makes the memory messy. So, they built a two-speed system inspired by how human brains work (specifically, how we sleep to process memories):

The Fast Writer (The Scribe): Every time the agent finishes a task, a "scribe" quickly jots down raw notes. It doesn't worry about being neat or organized; it just records everything that happened, like a court reporter. This happens online (while the agent is working).
The Slow Dreamer (The Editor): This is the new invention. Periodically, when the agent takes a break, the "Dreamer" wakes up. It looks at a chunk of those raw notes from the past. It doesn't just edit them; it rewrites the whole section.

How the "Dreaming" Works

Imagine you have a messy pile of 50 different recipes for making soup, some written on napkins, some on sticky notes, and some with typos.

Old Way: You keep all 50 notes. If you want to make soup, you have to read through all 50 to find the good parts.
Auto-Dreamer Way: The Dreamer reads all 50 notes, realizes they all say "boil water" and "add salt," and notices that some notes say "add sugar" (which is wrong). It then throws away the 50 messy notes and writes one single, perfect recipe card that captures the best parts and fixes the mistakes.

In technical terms, the Dreamer treats a section of memory as "read-only evidence." It uses tools to inspect the old notes and the original video of what happened (the "source trajectory"). Then, it synthesizes a fresh, compact replacement set. The old, messy notes are deleted and replaced by this new, clean version.

Why This is a Big Deal

The paper tested this on three different "worlds":

ALFWorld: A robot doing household chores (like putting a clean apple in the fridge).
ScienceWorld: A robot doing science experiments (like finding the longest-living animal).
WebArena: A robot navigating websites to do shopping or manage code.

The Results:

Smarter, Not Just Bigger: Auto-Dreamer solved more tasks than any other method.
Tiny Memory Footprint: While other methods needed memory banks that were 12 times larger (like a library vs. a pamphlet), Auto-Dreamer achieved better results with a tiny memory bank.
The "Sleep" Effect: The Dreamer was trained only on the ScienceWorld data. But when they tested it on the Household and Web tasks, it still worked perfectly! It learned how to "dream" and organize information generally, not just for one specific game.

The Secret Sauce: "Counterfactual Utility"

How does the Dreamer know what to keep and what to throw away? The paper uses a clever training trick called GRPO (a type of Reinforcement Learning).

Imagine the Dreamer is a chef trying to create the perfect menu.

If the chef creates a menu and the customers love the food, they get a reward.
But Auto-Dreamer adds a twist: It asks, "What if I removed this specific dish?"
- If removing a dish makes the meal worse, that dish is essential (keep it!).
- If removing a dish doesn't change anything, it was redundant (throw it away!).
- If removing a dish makes the meal better (maybe it was a bad ingredient), that dish was harmful (definitely throw it away!).

This ensures the memory bank only keeps the "load-bearing" information—the stuff that actually helps the agent succeed—while deleting the clutter.

Summary

Auto-Dreamer is like giving an AI agent a nightly routine where it doesn't just "sleep," but actively reorganizes its day's experiences. Instead of hoarding every single memory, it distills them into a few powerful, reusable rules. This allows the agent to be faster, smarter, and much more efficient, using a tiny fraction of the memory space required by previous methods.

Technical Summary: Auto-Dreamer: Learning Offline Memory Consolidation for Language Agents

Problem Statement

Language agents are increasingly deployed over streams of related tasks rather than isolated interactions. In these settings, long-term memory must serve not merely as a retrieval cache for past entities, but as a mechanism to convert raw experience into reusable procedures, environment knowledge, and behavioral priors. Existing memory systems face two primary challenges:

The Consolidation Problem: Current methods typically couple memory acquisition and consolidation into a single online process. Updates are made with limited evidence from the current session, making it difficult to discover recurring patterns, abstract shared procedures across sessions, resolve contradictions, or prune redundant entries.
The Memory-Utility Problem: Reinforcement learning (RL)-trained memory managers often optimize online construction or retrieval rather than offline consolidation under an explicit downstream utility objective. Consequently, they fail to directly learn which memories are "load-bearing," which entries are redundant, or how to trade off success against memory compactness.

Existing approaches, including retrieval-augmented episodic stores, structured memory systems, and reflection-based methods, generally lack a global view across sessions to perform effective cross-session abstraction.

Methodology: Auto-Dreamer

Inspired by Complementary Learning Systems (CLS) theory, which posits a fast hippocampal system for encoding episodes and a slower neocortical system for extracting shared structure, the authors propose Auto-Dreamer. This is a learned offline consolidator that decouples fast per-session memory acquisition from slow cross-session consolidation.

Core Architecture

The system operates on a typed memory bank containing semantic (factual) and procedural (how-to) entries. Each entry includes a short name, a body, and provenance links to source trajectories.

Fast Online Acquisition: A fixed, prompted "writer" model operates per-session. It is local and append-only, recording typed memory entries with provenance pointers to the source trajectory without searching the existing bank or rewriting old memories. This prioritizes plasticity.
Slow Offline Consolidation (Region Rewriting): The core innovation is the consolidator ( $C_\theta$ $C_{θ}$ ), a learned policy that operates offline. Instead of editing entries in place (CRUD), it performs region rewriting.
- Process: Given a selected working region $R$ of the memory bank and its associated provenance-linked source trajectories $T_R$ , the consolidator performs a bounded tool-use rollout. It uses tools to search memory, inspect candidate entries, and retrieve raw source trajectories.
- Output: It synthesizes a fresh, compact replacement set $S$ that abstracts across sessions. The original region $R$ is treated as read-only evidence and is entirely replaced by $S$ in the post-consolidation bank ( $B^* = (B \setminus R) \cup S$ ).
- Implication: This replacement semantics makes compactness structural; information survives only if it is re-synthesized into the replacement set, making abstraction, deduplication, and omission-based forgetting default behaviors.

Training via GRPO

The consolidator is trained using Group Relative Policy Optimization (GRPO).

Objective: The training isolates the consolidator's contribution by keeping the task agent, retriever, writer, and memory schema fixed.
Credit Assignment: Unlike online methods, the replacement set $S$ is evaluated in a local bank consisting only of the synthesized entries ( $B^*_g = S_g$ ). This allows for direct credit assignment to the policy's output without confounding factors from the rest of the bank.
Reward Signal: The reward $r_g$ $r_{g}$ combines downstream task performance with a counterfactual utility term:
$r_g = U_V(S_g) + \alpha \cdot r_{cf}(S_g; V)$
- $U_V(S_g)$ : The average return on a held-out evaluation task set $V$ .
- $r_{cf}$ : Estimated by randomly masking a fraction of entries in $S_g$ . This term penalizes redundant entries (where masking causes little performance drop) and rewards load-bearing entries (where masking hurts performance), while assigning negative credit to harmful entries.

Key Contributions

Two-Timescale Formulation: The paper distinguishes fast per-session acquisition from slow cross-session consolidation, formulating the latter as a learned decision problem over accumulated evidence.
Region Rewriting Primitive: It introduces provenance-grounded region rewriting as a compactness-inducing consolidation primitive. By replacing a region with a synthesized set rather than editing entries individually, it enforces cross-session abstraction and deduplication as default update semantics.
RL Training with Region-Local Credit: The method enables RL training without supervised memory labels by evaluating the self-contained replacement set directly. The use of counterfactual masking allows the system to learn to suppress redundant memories while preserving task utility.

Experimental Results

Auto-Dreamer was evaluated on three domains: ScienceWorld (text-based science experiments), ALFWorld (household instruction-following), and WebArena (web navigation).

Continual-Memory Deployment:
- On ScienceWorld, Auto-Dreamer achieved a 41.1% success rate, outperforming the strongest baseline (UMEM) by 7.0 points and the strongest prompted baseline (ReasoningBank) by 10.2 points.
- Crucially, it achieved this with an active memory bank 12× smaller than the strongest baseline (6.9k tokens vs. 80.9k tokens for UMEM).
- On ALFWorld, it reached 60.2% success (vs. 58.4% for UMEM) using 6× less memory.
- On WebArena, it led all baselines with 52.3% success using only 927 tokens (vs. 370k for LightMem).
Generalization: The consolidator was trained only on ScienceWorld trajectories. Despite this, it improved performance on held-out ALFWorld and WebArena without retraining, even when paired with different writer backbones (Qwen3-14B vs. Gemini-3.1-flash-lite-preview).
Controlled Consolidation: In fixed-bank experiments where the consolidation operator was applied once to a pre-built bank, Auto-Dreamer still outperformed baselines, confirming that the gains stem from the offline consolidation process itself rather than just the accumulation of data.

Significance and Claims

The paper claims that Auto-Dreamer demonstrates that offline learned consolidation can significantly improve the success–cost tradeoff for language agents. By separating acquisition from consolidation and using region rewriting, the system achieves higher task success with substantially smaller memory footprints compared to both prompted and RL-trained baselines.

The authors emphasize that the learned consolidator transfers across task domains and writer backbones, supporting the view that consolidation over textual memory entries and source trajectories can be domain- and writer-agnostic. The work suggests that when agent experience accumulates faster than it can be reorganized in-session, offline learned consolidation is a viable mechanism for maintaining compact, high-utility memory banks.

The paper remains modest regarding its scope, noting that evaluation is restricted to text-based environments and that the consolidator relies on a fixed writer schema; robustness to alternative writers or multimodal inputs is not yet established.