Mem-T: Densifying Rewards for Long-Horizon Memory Agents

Mem-T introduces an autonomous memory agent trained via the MoT-GRPO framework, which densifies sparse long-horizon rewards through tree-guided backpropagation to achieve superior performance and efficiency compared to existing memory management systems.

The Gist

Imagine you are trying to teach a very smart but forgetful robot assistant how to handle a massive, years-long conversation with a human. The problem is, the robot has a tiny "working memory" (like a sticky note) that can only hold a few sentences at a time. If the conversation gets too long, the robot forgets the beginning, gets confused, and starts making things up.

This paper introduces Mem-T, a new way to teach robots how to build and use a long-term memory effectively. Here is the breakdown using simple analogies.

The Problem: The "One-Prize" Lottery

Previously, when researchers tried to train these robots, they used a method similar to a lottery.

• How it worked: The robot would go through hundreds of steps (reading, thinking, searching, writing notes) for a whole conversation. Only at the very end, after the human asked a final question, would the robot get a reward: "Good job!" (1 point) or "Wrong answer" (0 points).
• The Flaw: The robot had no idea which specific step led to the win. Did it win because it remembered the name "Gina" in step 10? Or because it searched the right database in step 50? It was a mystery. This is called the "Sparse Reward" problem. The robot was guessing in the dark.

The Solution: Mem-T (The Smart Librarian)

The authors created Mem-T, a robot that acts like a super-organized librarian with three distinct types of shelves:

1. Factual Memory: Hard facts (e.g., "Gina was born in 1990").
2. Experiential Memory: Lessons learned (e.g., "If Gina is tired, she prefers short meetings").
3. Raw Memory: The unedited transcript of the conversation (just in case).

Mem-T doesn't just store things; it actively decides what to write down, what to update, and what to throw away, all while the conversation is happening.

The Secret Sauce: MoT-GRPO (The "Tree of Choices")

The real magic isn't just the memory shelves; it's how they trained the robot. They invented a new training method called MoT-GRPO.

Imagine the robot is trying to find a specific book in a giant library to answer a question.

• Old Way: The robot picks one path, walks down the aisle, and if it fails, it gets a "Game Over" signal. It learns nothing about why it failed.
• Mem-T's Way (The Tree):
  1. Branching Out: Instead of just walking one path, the robot imagines three different versions of itself walking down three different aisles at the same time.
  2. Dense Rewards: As each version walks, it gets small rewards for finding useful clues along the way (e.g., "Good job finding the 'Facts' section!").
  3. Backtracking: If one path leads to a dead end, the system looks at the other paths. It says, "Ah, the version that searched the 'Experience' shelf first found the answer!"
  4. Hindsight Credit: The system then goes back to the very beginning and tells the robot: "You were right to look at the Experience shelf first. That was the key move."

This turns the "one big prize at the end" into a continuous stream of feedback, teaching the robot exactly which actions matter.

The Results: Smarter and Cheaper

Because Mem-T knows exactly which steps matter:

• It's Smarter: It beats previous top-tier memory systems by a significant margin (up to 15% better) on complex, long-term questions.
• It's Efficient: It doesn't waste energy searching everywhere. It knows exactly where to look, saving about 24% of the computer power (tokens) needed to answer a question.

The Bottom Line

Think of Mem-T as upgrading a robot from a goldfish (who forgets everything after 10 seconds) to a seasoned detective (who keeps a detailed case file, knows how to cross-reference clues, and learns from every mistake).

By using a "Tree of Choices" to give the robot constant feedback instead of waiting until the end to say "Good job," the researchers solved the problem of teaching AI how to remember the long story, not just the last sentence.

Technical Summary

Here is a detailed technical summary of the paper "Mem-T: Densifying Rewards for Long-Horizon Memory Agents."

1. Problem Statement

The paper addresses a critical bottleneck in developing autonomous AI agents with long-term memory: temporal credit assignment in long-horizon tasks.

• Sparse and Delayed Rewards: Existing trainable memory agents (e.g., Memory-R1, Mem-$\alpha$) often rely on reinforcement learning (RL). However, in long-context scenarios (hundreds of turns), agents perform hundreds of memory operations (creation, evolution, retrieval) before receiving a final binary reward (0/1) based on the accuracy of a final answer.
• Optimization Failure: This extreme sparsity makes it difficult to attribute success or failure to specific intermediate actions. Current methods indiscriminately propagate the terminal reward across all steps, leading to ineffective optimization of memory construction and retrieval policies.
• Limitations of Heuristics: Pre-existing systems (e.g., MemGPT, Mem0) rely on hand-crafted prompts and static rules, which are bounded by the base model's instruction-following capabilities and lack adaptability.

2. Methodology

The authors propose Mem-T, a hierarchical memory agent, trained via MoT-GRPO (Memory Operation Tree-guided Group Relative Policy Optimization).

A. Mem-T Architecture

Mem-T operates on a continuous information stream and maintains a hierarchical memory state $M_t$ consisting of four components:

1. Working Memory ($M^{work}$): A concise, session-level summary updated iteratively.
2. Factual Memory ($M^{fact}$): Declarative knowledge (facts, entities) with validity time windows.
3. Experiential Memory ($M^{exp}$): Procedural knowledge (strategies, lessons learned).
4. Raw Memory ($M^{raw}$): Archives of raw dialogue turns.

The agent performs two parallel phases:

• Continuous Memory Construction: Proactively extracts facts, experiences, and updates summaries from the input stream using formation and evolution policies ($\pi_{form}, \pi_{evol}$).
• On-Demand Memory Retrieval: When a query arises, the agent executes a multi-turn search across the memory modules to synthesize an answer.

B. MoT-GRPO: The Training Framework

To solve the reward sparsity problem, the authors introduce a tree-guided RL framework that densifies rewards through backpropagation and hindsight credit assignment.

1. Memory Operation Tree (MoT) Construction:

• Instead of a single trajectory, the agent generates an ensemble of $G$ trees.
• Iterative Branching Rollout: Starting from a root node, the policy stochastically samples pivot nodes and expands them into new branches (rollouts). This creates a tree of diverse retrieval trajectories for a single query.

2. Node-wise Reward Backpropagation (for Retrieval):

• Dense Rewards: Every node $v$ in the tree is assigned a dense reward $R(v)$, not just the leaf nodes.
  • $R(v) = \mathbb{I}_{fmt}(v) \cdot (\alpha \cdot \text{Evid}(v) + \text{Perform}(v))$
  • Evidence Hit Rate: Measures immediate evidence retrieval.
  • Expected Performance: For leaf nodes, this is the final F1/Accuracy. For internal nodes, it is the average performance of their children.
• Dual-Scale Advantage Estimation:
  • Intra-Tree Advantage: Normalizes rewards within a single tree to identify locally optimal paths.
  • Inter-Tree Advantage: Normalizes rewards across the entire ensemble to encourage global exploration.
  • The total advantage $A_{total}$ combines both to guide the policy update.

3. Hindsight Credit Assignment (for Construction):

• Since memory construction happens before the query, rewards are delayed. The framework back-propagates the advantage signals from the successful retrieval trees to the upstream construction actions.
• Credit Coefficient ($\rho$): A memory construction action receives credit if:
  1. Evidence Alignment: The constructed memory contains ground-truth evidence required for the answer.
  2. Retrieval Trace: The constructed memory was actually retrieved and used in the successful trajectory.
• Offline Optimization: High-scoring construction actions are curated into a dataset $D^*_{mem}$ to fine-tune the construction policy via supervised learning (maximizing log-likelihood), effectively distilling "hindsight wisdom."

3. Key Contributions

1. Unified Memory Framework (Mem-T): A streamlined agent integrating factual, experiential, and working memory with agentic orchestration of the full memory lifecycle (formation, evolution, retrieval).
2. MoT-GRPO Paradigm: A novel RL approach that transforms sparse terminal rewards into dense, step-wise supervision. It solves the temporal credit assignment problem via:
   • Tree-based reward backpropagation for retrieval.
   • Hindsight credit assignment for construction.
3. State-of-the-Art Performance: Demonstrates that joint optimization of construction and retrieval yields superior results compared to decoupled or heuristic approaches.

4. Experimental Results

The authors evaluated Mem-T on four benchmarks: LoCoMo (long-term conversation), LongMemEval, HotpotQA, and NarrativeQA.

• Performance Gains:
  • On LoCoMo, Mem-T (with MoT-GRPO) achieved an Overall F1 of 58.65, surpassing the previous SOTA (GAM) by 14.92% (using the same Qwen3-4B backbone).
  • It outperformed training-free baselines (like RAG and MemGPT) by significant margins and also beat other trained methods (Memory-R1, MemAgent).
  • Cross-Domain Generalization: Mem-T maintained SOTA performance on out-of-domain tasks (HotpotQA, NarrativeQA), showing an average improvement of 13.52% over other methods.
• Efficiency (Pareto Frontier):
  • Mem-T operates on a favorable accuracy-efficiency frontier.
  • Compared to GAM, Mem-T reduced inference tokens per query by ~24.45% while achieving higher accuracy.
• Ablation Studies:
  • Removing the Factual Memory module caused the largest performance drop, highlighting its critical role.
  • Removing either the Retrieval Optimization or Construction Optimization components of MoT-GRPO significantly degraded performance, proving the necessity of joint training.
  • Inter-tree advantage was found to be more critical than intra-tree advantage for stable training.

5. Significance

• Paradigm Shift: The paper moves memory management from static, heuristic-based systems to fully learnable, attribution-centric systems.
• Solving Long-Horizon RL: It provides a robust solution to the "sparse reward" problem in long-context agents, enabling end-to-end optimization of complex, multi-step memory pipelines.
• Scalability: By densifying rewards, Mem-T allows smaller models (e.g., Qwen3-4B) to outperform larger, untrained models or heuristic systems, making high-performance memory agents more accessible and cost-effective.
• Future of Agents: This work lays the groundwork for self-evolving agents capable of lifelong learning, where the agent continuously refines its own memory strategies based on task outcomes.