Technical Summary: Memory as Action (MemAct)

1. Problem Statement

Long-horizon agentic tasks, such as deep research and software engineering, require Large Language Models (LLMs) to maintain extensive working memory. While Long-Context LLMs have expanded capacity, they suffer from attention dilution and "lost-in-the-middle" behaviors when contexts become saturated with irrelevant information.

Current approaches to managing this context fall into two suboptimal categories:

1. Passive Compression: Methods like token-level compression or periodic summarization operate without awareness of the agent's specific reasoning state, risking the loss of semantically critical dependencies.
2. External Controllers: Systems that use heuristic rules or external modules to manage memory decouple context curation from the agent's core reasoning policy. This prevents end-to-end optimization of information retention against task performance.

The core challenge identified is Context Curation: the need to strategically select, integrate, and prune information to maintain a focused, goal-relevant reasoning trace. Existing methods fail to treat this as an intrinsic, learnable capability of the agent.

2. Methodology: Memory-as-Action (MemAct)

The paper proposes MemAct, a framework that reconceptualizes context management as a set of learnable actions within a unified policy space, rather than an external mechanism.

2.1 Core Framework

MemAct integrates context curation directly into the agent's action space ($\mathcal{A} = \mathcal{A}_{task} \cup \mathcal{A}_{mem}$).

• Unified Policy: A single policy $\pi_\theta$ decides both task actions (e.g., search, tool use) and memory actions.
• Prune&Write Operator: The primary memory action is an in-place editing operation. When triggered, the agent:
  1. Prunes: Removes specific historical interaction turns identified by unique IDs.
  2. Writes: Synthesizes a new memory note (containing summaries, facts, or planning) and appends it to the context.
• State Representation: The working memory is modeled as a sequence of uniquely addressable interaction records $H_t = [z_1, z_2, \dots, z_k]$, where each record includes an action, observation, and a unique ID.

2.2 Training Challenge: Dynamic Context Updates

A significant hurdle in training MemAct is the train-inference mismatch. Causal LLMs assume monotonic context growth. When MemAct deletes content, the causal dependencies encoded in the Key-Value (KV) cache of subsequent tokens become invalid (the "deleted" information still physically influences the hidden states of tokens generated before the deletion). Simple attention masking is insufficient because it does not sever these causal dependencies.

2.3 Solution: Dynamic Context Policy Optimization (DCPO)

To enable efficient Reinforcement Learning (RL) with dynamic context updates, the authors introduce DCPO.

• Trajectory Segmentation: DCPO logically partitions the agent's interaction trajectory at every memory edit point.
• Independent Segments: The trajectory is reorganized into $K+1$ independent segments $\{\sigma_i\}$. Within each segment, the context prefix is fixed, restoring the standard auto-regressive causal structure required for stable gradient computation.
• Optimization: The policy is optimized using a group-relative normalization scheme (similar to GRPO) over these reconstructed, logically consistent segments, ensuring gradients are computed against the correct context history.

3. Key Contributions

1. Paradigm Shift: Proposes Memory-as-Action, shifting working memory management from external heuristics to an intrinsic, learnable policy capability. This allows agents to autonomously balance context curation and task execution.
2. Technical Innovation:
   • Formulates working memory as a Markov Decision Process (MDP) with ID-based addressable sequences and the Prune&Write operator for fine-grained editing.
   • Develops DCPO, a trajectory segmentation algorithm that reconciles dynamic context updates with efficient RL training on standard infrastructure without bespoke modifications.
3. Empirical Validation: Demonstrates that learned memory strategies are efficient, adaptive, and generalizable.

4. Experimental Results

The authors evaluated MemAct on synthetic multi-objective QA datasets (based on HotpotQA) and public benchmarks (2WikiMultihopQA, Bamboogle, Frames, BrowseComp-Plus).

• Performance vs. Efficiency:

  • MemAct-RL-14B (14B parameters) achieved 59.1% accuracy on multi-objective tasks, outperforming Qwen3-235B (235B parameters, 53.1%) and the specialized Tongyi-DeepResearch (56.0%).
  • Despite the smaller size, MemAct-RL-14B reduced the average context length per step by 51% compared to Qwen3-235B (3,500 tokens vs. ~7,000 tokens) and reduced total token consumption by 51%.
  • It matched the accuracy of models 16× larger while operating with significantly lower computational cost.

• Generalization:

  • The model was trained on tasks with at most 3 objectives but successfully generalized to tasks with 8 objectives, maintaining high accuracy where baselines suffered performance saturation.
  • It demonstrated robustness across different model scales (7B and 14B) and new tool environments (BrowseComp-Plus).

• Learned Strategies:

  • 7B Models: Learned to perform frequent, aggressive pruning (removing ~7 records per action) to compensate for limited context capacity.
  • 14B Models: Developed a bimodal strategy: fine-grained pruning (2 records) during reasoning to remove noise, and coarse-grained pruning (6 records) after completing sub-objectives to clear intermediate steps.

• Latency: MemAct reduced total inference duration by 40% compared to Search-R1 (an RL baseline without memory actions) due to reduced pre-fill times and higher prefix cache hit rates.

5. Significance and Claims

The paper claims that autonomous context curation can be internalized as a learnable skill, serving as a fundamental and scalable architectural building block for agentic behavior.

• Intrinsic Capability: The results suggest that agents do not need external memory managers; they can learn to "edit" their own working memory to maintain reasoning focus.
• Scalability: The framework allows smaller models to compete with much larger models by optimizing information density rather than just raw capacity.
• Adaptivity: The system autonomously discovers strategies tailored to the specific capacity of the backbone model, adjusting memory action intensity based on task complexity.

The authors acknowledge limitations, noting that the approach relies on sparse rewards which can make credit assignment difficult, and that the lossy nature of compression means original data cannot be recovered once summarized. However, they posit that this approach provides a principled interface at the decision layer, complementary to system-level infrastructures.