Technical Summary: ADAMEM – Test-Time Adaptive Memory for Language Agents

1. Problem Statement

Language agents face a central challenge in utilizing past experience to adapt to dynamic, long-horizon tasks. While recent advancements in agent memory mechanisms show promise, existing systems typically restrict memory retrieval to the episode initiation phase. This creates a "rigidity bottleneck": agents rely on static guidance (e.g., a single retrieved trajectory or distilled strategy) that cannot adjust to intermediate failures, shifting sub-goals, or distribution shifts occurring later in the task.

Traditional approaches to test-time adaptation, such as updating model parameters online, are prohibitively expensive for long-horizon settings. Conversely, training-free prompt adaptation methods often suffer from the limitation that their retrieved context becomes misaligned as the task unfolds. The paper identifies two primary research questions:

1. How can agent memory overcome the rigidity of static retrieval to enable continuous adaptation during inference?
2. How can agents be efficiently trained to synthesize strategies that explicitly optimize this test-time adaptation?

2. Methodology

The authors propose ADAMEM (Adaptive Memory Agent), a hybrid memory framework that decouples memory storage from abstraction to enable step-wise test-time adaptation without updating model parameters.

2.1 Hybrid Memory Architecture

ADAMEM operates via two distinct memory components:

• Long-Term Trajectory Memory ($M$): A scalable, offline repository of raw experiences. It stores successful trajectories $\tau = \{(s_t, a_t, r_t)\}$ indexed by state embeddings. Unlike systems that store distilled insights, $M$ retains the full interaction history to ensure reliable context retrieval.
• Short-Term Strategy Memory ($z_t$): A dynamic, state-specific strategy generated on-the-fly during inference. Instead of retrieving raw logs directly, the agent synthesizes a concise natural language strategy conditioned on the current state $s_t$ and retrieved experiences $E_{ret}$. This strategy serves as concrete guidance for the next action.

2.2 Inference Modes

To balance token efficiency and adaptability, ADAMEM introduces two inference modes:

• ADAMEM-HIGH (High Adaptation Effort): Prioritizes robustness by regenerating a fresh strategy $z_t$ at every critical decision step if the agent decides retrieval is necessary. The strategy is transient and discarded after use, ensuring the agent never relies on stale guidance.
• ADAMEM-LOW (Low Adaptation Effort): Prioritizes efficiency by maintaining a persistent strategy $z_{curr}$ in the context. The agent predicts whether to "refresh" the strategy based on the current state. If no refresh is needed, the existing strategy guides multiple steps; if a refresh is triggered, a new strategy is synthesized and replaces the persistent one.

2.3 STEP-MFT: Step-wise Memory Fine-Tuning

While ADAMEM can operate via zero-shot prompting, the authors introduce STEP-MFT to train the policy to generate high-utility strategies.

• Challenge: Standard outcome-based supervision is noisy, as it credits all steps in a successful trajectory equally, even if a specific strategy was redundant or misleading.
• Solution: STEP-MFT uses a rejection sampling approach with a dual-filter mechanism to curate training data:
  1. Outcome Success: The trajectory must be successful.
  2. Action Change: The generated strategy must causally alter the agent's action compared to a baseline policy acting without memory ($a_t \neq a'_t$).
• Theoretical Justification: The authors prove that for a strategy to have a strictly positive advantage over the baseline, it must change the agent's action. This filter ensures the model learns to generate strategies that actively drive decision-making rather than merely echoing context.
• Training: The filtered dataset is used for Supervised Fine-Tuning (SFT) via cross-entropy loss, optimizing the policy to synthesize "silver" strategies that improve success rates.

3. Key Contributions

1. ADAMEM Framework: A novel hybrid memory system that enables continuous test-time adaptation by dynamically synthesizing context-aware strategies rather than retrieving static artifacts.
2. STEP-MFT Technique: A training method that uses process-level rewards (specifically, action changes) to teach agents to generate high-utility strategies, avoiding the noise of outcome-only supervision.
3. New Scaling Dimension: The work establishes that dynamic memory curation is a critical scaling dimension for agentic reasoning, offering a superior Pareto frontier between inference cost and task performance.

4. Experimental Results

The authors evaluated ADAMEM on three benchmarks: ALFWorld (embodied navigation), WebShop (web shopping), and HotpotQA (multi-hop search).

• Performance Gains:
  • ALFWorld: ADAMEM-LOW achieved a 13% relative gain over static memory baselines on the unseen split (58.2% vs. 52.2% for Synapse).
  • WebShop: ADAMEM-LOW achieved an 11% relative gain (74.2% vs. 65.4% for Synapse). Notably, static baselines suffered from negative transfer on WebShop, performing worse than the memory-free baseline, whereas ADAMEM reversed this trend.
  • HotpotQA: ADAMEM achieved consistent leading performance (41.1% success rate) in an iterative cross-episode search setup.
• STEP-MFT Impact: Training with STEP-MFT yielded further improvements. On ALFWorld, the MFT variant achieved a 17% relative gain over static baselines (65.5% success rate), significantly outperforming both training-free ADAMEM and outcome-based MFT variants.
• Efficiency vs. Adaptability:
  • ADAMEM scales monotonically with the retrieval budget ($k$), whereas static baselines like Synapse degrade due to context overflow from raw trajectory injection.
  • ADAMEM reduces inference latency by 16% compared to Synapse by synthesizing concise strategies instead of processing lengthy raw logs.
• Cross-Model Generalization: In off-policy settings (using Gemma-27b-it with Qwen-generated memory), ADAMEM-LOW maintained robust improvements, demonstrating that the framework effectively decouples storage from the policy model.

5. Significance and Claims

The paper claims that ADAMEM addresses the fundamental rigidity of current agent memory systems. By decoupling long-term storage from short-term abstraction, ADAMEM allows agents to:

• Self-Evolve Post-Deployment: Agents can adapt to dynamic environments and correct for stale priors without expensive online parameter updates.
• Optimize Compute Trade-offs: The framework supports flexible inference modes (LOW/HIGH) and training variants (STEP-MFT) that allow practitioners to tune the balance between token efficiency and adaptability.
• Enable Continuous Reasoning: The work suggests that dynamic memory curation is a necessary scaling dimension for agents operating in open-ended, real-world environments, moving beyond static retrieval toward continuous, state-aware reasoning.

The authors acknowledge limitations, such as "strategy inertia" in ADAMEM-LOW (where agents may fail to trigger a refresh when necessary) and the current reliance on successful trajectories only for memory population, noting these as directions for future work.