AEM: Adaptive Entropy Modulation for Multi-Turn Agentic Reinforcement Learning

1. Problem Statement

The paper addresses a critical bottleneck in training Large Language Model (LLM) agents for multi-turn Reinforcement Learning (RL) tasks (e.g., software engineering, web navigation, embodied agents).

• Sparse Rewards: In agentic RL, feedback is typically outcome-only (success/failure at the end of a trajectory). This makes it difficult to assign credit to intermediate steps, leading to high-variance gradients.
• Limitations of Existing Solutions:
  • Process Reward Models (PRMs): Require dense intermediate supervision and additional model training, increasing complexity and reducing generalizability.
  • Self-Supervised Methods: Existing approaches (e.g., GiGPO, IGPO) often suffer from context inconsistency, grouping bias, or restrictive structural assumptions.
  • Tree-Structured Methods: While effective, they incur high computational overhead in multi-turn settings.
• The Core Challenge: There is a need for a supervision-free, scalable, and fine-grained credit assignment method that can adaptively balance exploration (trying new actions) and exploitation (refining known good actions) without relying on external annotations or heavy computation.

2. Methodology: Adaptive Entropy Modulation (AEM)

The authors propose AEM, a plug-in algorithm that modulates the advantage signal during RL training based on response-level entropy dynamics.

Theoretical Foundation

The authors establish a theoretical link between response-level entropy and credit assignment:

• Response-Level Entropy ($H_{resp}$): Instead of token-level entropy, they define entropy at the level of a complete agent response (action). This reduces sampling variance and provides a more stable measure of uncertainty.
• Entropy Drift Theorem: They prove that the drift in global entropy during training is governed by the interaction between the advantage ($A$) of a sampled response and its relative surprisal ($S - H_{resp}$).
  • If a response has a high advantage but low relative surprisal (it was expected but good), entropy should decrease (exploitation).
  • If a response has a negative advantage but high relative surprisal (it was unexpected and bad), entropy should increase (encouraging exploration away from this path).
• Natural Gradient: The analysis is conducted within the Fisher-Rao geometry of the policy simplex, showing that entropy dynamics are intrinsic to the policy update direction.

The AEM Algorithm

AEM acts as a lightweight modifier on top of standard group-based RL estimators (like GRPO):

1. Uncertainty Proxy: For each response in a rollout group, AEM calculates a proxy for relative surprisal using the average token-level entropy of that response.
2. Modulation Coefficient ($\alpha$):
   • Responses are normalized within their group (min-max scaling).
   • A temperature-scaled exponential function converts the normalized entropy into a coefficient $\alpha$.
   • Logic:
     • Low entropy (high confidence) $\rightarrow$ $\alpha > 1$ (upweight if positive, downweight if negative).
     • High entropy (low confidence) $\rightarrow$ $\alpha < 1$ (downweight if positive, upweight if negative).
3. Advantage Rescaling: The base advantage ($A_{base}$) is rescaled: $A_{AEM} = \alpha \cdot A_{base}$.
4. Adaptive Transition:
   • Early Training: Dominated by negative samples; AEM induces entropy-increasing pressure to promote exploration.
   • Late Training: Dominated by positive samples; AEM induces entropy-decreasing pressure to promote convergence and exploitation.

3. Key Contributions

1. Theoretical Characterization: The paper provides the first theoretical analysis of response-level entropy dynamics in agentic RL, proving that local entropy drift is driven by the product of the advantage and relative surprisal.
2. AEM Algorithm: A novel, supervision-free credit assignment method that requires no extra models, dense rewards, or complex structural assumptions. It functions as a plug-in to existing RL frameworks.
3. Implicit Exploration-Exploitation Control: AEM naturally transitions the policy from exploration to exploitation based on the evolving balance of positive/negative samples, eliminating the need for hand-crafted entropy schedules.

4. Experimental Results

The authors evaluated AEM across three challenging benchmarks (ALFWorld, WebShop, SWE-bench-Verified) using models ranging from 1.5B to 32B parameters.

• ALFWorld & WebShop:
  • AEM consistently improved state-of-the-art baselines (GRPO, GSPO, DAPO).
  • Performance Gains: Achieved peak gains of 8.8% on ALFWorld and 5.6% on WebShop using 1.5B models.
  • Scalability: Improvements were consistent across 7B models and various RL backbones.
• SWE-bench-Verified (Real-world Software Engineering):
  • Integrated into DeepSWE (a SOTA framework for software agents).
  • Result: Improved the success rate from 42.3% to 43.7% (+1.4%) on the highly challenging SWE-bench-Verified benchmark.
• Analysis:
  • Entropy Dynamics: Experiments confirmed that AEM prevents premature entropy collapse (a common failure mode where agents get stuck in local optima) and maintains higher diversity early in training.
  • Computational Cost: AEM introduces negligible overhead (~1.1% of total training time) as it reuses entropy calculations from the standard forward pass and requires no extra rollouts.

5. Significance and Impact

• Paradigm Shift: Moves credit assignment in agentic RL from reliance on external supervision (PRMs) or complex structural assumptions to intrinsic signal modulation (entropy).
• Efficiency: Offers a "plug-and-play" solution that significantly boosts performance without increasing training costs or requiring additional data annotation.
• Robustness: Demonstrates effectiveness across diverse domains (text-based navigation, e-commerce, and complex software engineering), suggesting that response-level entropy is a universal inductive bias for multi-turn agent optimization.
• Practicality: The method is particularly valuable for real-world applications like software engineering agents where outcome rewards are sparse and intermediate supervision is unavailable or expensive to generate.

In conclusion, AEM provides a theoretically grounded, computationally efficient mechanism to solve the credit assignment problem in multi-turn agentic RL, enabling LLM agents to learn more effectively from sparse, outcome-only feedback.