Co-Evolution of Policy and Internal Reward for Language Agents

1. Problem Statement

Large Language Model (LLM) agents operating in long-horizon interactive environments (e.g., web navigation, scientific experimentation) face a fundamental sparse reward bottleneck.

• The Challenge: Agents typically receive feedback only at the end of a trajectory (terminal reward), making it difficult to assign credit to intermediate decisions. This hinders both inference-time decision-making (unclear if the current path is correct) and training-time policy optimization (lack of dense signals for gradient updates).
• Limitations of Existing Methods:
  • Post-hoc credit assignment: Redistributes credit after trajectories are collected but offers no guidance during inference.
  • External Reward Models: Introduce overhead, potential distribution drift, and separate the reward generation process from the policy, preventing a unified learning loop.
  • Static Self-Reflection: Methods that use fixed self-guidance signals often fail because the guidance quality depends on the policy's maturity, creating a "chicken-and-egg" problem.

2. Methodology: Self-Guide

The authors propose Self-Guide, a unified framework where the agent generates its own internal reward signal that serves dual purposes: inference-time guidance and training-time supervision.

Core Mechanism

The agent operates in a loop where, at each step $t$:

1. Self-Generation: The model generates a verbal self-guidance signal $z_t$ (a natural language assessment of progress) based on the history $h_{t-1}$ and current observation $o_t$.
   $$z_t \sim \pi_\theta(\cdot | h_{t-1}, o_t)$$
2. Action Selection: The model selects the next action $a_t$ conditioned on both the history and the self-generated signal.
   $$a_t \sim \pi_\theta(\cdot | h_{t-1}, o_t, z_t)$$
3. Reward Conversion: The verbal signal $z_t$ is mapped to a scalar internal reward $r^{sg}_t$ (e.g., +0.1 for "positive/progress", 0 for "neutral", -0.1 for "negative/harmful").
4. Composite Reward: The internal reward is aggregated and combined with the sparse environment reward $R_{env}$ to form a unified training signal:
   $$R(\tau; u) = R_{env}(\tau) + \lambda(u) \sum_{t=1}^{T} r^{sg}_t$$
   Where $\lambda(u)$ is a stage-dependent coefficient controlling the weight of the internal reward.

Key Innovation: Stage-Wise Trust Schedule

To address the instability of using immature self-generated rewards early in training, the authors introduce a trapezoid trust schedule for $\lambda(u)$:

• Phase I (Warm-up): $\lambda = 0$. The agent uses self-guidance for inference (steering actions) but receives no internal reward. This allows the policy to learn to generate useful guidance purely from environmental feedback without being penalized by its own immature judgments.
• Phase II (Activation): $\lambda$ ramps linearly from 0 to 1. The internal reward is gradually introduced to avoid optimization shock.
• Phase III (Full Co-evolution): $\lambda = 1$. The agent fully utilizes the dense internal reward, creating a self-reinforcing loop where better policy $\to$ better guidance $\to$ better reward $\to$ better policy.
• Phase IV (Annealing): $\lambda$ ramps down from 1 to 0. This ensures the final policy converges to the true environment objective, preventing the policy from overfitting to the internal reward signal (which is not potential-based and could bias the solution).

3. Key Contributions

1. Self-Guide Framework: A novel method where the agent generates a single signal ($z_t$) that acts as both an inference-time steering mechanism and a training-time dense reward source.
2. Co-Evolutionary Training: Demonstrates that policy and internal reward must improve together online. The paper shows that offline distillation of self-guidance from a larger teacher model fails to transfer reliably due to distribution mismatch, whereas online co-evolution succeeds.
3. Stage-Wise Trust Schedule: A robust scheduling mechanism that stabilizes the training of self-generated rewards, solving the bootstrap problem of using immature signals for optimization.
4. Unified Optimization: The method is compatible with various RL algorithms (verified with GRPO and DAPO) and does not require external reward models or separate evaluators.

4. Experimental Results

The method was evaluated on three benchmarks: ALFWorld (household tasks), ScienceWorld (scientific reasoning), and WebShop (e-commerce). Models tested included Qwen3-1.7B, Qwen3-4B, and Qwen2.5-7B.

• Inference-Time Gains: Even without RL training, adding Self-Guide to ReAct (ReAct w/ SG) significantly improved performance on structured tasks (e.g., +10.9% on ALFWorld for Qwen3-1.7B) by reducing looping and redundant exploration.
• Training-Time Gains:
  • GRPO w/ SG & GR (Full method) consistently outperformed the baseline GRPO across all models and environments.
  • Quantitative Improvement: On Qwen3-4B, the full method achieved an ~8% average improvement over GRPO trained solely on environment rewards.
  • Ablation Studies:
    • Immediate Reward: Introducing internal reward immediately (without warm-up) performed worse than the baseline, confirming the need for the trust schedule.
    • Offline Distillation: Using a distilled self-guidance model from a larger teacher failed to provide stable gains, proving the necessity of online co-evolution.
    • Annealing: Removing the late annealing phase slightly degraded performance, highlighting the importance of aligning the final policy with the true environment objective.

5. Significance

This work fundamentally shifts the paradigm of agent training from relying solely on external feedback to learning to self-evaluate.

• Efficiency: It eliminates the need for external reward models or process reward models (PRMs), reducing computational overhead and engineering complexity.
• Scalability: By enabling agents to generate their own dense supervision signals, it addresses the sparse reward problem inherent in complex, long-horizon tasks.
• Stability: The stage-wise trust schedule provides a practical recipe for stabilizing the inherently unstable process of training an agent on its own generated rewards.
• Generalizability: The approach is algorithm-agnostic (works with GRPO, DAPO) and model-agnostic, suggesting a broad applicability for improving LLM agents in diverse interactive domains.

In conclusion, the paper demonstrates that language agents can significantly improve their performance not just by collecting more experience, but by learning to generate and refine their own internal reward signals during the acting and learning process.