🤖 AI

Agent-G $^2$ : Gaussian Guidance for Agentic Reinforcement Learning

Agent-G $^2$ is a novel reinforcement learning framework that optimizes hint-based guidance depth by sampling from an online-estimated Gaussian distribution, thereby outperforming existing deterministic and probing-based methods in agentic tasks while significantly reducing computational costs.

Original authors: Zixuan Wang, Yanrui Miao, Zhengxi Lu, Teng Pan, Yiwen Qiu, Hongxing Li, Peng Qiu, Ruiqing Zhang, Yongliang Shen

Published 2026-08-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zixuan Wang, Yanrui Miao, Zhengxi Lu, Teng Pan, Yiwen Qiu, Hongxing Li, Peng Qiu, Ruiqing Zhang, Yongliang Shen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a computer program trying to learn how to navigate a complex world, like a robot moving through a house or a virtual agent shopping on a website. These tasks often require a long chain of decisions, where the program must take many steps before it finally succeeds or fails. The core problem is that the program only receives a signal at the very end telling it whether it won or lost. This makes learning incredibly difficult, like trying to learn to ride a bicycle by only being told "good job" or "you fell" after you have already traveled a mile, with no feedback on whether you were steering correctly in the middle. To help the program learn, researchers have tried giving it a "hint" at the start of each attempt: a short snippet of a perfect path taken by an expert. This gives the program a head start, placing it closer to the finish line so it can learn from the remaining steps. However, a critical question remains: how much of that expert path should be shown? Show too little, and the program is still lost; show too much, and it simply copies the expert without learning anything new.

For a long time, researchers treated this question as if there were one perfect amount of help that applied to every single task, or they tried to guess the right amount for each specific task by running many expensive, extra practice rounds just to test different levels of help. A new study by a team at Zhejiang University and Baidu suggests that both of these approaches miss the mark. The researchers found that the "sweet spot" for help is not a single, precise point, but rather a range. For any given task, there is a band of different amounts of help that work well, and this band shifts depending on how difficult the task is. Instead of trying to pinpoint one exact number, the team developed a method that treats the right amount of help as a flexible range that changes as the computer learns. They call their system Agent-G2.

The researchers tested their idea on two challenging environments: a text-based world where an agent must complete household chores like picking up objects or cleaning, and a simulated online store where the agent must search for and buy specific items. They used a language model as the brain for these agents. In their experiments, they discovered that the most effective amount of help forms a bell-shaped curve around the ideal point. This means that for a difficult task, a wide range of help levels might work, while for an easier one, the range might be narrower. Crucially, they found that the difficulty of a task is closely tied to how long the expert's path is. A task requiring twenty steps is generally harder than one requiring only two.

To solve the problem of finding the right help level without wasting time, Agent-G2 groups tasks by their length and then uses the results of the current practice rounds to adjust the help for the next round. If the group of short tasks is succeeding too easily, the system automatically reduces the amount of help given to them. If the long tasks are failing, it increases the help. The system does this by calculating a center point and a spread for each group of tasks, essentially creating a dynamic map of where the learning should happen. It then randomly picks a specific amount of help from that map for each new attempt. This approach allows the system to explore a variety of helpful starting points simultaneously, ensuring that both easy and hard tasks get the right kind of push without needing to run extra tests just to figure out the settings.

The results were striking. On the household task environment, the new method achieved a success rate of 95.3 percent with a smaller model and 98.4 percent with a larger one. This outperformed the strongest existing methods that used hints, as well as those that tried to learn without any hints at all. Perhaps most importantly, the new method achieved these results using far fewer computer resources than the previous best methods that tried to guess the right help level for each task individually. Those older methods required running many extra practice rounds to find the answer, which is slow and expensive. The new system learned the right settings on the fly, using the same data it was already collecting to improve the agent.

The study also showed that the system works well across different sizes of computer models, proving that the way the help is delivered is just as important as the size of the brain doing the learning. In fact, the smaller model using this smart guidance system performed better than much larger models using older, less sophisticated methods. The researchers noted that their method relies on having a perfect example path available for every task to start with. If such perfect paths do not exist, the system cannot be used directly. However, for the many real-world problems where experts can demonstrate the correct path, this approach offers a powerful way to teach artificial intelligence how to learn from its own mistakes, rather than just copying what it sees. By treating the right amount of guidance as a flexible range rather than a fixed number, the researchers have found a more efficient and robust way to help machines master complex, long-term goals.

Technical Summary: Agent-G2: Gaussian Guidance for Agentic Reinforcement Learning

1. Problem Statement

Reinforcement Learning (RL) for Large Language Model (LLM) agents faces a critical challenge in long-horizon decision-making tasks: reward sparsity. In environments like web navigation or embodied control, agents often receive only a binary success signal at the terminal state after dozens of sequential decisions. Consequently, on-policy exploration from the initial state rarely reaches a successful terminal state, leading to "advantage collapse" in training algorithms like Group Relative Policy Optimization (GRPO).

Hint-based RL attempts to mitigate this by retaining a prefix of an expert trajectory before each rollout, allowing the policy to explore from a state closer to success. However, the effectiveness of this approach hinges on the guidance depth ( $d$ ): the proportion of the expert trajectory to retain.

Too little guidance: The policy starts too early, resulting in few successful rollouts.
Too much guidance: The reward becomes saturated, eliminating the contrast necessary for effective advantage estimation.

Existing methods treat this depth as a deterministic scalar, leading to two suboptimal paradigms:

Schedule-based methods: Share a single depth value across all samples in a batch (e.g., based on training steps). This ignores per-task heterogeneity, as tasks within the same batch can vary significantly in difficulty (e.g., a 2-step task vs. a 20-step task).
Per-sample probing methods: Estimate an optimal depth for each sample via binary search or enumeration. While more accurate, this incurs a high computational cost ( $O(\log n)$ or $N \times$ extra rollouts per sample) and introduces noise under limited rollout budgets.

The authors identify a fundamental structural mismatch in these approaches: they assume a single optimal depth exists per task. Empirical analysis reveals that useful guidance occupies a "band" of depths around an optimal point, with an informativeness profile that is approximately Gaussian.

2. Methodology: Agent-G2

The authors propose Agent-G2, a Gaussian guidance framework that models the guidance depth as a per-task Gaussian distribution ( $N(\mu_i, \sigma_i^2)$ ) rather than a scalar. This distribution is estimated online from rollout statistics collected during policy optimization, requiring no extra probe rollouts or learned depth predictors.

Core Components

Offline Clustering:
Tasks are partitioned into $K$ clusters based on expert trajectory length (a proxy for difficulty). This allows the system to handle heterogeneity by maintaining separate statistics for different difficulty levels.
Adaptive Gaussian Schedule:
For each task $i$ in cluster $k$ , the parameters $\mu_i$ (center) and $\sigma_i$ (spread) are computed dynamically:
- Global Baseline ( $\mu_{global}$ ): Tracks the overall policy progress. It is adjusted based on the batch success rate ( $acc_B$ ) relative to a target success rate ( $p_{target} = 0.5$ ), the "just-learnable" regime.
- Cluster Statistics: For each cluster $k$ , the system maintains Exponential Moving Averages (EMA) of the empirical success rate ( $A_k$ ) and variance ( $V_k$ ).
- Parameter Calculation:
  - $\mu_i = \text{clip}(\mu_{global} + \lambda(p_{target} - A_k), 0, 1)$ : Adjusts the center based on cluster difficulty relative to the global baseline.
  - $\sigma_i = \max(\gamma V_k, \sigma_{min})$ : Sets the spread based on the variance within the cluster, ensuring broader coverage for heterogeneous tasks.
Sampling and Rollout:
For each task, a guidance ratio $r_i$ is sampled from $N(\mu_i, \sigma_i^2)$ and clipped to $[0, 1]$ . This determines the prefix length $n_i$ . The agent executes the expert prefix to reach a state $s_i$ , then performs $R$ independent rollouts from $s_i$ .
Training Loop:
- Policy Update: The policy is updated using a combination of the GRPO loss (on the $R$ rollouts) and an auxiliary loss ( $L_{aux}$ ) on the sampled expert prefixes (imitation learning).
- Schedule Update: The same rollouts used for policy updates refresh the global baseline and cluster statistics ( $A_k, V_k$ ) for the next batch. This creates a closed-loop system where the guidance distribution evolves with the policy's capabilities.

3. Key Contributions

Structural Insight: The paper reveals that informative guidance depths form a band rather than a single point. The training signal profile within this band is unimodal, symmetric, and well-fit by a Gaussian distribution. This explains the failure modes of both shared-depth schedulers (structural mismatch) and per-sample probing (noise/cost trade-off).
Agent-G2 Framework: A novel framework that estimates per-task depth distributions online using a global-local decomposition. It achieves per-task depth variation without extra probe rollouts or auxiliary networks.
Empirical Superiority: Extensive evaluation on ALFWorld and WebShop using Qwen2.5-1.5B and 7B models demonstrates that Agent-G2 outperforms state-of-the-art baselines (including hint-based, hint-free, and auxiliary-RL methods) while operating at under one-third the rollout cost of per-sample probing methods.

4. Experimental Results

ALFWorld: Agent-G2 achieved 95.3% success (1.5B model) and 98.4% (7B model). This represents improvements of +2.3 and +3.9 points over the strongest hint-based and hint-free baselines, respectively.
WebShop: Agent-G2 achieved a 92.3 reward score and 84.4% purchase success (7B model), outperforming all non-probing baselines.
Efficiency: While Agent-G2 has a modest per-step overhead compared to simple schedule-based methods, it is significantly cheaper (3.24x–4.83x faster) than probing-based methods (Binary Search, Enumeration) because it avoids extra rollouts.
Ablation Studies:
- Removing stochastic sampling (using only the mean $\mu_i$ ) dropped performance by 5.5 points, confirming the necessity of covering the informative band.
- Removing cluster-specific adaptation ( $K=1$ ) dropped performance by 6.2 points, highlighting the importance of handling task heterogeneity.
- The auxiliary loss ( $L_{aux}$ ) provided stability (+8.6 points) but was not the primary driver of performance; the GRPO learning on post-prefix states was critical.

5. Significance and Claims

The paper claims that Agent-G2 represents a shift from treating guidance depth as a static, deterministic hyperparameter to a dynamic, probabilistic variable. By modeling the "neighborhood" of useful guidance rather than a single point, the method effectively balances exploration and exploitation in long-horizon tasks.

The authors emphasize that schedule design can substitute for backbone scaling. Notably, the 1.5B model with Agent-G2 surpassed the 7B BEACON baseline, suggesting that intelligent guidance strategies can compensate for limited model capacity.

Limitations Acknowledged:

The method requires one expert trajectory per task; it cannot be directly applied where such data is unavailable.
The Gaussian parameterization is empirically motivated; tasks with multimodal or heavily skewed depth profiles might require richer distributions.
Difficulty clustering is currently based on offline trajectory length; online difficulty estimation is a potential future direction.

In summary, Agent-G2 provides a cost-effective, scalable solution to reward sparsity in agentic RL by leveraging the statistical structure of guidance depth, eliminating the need for expensive probing while adapting to task heterogeneity in real-time.