Technical Summary: Learning to Act under Noise (NoisyAgent)

Problem Statement

Recent advances in Large Language Models (LLMs) have enabled the deployment of interactive agents capable of reasoning, planning, and tool use. However, a fundamental discrepancy exists between the idealized training settings used in current research and the stochastic, imperfect nature of real-world deployments. Existing paradigms rely on carefully curated instructions and stable, well-controlled environments. In contrast, real-world interactions involve:

• User Noise: Ambiguity, variability, and unpredictable behaviors in user instructions.
• Tool Noise: Failures, incomplete outputs, misleading information, and anomalies in external tool execution.

This mismatch leads to significant performance degradation when agents are deployed in dynamic environments. Current training methods often fail to equip agents with the robustness necessary to recover from errors or resolve ambiguities encountered in practice.

Methodology: NoisyAgent

The authors propose NoisyAgent, an agentic Reinforcement Learning (RL) framework designed to explicitly incorporate environmental imperfections into the learning process. The framework consists of two primary components: an automatic noise injection pipeline and an adaptive training strategy.

1. Automatic Noise Injection Pipeline

The system simulates real-world imperfections by perturbing the agent-environment interaction at two levels:

• User-side Injection: Modifies interaction patterns before the task begins to simulate non-ideal user behaviors. This includes:
  • Ambiguous: Underspecified user intent.
  • Inconsistent: Conflicting or changing user needs over time.
  • Redundant: Inclusion of irrelevant information.
  • Constraint: The underlying task objective ($q$) is preserved to ensure solvability, while the interaction process becomes more stochastic.
• Tool-side Injection: Perturbs tool execution results during agent rollouts to simulate execution anomalies. This includes:
  • Failures: Tool requests returning errors.
  • Incomplete: Truncated outputs.
  • Misleading: Incorrect or inconsistent information.
  • Redundant: Unnecessary details in responses.

2. Adaptive Noise Training Strategy

To prevent the destabilization of training dynamics caused by uncontrolled noise, the authors employ a hybrid training scheme combined with curriculum learning:

• Hybrid Rollouts: For each task, the model generates a set of parallel rollouts. A subset ($N_{noise}$) is executed in perturbed (noisy) environments, while the remainder ($N - N_{noise}$) are executed in clean, idealized environments.
• Group-Wise Normalization: The training objective (based on Group Relative Policy Optimization, GRPO) computes advantages separately within the clean and noisy groups. This prevents either group from dominating the gradient updates and stabilizes learning under heterogeneous conditions.
• Noise Scheduling: The difficulty and proportion of noise are not static. The system quantifies the model's robustness via the performance gap ($\Delta$) between clean and noisy rollouts.
  • Training begins in fully idealized environments.
  • As the model adapts (i.e., $\Delta$ falls below a threshold $\theta$), the system progressively increases the noise scale (proportion of noisy rollouts) and noise difficulty (severity of perturbations).
  • This creates a curriculum that gradually increases interaction complexity without overwhelming the agent.

Key Contributions

1. Identification of a Training-Deployment Gap: The paper highlights that the reliance on idealized assumptions in current agent training is a primary cause of robustness failure in real-world settings.
2. Noise-Aware Framework: The development of a systematic framework that integrates instruction noise (user-side) and tool noise (tool-side) into the agentic training pipeline.
3. Adaptive Curriculum: A novel scheduling strategy that balances training stability with robustness by progressively increasing noise difficulty based on model performance.
4. Empirical Validation: Extensive experiments demonstrating that training under noise improves performance not only in noisy environments but also on standard, idealized benchmarks.

Experimental Results

The authors evaluated NoisyAgent using Qwen3-8B and Qwen3-32B backbones against strong baselines (GRPO, DAPO, GSPO) on AgentNoiseBench (noisy setting) and standard benchmarks like $\tau^2$-Bench and Vita-Bench (ideal setting).

• Robustness in Noisy Environments: NoisyAgent consistently outperformed all baselines on AgentNoiseBench across various domains (Retail, Airline, Telecom, Delivery, etc.). For instance, on the Retail domain with Qwen3-32B, NoisyAgent achieved an Avg@4 of 43.20 and Pass@4 of 65.79, significantly surpassing the next best method (GSPO: 37.72 / 60.53).
• Generalization to Ideal Settings: Contrary to the hypothesis that noise training might degrade performance on clean tasks, NoisyAgent achieved consistent gains on standard benchmarks. On the Retail domain of $\tau^2$-Bench (Qwen3-32B), it achieved an Avg@4 of 60.31 and Pass@4 of 86.84, outperforming GSPO (58.55 / 84.21).
• Behavioral Analysis:
  • Efficiency: Under noisy conditions, NoisyAgent reduced tool usage by 18% (from 13.9 to 11.4 calls per episode) compared to the base model, avoiding redundant interactions.
  • Clarity: The model produced longer, more detailed responses (2.1x increase in output tokens under noise), suggesting a shift toward explicit interaction to resolve ambiguity rather than relying on further tool calls.
• Ablation Studies: Removing controlled injection (applying noise to all rollouts) caused the largest performance drop, confirming the necessity of the hybrid training scheme. Removing the scheduling strategy also led to degradation, validating the importance of progressive difficulty.

Significance and Claims

The paper argues that controlled exposure to environmental noise serves as a form of implicit difficulty augmentation. By training agents to recover from errors, resolve ambiguities, and adapt to unexpected outcomes, the framework promotes more generalizable reasoning and decision-making behaviors.

The authors claim that their findings bridge the gap between agent training and real-world deployment by:

1. Validating that robustness is an emergent property of exposure to diverse imperfections during learning.
2. Demonstrating that training with noise does not compromise, but rather enhances, performance in idealized settings.
3. Providing a practical, scalable framework (NoisyAgent) that can be applied to any agentic environment to improve reliability in stochastic, real-world scenarios.

The work concludes that modeling interaction imperfections is essential for developing LLM-based agents that can function effectively outside of curated, controlled benchmarks.