StressWeb: A Diagnostic Benchmark for Web Agent Robustness under Realistic Interaction Variability

1. Problem Statement

Large Language Model (LLM) based web agents have demonstrated impressive capabilities in complex tasks like form filling and online shopping. However, current evaluation benchmarks (e.g., WebArena, WebVoyager) predominantly operate under stable, idealized, and well-behaved interaction conditions.

• The Gap: Real-world web environments are inherently dynamic, featuring asynchronous responses, layout shifts, delayed updates, and evolving interaction semantics.
• The Consequence: High success rates in clean benchmarks do not necessarily translate to robustness in realistic settings. Existing evaluations fail to stress-test agents against the variability that causes failure in production, leading to an overestimation of agent reliability.

2. Methodology: The StressWeb Framework

The authors propose StressWeb, a diagnostic stress-testing benchmark designed to systematically evaluate agent robustness by introducing controlled perturbations into realistic web environments.

A. Benchmark Construction

• Reference Environments: The team constructed 10 diverse, full-stack web applications (covering e-commerce, scheduling, email, file management, etc.) containing 149 task queries. These environments are generated via an automated, task-driven pipeline to ensure structural coherence, persistent state, and deterministic execution.
• Interaction Pipeline Decomposition: The framework views agent interaction as a three-stage pipeline:
  1. Perception: Observing the interface (visual/DOM).
  2. Action Selection: Deciding on the next step based on semantics.
  3. Execution: Realizing the action and updating the state.

B. Structured Perturbations

StressWeb injects specific, controllable perturbations aligned with the interaction pipeline to simulate real-world variability:

1. Perception Perturbations:
   • Layout Chaos (Visual): Modifies rendered presentation (text size, rotation, position shifts).
   • DOM Noise (Structural): Introduces structural redundancy, hidden elements, decoys, and mild content obfuscation.
2. Semantic Perturbations (Action Selection):
   • RemapE (Explicit): Changes interaction rules (e.g., requiring double-clicks) with explicit textual instructions.
   • Remap (Implicit): Changes rules without instructions; the agent must infer the new rule (e.g., single-click only selects, double-click navigates) through feedback.
3. Execution Perturbations:
   • Failure: Randomly causes actions (clicks, inputs) to fail.
   • Pop-Up: Introduces disruptive pop-up windows that interrupt the task flow.

C. Evaluation Protocol

• Deterministic Multi-Checkpoint Evaluation: Instead of relying solely on final task completion, the system uses script-based evaluators to verify intermediate milestones (checkpoints) against the DOM state.
• Metrics:
  • Checkpoint Pass Rate (ckpt%): Success rate at specific milestones.
  • Interaction Steps: Efficiency of the trajectory.
  • Self-Assessment Reliability: Comparison between the agent's claimed success and actual success.

3. Key Contributions

1. Diagnostic Stress-Testing Benchmark: Introduction of StressWeb, the first benchmark to systematically evaluate web agent robustness under controlled, realistic interaction variability across perception, selection, and execution stages.
2. Stage-Aligned Perturbation Mechanisms: A novel design that isolates failure modes by injecting specific interferences at distinct stages of the agent's decision loop.
3. Comprehensive Empirical Analysis: Extensive evaluation of state-of-the-art multimodal agents (including GPT-5.2, Claude 4.5, Gemini 2.5-Pro, Qwen3, etc.) revealing hidden failure modes not visible in clean benchmarks.

4. Key Results & Findings

The evaluation of 8 SOTA models and a human baseline yielded several critical insights:

• Significant Robustness Gaps: While agents achieved 46–61% success in clean environments, performance dropped to 34–50% under perturbations. The human baseline maintained near-100% success, proving the tasks themselves were not too difficult, but the agents lacked adaptability.
• Hierarchy of Failure Modes:
  • Perception: Agents showed moderate robustness to visual/DOM noise (performance drop ~5-8%).
  • Execution: Agents struggled with execution uncertainty (drop ~8-11%).
  • Semantic: The most severe degradation. When interaction rules changed (especially implicitly), success rates plummeted to 26–31%. This indicates agents rely heavily on rigid, learned priors rather than dynamic grounding.
• Online Adaptation Failure: Under semantic perturbations, agents failed to revise incorrect hypotheses. Instead of adapting, they often entered repetitive action loops (e.g., clicking the same button 90+ times) or terminated prematurely.
• Self-Assessment Miscalibration: Agents exhibited severe overconfidence. Under semantic perturbations, the ratio of "claimed success" to "actual success" reached up to 5.3x. Agents frequently reported task completion even when they had failed, posing a risk of "silent failures" in real-world deployment.
• Model Ranking Instability: Models that performed best in clean environments (e.g., o4-mini, Gemini 2.5-Pro) did not necessarily maintain the top ranking under stress, highlighting that robustness is a distinct capability dimension from raw task capability.

5. Significance and Implications

• Redefining Evaluation: The paper argues that robustness must be treated as a "first-class" evaluation dimension alongside task success. High performance in static environments is insufficient for real-world deployment.
• Diagnostic Value: StressWeb moves beyond aggregate success rates to provide mechanistic insights into why agents fail (e.g., inability to infer implicit rules, lack of error recovery).
• Future Directions: The findings suggest that future web agents need:
  • Mechanisms for online adaptation to infer changing interaction rules.
  • Improved self-assessment calibration to detect failure and avoid silent errors.
  • Training strategies that expose agents to compositional perturbations (multiple stressors simultaneously).

In summary, StressWeb exposes a critical fragility in current LLM-based web agents: they are highly capable in idealized settings but lack the resilience to handle the dynamic, noisy, and rule-shifting nature of the real web.