Technical Summary: Where Did It Go Wrong? Process-Level Evaluation of Web Agents with Semantic State Tracking

Problem Statement

Web agents automate complex, multi-step interactive tasks such as shopping, booking, and document management. Unlike single-turn instruction following, these tasks require long interaction sequences involving navigation, information gathering across pages, and dependent action execution. However, existing benchmarks (e.g., WebArena, Mind2Web, WebVoyager) primarily evaluate agents based on terminal success—a binary outcome indicating whether the final task goal was achieved.

This outcome-only view compresses the entire interaction trajectory into a single result, obscuring qualitatively different failure modes. For instance, an agent that finds the correct target but performs the wrong final action receives the same score as an agent that never finds the target at all. Consequently, terminal success provides little actionable insight into where an agent failed, which capability limited performance, or how to improve specific interaction skills. It fails to support accurate credit assignment for intermediate steps.

Methodology: WEBSTEP and Semantic MDP Duals

To address these limitations, the authors introduce WEBSTEP, a benchmark designed for process-level evaluation. The core innovation is the construction of each website as a Semantic Markov Decision Process (MDP) dual.

1. Semantic MDP Dual Architecture

Instead of relying on manual annotation or heuristic alignment of GUI actions, WEBSTEP constructs each website such that the semantic MDP is the "source of truth" for all behavior:

• Dual Construction: The agent interacts only with the rendered Graphical User Interface (GUI). Simultaneously, the benchmark tracks the corresponding semantic state transitions underneath.
• Deterministic Dynamics: The frontend forwards GUI actions (e.g., clicks, typing) to a semantic model. The model applies deterministic state transitions, updates the internal state, and re-renders the page based on that state.
• Automatic Trace Recovery: Because the GUI is rendered directly from the semantic model, the full semantic trajectory is recovered exactly from raw GUI actions without manual intervention.

2. Task and World Generation

Tasks are generated by combining natural language instructions with a "world" (a set of items with concrete attributes).

• Hard Negatives: To control difficulty, the system injects "hard negative" items that share visible attributes with the target on list/search pages but differ on attributes visible only on detail pages. This forces agents to navigate to detail pages to discriminate targets.
• Oracle Trajectories: Since the MDP and item sets are fully specified, an optimal (oracle) trajectory can be statically derived for every task, serving as a ground truth for process metrics.

3. Process-Level Evaluation Framework

The framework decomposes agent behavior into several metrics derived automatically from the semantic MDP trace:

• Exploration Success: Whether the agent isolated the correct target item before committing.
• Execution Success: The success rate conditioned strictly on successful exploration (isolating the ability to finalize a task once the target is found).
• Informational Coverage: The fraction of task-relevant attributes witnessed by the agent.
• Skill Invocation: Mapping actions to five categories: Inspect, Navigate, Search, Filter, and Commit.
• Trajectory Bifurcation Analysis: Comparing successful and failing trajectories to identify the "first divergent action" (the decisive error point) and categorizing failures as Wrong Branch, Delayed Commit, or Premature Commit.

Key Contributions

1. WEBSTEP Benchmark: A dataset of 1,800 task instances across 10 self-hosted websites (spanning domains like e-commerce, housing, coding, and productivity). Each task is paired with a semantic MDP dual for automatic state tracking.
2. Process-Level Evaluation Framework: A methodology that decomposes behavior into exploration, execution, and skill invocation, enabling trajectory comparison at shared semantic states to localize failure modes without manual annotation.
3. Empirical Analysis of Web Agents: A comprehensive evaluation of five agents (Fara-7B, UI-TARS-1.5-7B, GUI-Owl-1.5-8B, Qwen3.5-122B, and OpenAI CUA) revealing that terminal success rates often mask significant behavioral differences.

Results and Findings

The authors evaluated five agents, finding that while small models clustered in a narrow terminal success range (31–33%), process-level metrics revealed distinct behavioral profiles:

1. Terminal Success Hides Behavioral Differences:

   • UI-TARS showed higher exploration success but lower execution success, indicating it finds targets but struggles with the final action.
   • Fara exhibited the lowest step count and information coverage, suggesting under-exploratory behavior.
   • OpenAI CUA and Qwen3.5 performed better in both exploration and execution, with CUA showing higher efficiency.

2. Skill-Specific Weaknesses:

   • Agents exhibit distinct skill profiles. For example, on the Housing domain, OpenAI CUA outperformed Qwen3.5 by 23.7% on Commit actions but underperformed by 15.6% on Filter actions.
   • This demonstrates that an agent's weakness can be concentrated in a specific interaction skill within a single domain, which broad website-level scores would obscure.

3. Decisive Error Localization:

   • Bifurcation analysis showed that the step where agents first diverge from the optimal path varies by model.
   • GUI-Owl most often failed at the Filter step.
   • Qwen3.5 and others most often failed at the Inspect step (opening the wrong item page).
   • Premature Commitment: Many agents (e.g., Fara) often skipped the Inspect step, committing to a target before gathering sufficient evidence, leading to failures on hard negatives.

4. Complexity Sensitivity:

   • As task complexity increased (measured by hard negative count, oracle trajectory length, and information access level), the performance gap between agents widened.
   • On easy tasks, agents performed similarly; on complex tasks requiring deep exploration, OpenAI CUA maintained performance while others degraded significantly.

Significance and Claims

The paper claims that WEBSTEP opens a new avenue in web agent evaluation by shifting the focus from ranking agents based on terminal outcomes to diagnosing how and where they fail.

• Actionable Insights: By identifying specific skill deficits (e.g., poor filtering vs. poor committing) and failure patterns (e.g., premature commitment), the framework provides concrete targets for improvement, such as training on specific interaction types or adjusting exploration strategies.
• Diagnosis over Ranking: The authors argue that process-level evaluation transforms evaluation from a black-box ranking system into a diagnostic tool that can disentangle exploration failures from execution failures.
• Reproducibility: The deterministic nature of the semantic MDPs ensures that the environment dynamics and verification outcomes are fully reproducible, addressing a key limitation of live-web benchmarks.

The work concludes that terminal success alone is insufficient for understanding agent capabilities and that fine-grained, process-level analysis is necessary to guide the development of more robust web agents.