Technical Summary: When Web Agents Finish but Still Fail

Problem Statement

Long-horizon web agents are increasingly evaluated on tasks requiring navigation, information gathering, and reasoning over real websites. However, standard evaluation metrics often mask a critical failure mode: agents may successfully emit a final, well-formed answer and terminate confidently while still failing the core task. These "finish-but-fail" scenarios include missing required fields, over-including unsupported items, or relying on stale evidence.

Unlike sequential single-answer retrieval tasks common in existing benchmarks (e.g., WebWalkerQA, WebSailor), realistic web tasks often require parallel web exploration. In this setting, an agent must decompose a query into multiple sub-goals, retrieve information from disparate sources (e.g., different subpages of a conference site), and aggregate these into a structured output. Current evaluation methods, which often rely on binary completion or final-answer accuracy, fail to capture the nuance of agents that complete the interaction loop but produce factually incomplete or incorrect structured answers.

Methodology

Benchmark Construction: Parallel WebBench

To address the lack of evaluation data for parallel exploration, the authors introduce Parallel WebBench, a benchmark containing 1,679 verified records. The dataset is constructed via two pipelines:

1. Manual Curation (350 tasks): Root URLs are converted into multi-field questions requiring evidence from multiple subpages. Human annotators verify answerability, record sequential steps, maximum parallel degree (the number of distinct evidence branches required), visited URLs, and golden answers.
2. Automated Reconstruction (1,329 tasks): Existing web QA records are filtered and reconstructed. This pipeline verifies if answers remain valid on the live web, classifies records by path validity, and reconstructs broken URL chains using strict link-chain verification (ensuring every consecutive URL pair is a valid hyperlink).

The dataset emphasizes live evidence, filtering out stale content and ensuring trajectories are navigable on the current web.

Experimental Setup

The authors train WebExplorer-style agents using Group Relative Policy Optimization (GRPO) under three data mixture conditions:

• HUMAN: Only manually verified human-curated tasks.
• BALANCED: Equal parts human tasks and synthetic data (Category A: verified paths; Category B: reconstructed paths).
• SYNTH-HEAVY: Reduced human supervision with a high proportion of synthetic data.

Reward Design: To address reward sparsity in structured multi-field tasks, the authors employ a partial-credit reward system. Unlike exact-match rewards that assign zero credit for any error, this system recursively scores dictionaries, lists, and scalar fields using fuzzy matching, containment, and token-level F1. This provides denser feedback while preserving a non-LLM training loop.

Evaluation: Models are evaluated on a held-out set of 150 examples under varying inference budgets (16k/64k context, 8/16 interaction rounds). Metrics include:

• Completion: Whether a parseable final answer is emitted.
• Binary Accuracy: Whether the full structured answer is judged correct by GPT-4.1-mini.
• Element-wise F1: Partial credit based on the recovery of individual factual elements.

Key Contributions

1. Definition of Parallel Web Exploration: The paper defines a task setting where agents must retrieve and synthesize multiple independent sub-goals, exposing a "completion-correctness gap" where agents emit structured answers that are partially supported or entirely unsupported by the retrieved evidence.
2. Parallel WebBench: A benchmark of 1,679 records with verified trajectories, answers, and URL-level metadata, distinguishing between human-curated and automatically reconstructed tasks.
3. Identification of Three Reproducible Failure Modes: Through trace-level diagnostics, the authors identify specific triggers for agent failure:
   • FM1 (Context-bound search loops): Agents enter redundant search loops when canonical phrasing mismatches occur, failing to utilize non-canonical evidence retrieved earlier.
   • FM2 (Premature termination on partial answers): Agents emit a final answer before exhausting the interaction budget, often triggered by list-valued questions, driven by dense rewards that credit format/process over coverage.
   • FM3 (Synthesis collapse): Agents retrieve sufficient evidence but fail to preserve it in the final output, collapsing distinct rows into repeated values or replacing facts with "Not found" placeholders.
4. Trace-Level Diagnostics: The paper provides operational detectors for these failure modes, analyzing redundant search behavior, round usage, element-level recall, and evidence-to-answer binding.

Results

Performance Improvements and Gaps

Training with GRPO and synthetic data significantly improves completion rates but yields diminishing returns on correctness:

• Completion: The best GRPO model (SYNTH-HEAVY) improved completion from 50.7% (base WebExplorer-8B) to 96.0% at 16k context/16 rounds.
• Partial Correctness: Element-wise F1 improved from 0.2489 to 0.4529.
• Binary Accuracy: Despite high completion, binary accuracy remained low (~33.3%), indicating that while agents finish tasks, they often fail to solve them correctly.

Analysis of Interventions

• Synthetic Data: Synthetic-heavy training reduces abstention and improves answer emission reliability but does not significantly improve evidence-grounded correctness. Models trained on synthetic data perform better on synthetic test sets than on human-verified sets.
• Context and Rounds: Increasing context (to 64k) or interaction rounds (to 16) improves completion but does not monotonically improve binary accuracy. In some cases, larger budgets led to a decrease in element-wise F1, suggesting that more interaction does not fix core reasoning or synthesis failures.
• Failure Mode Persistence: The three failure modes (loops, premature termination, synthesis collapse) persist across all training mixes. Synthetic-heavy training reduced FM1 (loops) but increased FM2 (premature termination), suggesting a trade-off where agents learn to "finish" faster but with less verification.

Significance and Claims

The paper argues that outcome-level optimization (maximizing completion or final answer emission) is insufficient for reliable agentic AI. The central finding is constructive: while GRPO and synthetic supervision effectively teach agents to emit answers, they do not eliminate the fundamental failure modes of web exploration.

The authors claim that the field must move beyond simple completion metrics. The observed "completion-correctness gap" highlights the need for:

• Coverage-aware stopping: Preventing agents from terminating before all sub-goals are verified.
• Evidence-bound synthesis: Ensuring the final answer strictly adheres to retrieved evidence rather than collapsing or hallucinating.
• Trace diagnostics: Separating the act of "finishing" a task from "solving" it.

The work suggests that future interventions should combine partial-credit training with explicit coverage checks and verification steps, rather than relying solely on scaling context, interaction budgets, or synthetic data. The paper concludes that current agents are becoming more confident in their failures, necessitating diagnostics that can distinguish between a solved task and a merely completed interaction.