Why Do LLM-based Web Agents Fail? A Hierarchical Planning Perspective

This paper introduces a hierarchical planning framework to diagnose LLM web agent failures across high-level planning, low-level execution, and replanning layers, revealing that while structured PDDL plans improve strategy formulation, low-level execution and perceptual grounding remain the primary bottlenecks to achieving human-level reliability.

The Gist

Imagine you hire a very smart, well-read but slightly clumsy robot assistant to do your online shopping or research for you. You give it a big, complex instruction like, "Find me the cheapest winter coat for my dog in my size, and make sure it's on sale."

This paper is essentially a "autopsy report" on why these robot assistants (called LLM Web Agents) often fail at these tasks, even though they are getting smarter.

The authors argue that we've been looking at the wrong thing. Instead of just asking, "Did the robot finish the job? Yes or No?", they decided to look at how the robot thinks and acts. They broke the robot's brain down into three distinct layers, like a construction crew building a house.

Here is the breakdown using simple analogies:

1. The Three Layers of the Robot's Brain

The authors say every web agent has three jobs, and they fail at different points in each:

• Layer 1: The Architect (High-Level Planning)

  • The Job: This is the part that reads your request and makes a master plan. It says, "First, go to the store. Second, find the dog section. Third, filter by size."
  • The Problem: Sometimes the Architect gets confused. It might write a plan that is too vague ("Find a coat") or too specific ("Click the third button on the left"), missing the big picture.
  • The Fix: The paper found that if you force the Architect to write the plan in a strict, structured language (like a computer code called PDDL) rather than just chatting in normal English, the plans become much clearer and more logical. It's like switching from a messy handwritten note to a precise blueprint.

• Layer 2: The Builder (Low-Level Execution)

  • The Job: This is the part that actually touches the screen. It has to click the right button, type the right words, and scroll to the right spot.
  • The Problem: This is where the robot fails the most. Even if the Architect draws a perfect blueprint, the Builder is clumsy. It might click the wrong link, get stuck in a loop clicking the same button over and over, or get distracted and wander off to a different website (like searching for "dog coats" on a flight tracker site).
  • The Metaphor: Imagine a brilliant architect who designs a perfect bridge, but the construction crew keeps tripping over their own shoelaces and dropping bricks. The paper found that this clumsiness is the biggest bottleneck. The robot just isn't good enough at "seeing" the website and clicking the right things yet.

• Layer 3: The Fixer (Replanning)

  • The Job: When things go wrong (the website crashes, or the button doesn't work), this part has to say, "Okay, Plan A failed. Let's try a different way."
  • The Finding: The paper found that giving the robot a "second chance" to rethink its strategy after it gets stuck actually helps a lot. It's like telling a lost hiker, "Okay, that path is blocked. Let's look at the map again and find a new route." This simple step of "re-planning" significantly improved success rates.

2. The Big Takeaways

The authors ran experiments with three different "brains" (AI models) and found three main things:

1. Structure helps the Planner: When you force the AI to write its plan in a structured format (PDDL) instead of free-flowing text, it makes better, more concise plans. It stops rambling and gets straight to the point.
2. The "Hands" are the problem: The biggest issue isn't that the AI can't think; it's that it can't do. It struggles to translate a good idea into a specific click or type action. It's like having a genius chef who can describe a recipe perfectly but burns the toast every time they try to use the toaster.
3. Getting stuck is normal, but fixing it works: Robots get confused easily. But if you let them pause, realize they are stuck, and try a new plan, they succeed much more often.

3. What Should We Do Next?

The paper suggests that to make these robots truly reliable (like a human), we need to stop trying to make one giant AI do everything perfectly. Instead, we should:

• Separate the jobs: Have a specialized module for planning (the Architect) and a different, specialized tool for clicking and typing (the Builder).
• Teach them to admit confusion: Instead of guessing and clicking the wrong thing, the robot should be able to say, "I'm not sure where to click, let me ask for help or try a different approach."
• Stop just looking at the final score: We need to grade them on how they did the task, not just if they finished it. This helps us see if they failed because they were stupid, or just because they were clumsy.

In short: These AI web agents are like brilliant but clumsy interns. They can understand the assignment, but they trip over their own feet when trying to do the actual work. To fix them, we need to give them better instructions (structured planning) and better training on how to use a mouse and keyboard (perceptual grounding), rather than just hoping they get smarter at thinking.

Technical Summary

1. Problem Statement

Large Language Model (LLM) web agents have shown promise in automating complex, multi-step web navigation tasks. However, they currently fall significantly short of human reliability, particularly on realistic, long-horizon tasks involving dynamic websites.

The Core Gap: Existing evaluations rely primarily on coarse, end-to-end success metrics (i.e., did the agent finish the task?). While these metrics quantify overall performance, they fail to diagnose where and why failures occur. Failures could stem from:

• Incorrect task interpretation.
• Flawed high-level strategy.
• Poor grounding of plans into concrete UI actions.
• Inadequate recovery when the environment diverges from expectations.

The authors argue that without a granular analysis of the agent's internal processes, it is impossible to systematically improve these systems.

2. Methodology

The authors propose a Hierarchical Planning Framework to decompose web agent performance into three distinct layers, mirroring both automated planning theory and human reasoning.

A. The Three-Layer Framework

1. High-Level Planning: The LLM decomposes a natural language instruction into a sequence of abstract subgoals (e.g., "filter by price," "sort results").
2. Low-Level Execution: The LLM translates each subgoal into a sequence of executable, concrete actions (e.g., clicking specific DOM elements, typing, scrolling).
3. Replanning: When the environment state diverges from expectations (e.g., a page fails to load or a search yields no results), the agent must detect the failure and revise its plan (either locally adjusting or globally replanning).

B. Experimental Setup

• Benchmark: The authors extended the Mind2Web-Live benchmark (104 live web tasks) by adding human-aligned high-level plans. These were derived from expert-annotated "key nodes" (essential intermediate subgoals) and converted into coherent, high-level steps.
• Models Evaluated:
  • gpt-5-nano (Primary model for all stages).
  • claude-haiku-4.5 and gemini-flash-2.5 (Evaluated for generalizability).
• Representations Compared:
  • Natural Language (NL): Free-form text subgoals.
  • PDDL (Planning Domain Definition Language): Structured, symbolic representations enforcing explicit preconditions and effects.
• Evaluation Mechanism:
  • LLM-as-Judge: Used for postcondition checking (verifying if a subgoal was met) and final task evaluation.
  • Alignment Metrics: Developed specific metrics to compare LLM-generated plans against human-annotated gold standards (e.g., Perfect Match, Partial Rate, Missing Rate, Decomposed Rate, Unmatched Rate).

3. Key Contributions

1. Hierarchical Evaluation Framework: A principled method to disentangle planning, execution, and recovery failures, moving beyond simple pass/fail metrics.
2. Structured Planning (PDDL) Analysis: The first work to incorporate formal planning representations (PDDL) into web agents, demonstrating that structured constraints improve plan quality.
3. Diagnostic Insights: Identification of low-level execution (perceptual grounding) as the primary bottleneck, rather than high-level reasoning.
4. Replanning Efficacy: Empirical evidence that a single round of replanning significantly boosts task success, even if the revised plan is less aligned with human abstractions.

4. Key Results & Findings

A. High-Level Planning

• PDDL vs. Natural Language: Structured PDDL plans are significantly more concise and goal-directed than NL plans.
  • Alignment: PDDL achieved a 67.7% perfect match rate with human plans vs. 60.6% for NL.
  • Efficiency: PDDL reduced "decomposed" (overly granular) and "unmatched" (redundant) steps.
  • Executability: While human plans were best, PDDL plans performed nearly as well as humans when executed, whereas NL plans struggled more.
• Failure Mode: LLMs tend to over-specify plans (breaking one human step into many LLM steps) rather than omitting steps.

B. Low-Level Execution (The Dominant Bottleneck)

Even when provided with accurate human-authored high-level plans, LLMs struggle to execute them:

• Success Rates: With human plans, the Plan Completion Rate was only 38.5%, and the final Task Success Rate was 36.4%.
• Primary Failure Modes:
  • Hallucinated Links: 32% of goto actions led to non-existent links.
  • Redundant Actions: 34% of actions produced no state change in the DOM.
  • Repetitions: Agents frequently repeated failed actions (10.4% of failures) despite environmental feedback.
  • Out-of-Domain Navigation: 16.7% of actions occurred outside the required website domain.
• Model Differences:
  • gpt-5-nano: Best overall performance.
  • gemini-flash-2.5: Generated compact plans but had the worst execution (highest redundancy, lowest success).
  • claude-haiku-4.5: Least reliant on navigation tools but suffered from high repetition rates.

C. Replanning

• Impact: One round of replanning significantly improved performance.
  • Subgoal Completion: Increased from ~63% to 84% (NL) and 70% to 93% (PDDL).
  • Task Success: Increased from ~32% to 41% (NL) and 36% to 45% (PDDL).
• Trade-off: Replanning often reduced alignment with the original human plan (as agents adapted to reality), but this adaptation led to higher actual success rates.

5. Significance and Recommendations

The paper fundamentally shifts the focus of web agent research from "better reasoning" to "better grounding."

• Separation of Concerns: Planning and execution are distinct skills. High-level planning benefits from structured representations (PDDL), while low-level execution requires robust perceptual grounding and state tracking.
• Action Space Design: Current action spaces force agents to make concrete UI decisions even under uncertainty. The authors suggest designing action spaces that allow agents to explicitly signal uncertainty or request clarification rather than hallucinating actions.
• Standardization: The authors advocate for process-based evaluation (analyzing planning, execution, and replanning separately) as the new standard for diagnosing agent failures, rather than relying solely on end-to-end success rates.

Conclusion: To achieve human-level reliability, future web agents must prioritize perceptual grounding and adaptive control mechanisms over purely improving high-level reasoning capabilities. The proposed hierarchical framework provides the necessary diagnostic tools to guide this evolution.