Technical Summary: Beyond the Leaderboard

Problem Statement

The rapid proliferation of Large Language Model (LLM) agents has been accompanied by a surge in benchmark evaluations (e.g., GAIA, WebArena, SWE-bench) that report steady, often dramatic, year-over-year improvements in agent capabilities. However, a closer examination of the underlying literature reveals a "mixed picture": while aggregate scores rise, recurring failure modes persist across unrelated evaluation efforts. These include agents fabricating tool arguments, plans that satisfy individual constraints but fail under joint constraints, performance degradation in long-horizon tasks before context windows are exhausted, and multi-agent systems where coordination overhead negates collaboration benefits. Furthermore, questions regarding measurement validity—such as test-set contamination and solution leakage—suggest that reported gains may partly reflect corrections of earlier measurement errors rather than genuine capability leaps. The core problem is the lack of a unified framework to synthesize these disparate failure modes, distinguish genuine progress from measurement artifacts, and understand the structural limitations of current agent architectures.

Methodology

This paper employs a narrative synthesis of 27 benchmark, taxonomy, and audit papers published between 2023 and 2026, covering 19 distinct benchmarks. The scope encompasses five overlapping areas: tool use, web navigation, real-world planning, multi-agent coordination, and agent safety/security.

The methodology involves:

1. Iterative Taxonomy Derivation: Rather than assuming a pre-defined structure, the authors extracted over sixty distinct error categories from the source literature (e.g., ToolScan's seven error types, MAST's fourteen failure modes). These were grouped into conceptually overlapping themes corresponding to stages of an agent's reasoning-to-action pipeline.
2. Cross-Validation: The resulting six-cluster taxonomy was validated by observing that independent annotation processes in different source papers converged on similar distinctions (e.g., tool invocation errors, context accumulation bottlenecks) despite unrelated expert panels.
3. Quantitative Consolidation: Reported quantitative results were compiled into a categorized comparative table, explicitly flagging ranges where multiple studies reported different figures for the same benchmark and distinguishing between peer-reviewed findings, pre-prints, and non-peer-reviewed industry reports.

Key Contributions

The paper makes four primary contributions:

1. A Unified Taxonomy of Limitations: It presents the first synthesis integrating evidence across tool use, planning, long-horizon reasoning, multi-agent coordination, safety, and measurement validity into a single, cross-cutting taxonomy of six failure clusters.
2. Explicit Documentation of Genuine Progress: Unlike prior surveys focusing solely on failures, this work explicitly delineates where agents have measurably and credibly improved (e.g., single-turn tool selection, short-horizon web navigation, narrowly scoped coding), distinguishing these from areas of stagnation.
3. Quantitative Comparative Synthesis: It consolidates reported quantitative findings into a structured table, clarifying the scale of recent progress versus the remaining gaps, and highlighting the impact of measurement corrections (e.g., downward revisions of SWE-bench success rates).
4. Identification of Structural Patterns: The paper identifies recurring structural patterns across failure clusters, including non-linear failure compounding, the non-composability of sub-skill competence, and the uneven returns of additional scaffolding.

Key Results and Findings

The Six Failure Clusters

The synthesized taxonomy identifies six distinct failure modes:

1. Tool Invocation and Parameter-Level Errors: Failures include hallucinated function names, incorrect argument types/values, and redundant API calls. Notably, structured output modes (dedicated function-calling) can trade flexibility for a different profile of errors, sometimes producing higher rates of incorrect call counts than free-text prompting.
2. Planning and Constraint-Satisfaction Failures: Agents struggle to satisfy multiple constraints jointly. For example, in TravelPlanner, GPT-4 achieved a 0.6% success rate on multi-constraint itineraries despite high rates of satisfying single constraints in isolation. This "compositionality gap" does not close simply with model scale.
3. Long-Horizon Degradation: Performance collapses non-linearly as task length increases. Agents often lose track of original objectives and enter repetitive loops, even when relevant information remains within the context window. This is attributed to a "no-recovery bottleneck" where early errors propagate irreversibly.
4. Multi-Agent Coordination Failures: Multi-agent systems frequently fail due to specification ambiguity, inter-agent misalignment, and verification breakdowns. Coordination overhead often offsets the benefits of task decomposition, with specification and misalignment issues accounting for the majority of observed failures.
5. Safety and Security Failures: Agents exhibit vulnerabilities to indirect prompt injection and underspecified instructions. Even "safe" agents fail in ~24% of test cases involving ambiguous instructions, often proceeding on unwarranted assumptions. Safety vulnerabilities appear largely orthogonal to capability metrics.
6. Measurement Validity Problems: Re-analysis of benchmarks like SWE-bench and GAIA reveals that solution leakage and weak test suites have inflated reported success rates. Correcting for these issues has led to significant downward revisions (e.g., SWE-bench success rates dropping from ~12.5% to ~4% in some configurations).

Genuine Progress

Despite these limitations, the paper identifies credible areas of improvement:

• Single-Turn Tool Selection: Frontier models show strong, steadily improving accuracy in single-turn function calling and parameterization.
• Short-Horizon Web Navigation: Success rates on filtered web tasks have risen from <50% to ~85-90% for some scaffolds, driven by architectural convergence (modular planner-executor-memory designs).
• Narrowly Scoped Coding: Corrected SWE-bench resolution rates for 2026 remain meaningfully higher than 2024 baselines, suggesting genuine progress in well-specified, single-repository tasks.
• Safety Fine-Tuning: Targeted safety fine-tuning has been shown to substantially reduce vulnerability to indirect prompt injection, demonstrating that specific interventions can mitigate specific failure modes.

Structural Patterns

The analysis reveals several cross-cutting patterns:

• Non-linear Compounding: Failure probability increases faster than linearly with task length due to the inability to recover from early errors.
• Non-composability: Competence in sub-skills (e.g., individual tool calls) does not reliably compose into end-to-end task success.
• Uneven Returns to Scaffolding: Adding more agents, tools, or reasoning effort does not uniformly improve reliability and can sometimes degrade performance (e.g., higher reasoning effort reduced accuracy in the majority of runs on the Holistic Agent Leaderboard).
• Partial Independence of Capability and Safety: High capability in reasoning and tool use does not guarantee safety; safety requires specific, targeted interventions.

Significance and Claims

The paper argues that interpreting agent progress requires a nuanced distinction between genuine capability gains and the correction of earlier measurement errors. It posits that the field is in a state of maturation, evidenced by the emergence of rigorous, cost-aware, and human-calibrated evaluation infrastructure (e.g., METR's time-horizon analysis, Holistic Agent Leaderboard) that actively corrects its own measurement practices.

The significance of this work lies in its provision of a balanced organizing frame for research and deployment. It challenges the narrative that agentic AI problems are "close to being solved" by highlighting that current systems remain brittle at the reasoning-action interface, particularly in multi-turn, long-horizon, and multi-agent settings. Conversely, it avoids a dismissive stance by acknowledging substantial, credible progress in narrow, well-specified domains. The authors conclude that future evaluation practices must treat compounding failure, non-composability, safety, and measurement validity as first-class concerns alongside capability metrics.