Holistic Evaluation and Failure Diagnosis of AI Agents

Technical Summary: Holistic Evaluation and Failure Diagnosis of AI Agents

Problem Statement
As AI agents are increasingly deployed in production for complex, multi-step workflows, reliable evaluation has become critical. Current evaluation paradigms suffer from two primary limitations:

1. Outcome-only metrics: Benchmarks like GAIA and SWE-bench typically score agents based on final task success or answer correctness. While they indicate whether an agent failed, they do not explain why or where the failure occurred.
2. Process-level limitations: Existing granular approaches struggle to connect specific failure types to their precise locations within long, structured execution traces. Monolithic "LLM-as-a-judge" approaches scale poorly to long traces due to context window constraints and "lost-in-the-middle" biases, while multi-stage pipelines often localize errors better but fail to reliably categorize them.

Methodology
The authors propose a holistic agent evaluation framework that integrates complementary top-down and bottom-up perspectives to decompose analysis into independent, per-span assessments. This approach leverages the hierarchical trace structure provided by observability infrastructure (e.g., OpenTelemetry).

• Bottom-Up Evaluation (Span-Level):

  • Mechanism: Computes metrics at leaf spans (individual LLM calls and tool invocations) and propagates verdicts upward through the trace tree.
  • Metrics: Includes semantic assessments (e.g., Instruction Following, Reasoning Integrity, Tool Completeness) rated on a 1–5 scale with natural language rationales, as well as categorical labels (e.g., error detection) and numerical measurements (latency, token counts).
  • Aggregation: Uses an existential failure propagation policy (a parent span fails if any child fails), though configurable policies (threshold-based, type-filtered) are supported.
  • Output: Provides precise error localization, identifying the specific subset of leaf spans responsible for a trace-level failure, along with the specific metrics and evidence causing the failure.

• Top-Down Evaluation (Agent-Level):

  • Mechanism: Assesses the agent trace as a whole by analyzing relationships and outcomes across descendant spans.
  • Metrics: Evaluates behavioral patterns invisible at the individual span level, such as Plan Efficiency (deviation from stated plans), Tool Coverage (collective addressing of goals), Tool Abuse (redundant calls or lack of adaptation), and Completeness.
  • Output: Detects holistic inefficiencies, such as resource abuse or poor retrieval strategies, which require reasoning about the entire execution flow.

• Integration: The framework combines these views to provide both diagnostic precision (identifying exact spans) and holistic assessment (identifying systemic behavioral gaps).

Key Contributions

1. Holistic Framework: A trace-aware evaluation system that unifies top-down agent-level assessment with bottom-up span-level analysis for fine-grained failure diagnosis.
2. Span-Level Diagnosis Method: A technique that localizes errors, categorizes their causes using a fixed taxonomy, and generates natural language rationales grounded in span input-output pairs.
3. Comprehensive Benchmark Analysis: A broad evaluation on the TRAIL benchmark (covering GAIA and SWE-bench datasets) demonstrating state-of-the-art performance across all metrics.

Results
The framework was evaluated on the TRAIL benchmark, which contains 148 OpenTelemetry traces with 841 span-level errors across a 20+ category taxonomy. The evaluation focused on three metrics: Localization Accuracy, Weighted Category F1, and Joint Localization-Categorization Accuracy.

• Performance Gains: The framework achieved the best results across all metrics on both GAIA and SWE-bench.
  • Category F1: Relative gains of up to 38% over the strongest prior baselines.
  • Localization Accuracy: Relative gains of up to 3.5×.
  • Joint Accuracy: Relative gains of up to 12.5×.
• Scalability: Unlike monolithic baselines whose localization accuracy degrades sharply or fails entirely as trace length increases (particularly on the longer SWE-bench traces), the framework's accuracy remained steady.
• Methodology vs. Model Capability: A critical finding is that the bottleneck in evaluation is methodology, not model capability. When the same frontier model (GPT-5.4) was used as a monolithic judge over the full trace, its localization accuracy was significantly lower (e.g., 0.292 on GAIA vs. 0.823 within the framework; a 2.8× difference, widening to 12× on SWE-bench).
• Per-Category Dominance: The framework led in more error categories than any other evaluator, successfully detecting both behavioral failures (via top-down metrics) and localized implementation errors (via bottom-up metrics).

Significance and Claims
The paper claims that the proposed decomposition of evaluation into independent span-level assessments effectively solves the scalability and precision issues plaguing current agent evaluation.

• Overcoming Context Limits: By avoiding the need to process entire long traces in a single prompt, the framework sidesteps context-window overflow issues that exclude 22–29% of SWE-bench traces for monolithic baselines.
• Diagnostic Precision: The framework demonstrates that combining top-down behavioral analysis with bottom-up granular inspection is necessary for complete failure diagnosis; neither approach alone is sufficient.
• Bottleneck Identification: The results suggest that the primary barrier to accurate agent evaluation is not the lack of capable foundation models, but rather the evaluation methodology itself. The framework proves that structured, decomposed evaluation allows existing frontier models to achieve significantly higher diagnostic accuracy than when used as monolithic judges.

Limitations and Future Work
The authors acknowledge that the TRAIL benchmark annotations themselves contain systematic issues, including inconsistent error span localization and ambiguous category boundaries. Furthermore, the current aggregation of local errors into overall scores relies on predefined rules (e.g., existential failure propagation). Future work is identified as needing smarter aggregation strategies that incorporate severity weighting and outcome-relevance filtering to distinguish between minor span-level inconsistencies and material reasoning failures.