Early Diagnosis of Wasted Computation in Multi-Agent LLM Systems via Failure-Aware Observability

Technical Summary: Early Diagnosis of Wasted Computation in Multi-Agent LLM Systems via Failure-Aware Observability

Problem Statement
Tool-using multi-agent Large Language Model (LLM) systems decompose tasks across specialized agents, retrieval mechanisms, and code execution. While these systems can improve task coverage, they often generate long, complex failure trajectories that are difficult to interpret when relying solely on final-answer accuracy. A failed trace may contain redundant searches, unproductive retrieval, execution errors, or weak evidence, yet standard benchmarks compress these multi-step processes into a single binary outcome. This paper treats "wasted computation"—defined as tokens, tool calls, execution attempts, and latency spent on runs that do not produce a usable, grounded final answer—as an observability problem. The authors argue that multi-agent systems require a failure-aware observability layer in addition to final-answer evaluation to diagnose where and why computation is spent productively or wastefully.

Methodology
The study instantiates a failure-aware observability framework in a three-agent question-answering system consisting of an orchestrator, a search component, and an execution component. The system records structured event traces independently of final correctness, capturing online signals such as budget updates, repeated action signatures, information-change proxies, loop indicators, and evidence availability.

The framework maps specific failure modes to structured metrics at two levels:

1. Online Metrics: Emitted during execution to detect issues like tool instability, orchestration loops (e.g., repeated action keys or ABAB cycles), low information gain, and evidence gaps.
2. Offline Metrics: Computed from the completed trace to assess reliability, grounding, and semantic support. This includes a sentence-level support proxy using a lightweight semantic encoder (LSA-style representation) to measure the alignment between answer sentences and evidence chunks.

The framework was evaluated on the complete GAIA 2023 validation snapshot (165 tasks across three difficulty levels) under identical execution caps (8 orchestrator steps, 8 tool calls, 25,000-token soft ceiling). A "usable final result" is defined operationally as an orchestrator final tool result with status "ok," distinct from benchmark correctness. To validate the observability signals, the authors conducted two offline audits: a 10-trace grounding audit using an LLM judge to verify evidence support, and a 30-trace correctness audit against GAIA reference answers.

Key Contributions
The paper presents three primary contributions:

• A Failure Taxonomy: A classification for tool-using multi-agent LLM systems distinguishing between execution failure, orchestration loops, evidence failure, no-final termination, and cost waste.
• A Failure-Aware Observability Framework: A system that maps each failure mode to structured online metrics and offline analysis signals, allowing for the diagnosis of wasted computation before the final outcome is known.
• Empirical Analysis: A level-stratified analysis of 165 GAIA traces demonstrating how early warnings, grounding signals, and budget proxies relate to failed computation.

Results
Operational failures were common across all difficulty levels: 22 of 53 level-1 runs, 33 of 86 level-2 runs, and 12 of 26 level-3 runs failed to produce a usable final answer.

• Failure Mechanisms: Traces revealed distinct mechanisms behind these outcomes, including insufficient evidence, repeated-action loops, max-step termination, tool-failure streaks, and execution calls that succeeded without useful output.
• Resource Consumption: Mean token use increased sharply with task difficulty, rising from 8,152 tokens at level 1 to 16,389 tokens at level 3.
• Evidence vs. Support: While evidence availability was high in levels 2 and 3, semantic support did not increase monotonically. A significant portion of runs (28.3% to 46.2% across levels) contained evidence without proper citations, highlighting a gap between evidence presence and answer support.
• Correlations: Descriptive correlations showed that triage labels and tool-error rates are positively associated with unusable outcomes, while sentence support and answer-evidence similarity are negatively associated with them. Conversely, in the LLM-judge audit, semantic metrics were positively associated with grounding support.

Significance and Claims
The paper positions failure-aware observability as a diagnostic layer situated between raw execution logs and final-answer accuracy. The authors claim that this approach allows for the explanation of why computation was spent productively or wastefully, rather than merely summarizing the endpoint.

The study emphasizes that final-answer accuracy is too coarse for agent engineering; a trace with no final result, a weakly grounded answer, and a correct answer requiring many redundant calls have different operational meanings. The proposed framework enables the identification of "candidate intervention points" (e.g., changing retrieval strategies or requiring stronger evidence) rather than serving as automatic stop conditions. The authors conclude that while cheap online signals can identify wasteful trajectories early and semantic metrics can deepen grounding analysis, the next step is to translate these diagnostics into controlled intervention policies to measure improvements in cost-adjusted task success. The paper explicitly notes limitations, including the use of a single agent configuration and the observational nature of the analysis, which does not yet prove that online interventions improve accuracy or cost.