🤖 AI

DoVer: Intervention-Driven Auto Debugging for LLM Multi-Agent Systems

DoVer is an intervention-driven auto-debugging framework for LLM multi-agent systems that overcomes the limitations of log-only failure localization by actively verifying hypotheses through targeted interventions and measuring success via task recovery and progress rather than attribution accuracy.

Original authors: Ming Ma, Jue Zhang, Fangkai Yang, Yu Kang, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang

Published 2026-02-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ming Ma, Jue Zhang, Fangkai Yang, Yu Kang, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a team of digital assistants (AI agents) working together to solve a complex puzzle, like finding a specific historical fact on the internet or solving a difficult math problem. Sometimes, the team gets stuck or gives the wrong answer. This is called a "failure."

For a long time, when these teams failed, developers tried to fix them by reading the "diary" of what happened (the logs) and asking a smart AI to guess who made the mistake and when. It's like watching a movie of a car crash and trying to point at the exact second the driver swerved.

The paper argues that this "guessing game" is flawed for two main reasons:

The diary is confusing: Often, there isn't just one clear mistake. The team might try Strategy A, fail, then try Strategy B, and fail again. Pinpointing a single "bad guy" is often impossible because the whole process is messy.
Guessing isn't enough: Even if you guess the right person, you don't know if you're right until you actually try to fix it.

The Solution: DoVer (Do-then-Verify)

The authors introduce a new system called DoVer. Instead of just guessing who is wrong, DoVer says: "Let's try to fix it and see if it works."

Think of DoVer like a mechanic with a time machine who doesn't just look at the broken car; they actually go back in time, tweak the engine, and drive it again to see if it runs.

Here is how DoVer works, step-by-step, using a simple analogy:

1. Breaking the Story into Chapters (Trial Segmentation)

When the AI team fails, they often go through several "attempts" or "chapters." In the first chapter, they try to find the answer by scrolling through a website. In the second chapter, they realize that didn't work and try using a calendar tool instead.

DoVer's move: It cuts the long, messy story into these distinct chapters (trials) so it can analyze each attempt separately.

2. Making an Educated Guess (Hypothesis Generation)

For each chapter, DoVer asks a smart AI: "Based on this story, where did things go wrong?"

The Guess: "I think the problem was that the 'Web Browser' agent tried to click a button that didn't exist."

3. The "Do" Part: The Intervention

This is the magic step. Instead of just writing down the guess, DoVer actually changes the story. It goes back to that specific moment in the "diary" and edits the instruction.

The Edit: It tells the Web Browser agent: "Don't click that button. Instead, scroll to the bottom of the page."
The Analogy: Imagine you are directing a play. The actor says the wrong line. DoVer doesn't just write a note saying "You said the wrong line." DoVer stops the play, walks on stage, whispers the correct line to the actor, and says, "Now, say this instead."

4. The "Verify" Part: Re-Running the Play

After making the edit, DoVer lets the AI team continue from that exact point.

If the team solves the puzzle: The guess was correct! The intervention worked.
If the team still fails: The guess was wrong. Maybe the problem wasn't the button, but something else entirely.

What Did They Find?

The researchers tested this on two different AI team setups and several difficult datasets (like GAIA and AssistantBench, which are like hard exams for AI).

Turning Fails into Wins: DoVer managed to turn 18% to 28% of failed attempts into successful ones. In one specific math dataset, it fixed 49% of the failures.
Making Progress Even When It Doesn't Win: Even when it didn't solve the whole puzzle, DoVer often helped the team get "further" along the path (like reaching the next milestone) than they were before.
Testing the Guesses: DoVer proved that about 30% to 60% of the initial guesses about who was wrong were actually correct (or incorrect). This is huge because it means we can stop relying on uncertain human guesses and start using actual proof.

Why This Matters (According to the Paper)

The paper claims that intervention (actually fixing and re-running) is a much better way to debug AI teams than just attribution (guessing who is to blame).

Old Way: "I think Agent A messed up at Step 5." (Unproven guess).
DoVer Way: "I changed Agent A's instruction at Step 5. Now the team succeeded. Therefore, Agent A was the problem." (Proven fact).

The authors conclude that this "Do-then-Verify" approach makes AI systems more reliable because it focuses on results (did it work?) rather than just blame (who did it?). It also highlights that sometimes the problem isn't just a bad instruction, but a lack of capability in the tools the agents use (like a browser that can't scroll properly), which the system can now identify.

In short: DoVer stops AI debugging from being a game of "Who Dunnit?" and turns it into a game of "Let's Fix It and See."

Technical Summary: DoVer – Intervention-Driven Auto Debugging for LLM Multi-Agent Systems

Problem Statement

Large Language Model (LLM)–based multi-agent systems are increasingly deployed for complex tasks, yet debugging their failures remains a significant bottleneck. Unlike traditional software errors (e.g., crashes), these systems often execute without interruption but produce incorrect or unsatisfactory results due to long, branching interaction traces and inter-agent misalignment.

The prevailing approach to debugging these systems relies on log-based failure attribution, where an LLM analyzes execution logs to identify the specific agent and step responsible for a failure. The paper identifies two fundamental limitations in this paradigm:

Lack of Validation: Log-only debugging generates hypotheses without verification, leading to untested and potentially incorrect attributions.
Ill-Posed Attribution: In multi-agent settings, failures often stem from complex interactions where multiple distinct interventions can independently repair a task. Consequently, attributing failure to a single "decisive" step or agent is often ambiguous and lacks a unique ground truth.

The authors demonstrate through a reproduction study of the Who&When (WW) dataset that ground-truth annotations for failure steps are frequently uncertain (15–30% annotator disagreement) due to multiple trials within a single session, inter-agent coordination ambiguity, and cross-annotator alignment challenges. This uncertainty severely limits the accuracy of log-based attribution models.

Methodology: DoVer Framework

To address these limitations, the authors propose DoVer (Do-then-Verify), an intervention-driven debugging framework. Instead of relying solely on static log analysis, DoVer operationalizes debugging as a dynamic process of hypothesis generation followed by active verification through targeted interventions.

The DoVer pipeline consists of four stages:

Trial Segmentation:
The system segments a long execution log into distinct "trials" based on planning updates (re-planning steps). This decomposition acknowledges that modern agent systems often employ ReAct-style loops with multiple planning-execution cycles. Segmenting the log allows for independent, parallel interventions on specific causal chains rather than treating the entire session as a monolithic block.
Failure Attribution (Hypothesis Generation):
For each trial, the system generates a hypothesis ( $h_i$ ) identifying a suspected faulty agent and step, along with a rationale. This stage leverages existing log-based attribution methods but treats the output as a testable hypothesis rather than a definitive ground truth.
Intervention Generation:
The hypothesis is converted into a concrete, actionable intervention. The framework focuses on orchestrator-level interventions to maintain generality across different agent architectures. Interventions are categorized into:
- Modified Instructions: Editing the orchestrator's messages to sub-agents to clarify intent, correct arguments, or supply missing context.
- Plan Updates: Revising the orchestrator's high-level plan (e.g., reordering or decomposing steps) to route around the identified failure.
  The system avoids modifying sub-agent code or capabilities directly, focusing instead on the message-passing layer.
Intervention Execution & Verification:
The system replays the trajectory from the point of intervention, preserving all prior context. It then evaluates the outcome using two primary metrics:
- Trial Success Rate: Whether the intervention turns a failed trial into a successful one.
- Progress Made: A fine-grained metric measuring whether the intervention advances the system toward success (e.g., achieving more milestones) even if full success is not reached.
- Hypothesis Validation: The system classifies outcomes as Validated (success), Refuted (failure persists despite faithful execution), Partially Validated (progress made but no success), or Inconclusive (intervention not executed due to tool constraints).

Key Contributions

DoVer Framework: The introduction of an intervention-driven debugging pipeline that shifts the paradigm from static log analysis to active "do-then-verify" debugging.
Analysis of Ground-Truth Uncertainty: A detailed analysis demonstrating that log-based failure attribution is fundamentally limited by the ambiguity of ground-truth labels in multi-agent systems, particularly due to multiple trials and inter-agent misalignment.
Empirical Validation: Experimental evidence showing that DoVer not only recovers a significant portion of failed cases but also provides explicit validation or refutation of failure hypotheses, reducing reliance on noisy human annotations.

Experimental Results

The authors evaluated DoVer on two agent frameworks (Magentic-One and AutoGen2) across multiple datasets (GAIA, AssistantBench, and GSMPlus).

Failure Recovery:
- On Magentic-One with GAIA and AssistantBench datasets, DoVer flipped 18–28% of failed trials into successes.
- On the GSMPlus dataset using the AutoGen2 framework, DoVer achieved a 49% flip rate on failed cases.
Progress Measurement:
- Interventions resulted in quantifiable progress (measured by milestone achievement) in up to 16% of cases, even when full success was not achieved.
Hypothesis Validation:
- DoVer successfully validated or refuted 30–60% of failure hypotheses, depending on task complexity.
- Inconclusive cases (where the system failed to execute the intervention) highlighted specific sub-agent capability gaps (e.g., missing "scroll-to-bottom" tools), suggesting a path for targeted tool enhancement.
Model Agnosticism:
- Ablation studies showed that DoVer performs effectively with open-source models (Qwen3-8B, Qwen3-32B), narrowing the performance gap with proprietary models (GPT-4o) when using few-shot prompting.
Comparison:
- DoVer significantly outperformed self-improvement baselines (Self-Refine and CRITIC), which achieved 0% recovery on the same datasets, demonstrating the superiority of in-situ intervention over end-of-trace refinement.

Significance and Claims

The paper claims that DoVer represents a practical and scalable mechanism for improving the reliability of agentic systems. By shifting the focus from "attribution accuracy" (which is often ill-posed) to "outcome-oriented debugging" (measuring repair and progress), DoVer offers a more robust approach to debugging.

The authors emphasize that intervention serves as a practical tool for:

Reducing Annotation Burden: Explicit validation via execution reduces dependence on ambiguous human labels.
Identifying Systemic Bottlenecks: The "Inconclusive" cases serve as a diagnostic tool to surface specific sub-agent capability gaps (e.g., tool limitations) that require human or automated refinement.
Enabling Self-Improving Systems: The framework bridges failure analysis with practical repair, laying the groundwork for more robust, verifiable, and self-improving multi-agent systems.

The authors remain modest, noting that their results are evidence of feasibility within specific frameworks and task domains (web search and math) rather than universal guarantees, and that future work is needed to address long-running production workloads and safety-critical constraints.