🤖 AI

When Helping Hurts and How to Fix It: Multi-Agent Debate for Data Cleaning

This paper reveals that while multi-agent debate often degrades data generation due to critique-induced confusion, it significantly improves error detection, leading to a derived condition and a specific adversarial configuration with code-execution grounding that successfully leverages debate to outperform single-agent models on generative tasks.

Original authors: Chirag Parmar, Akshat Mehta, Henglin Wu, Jagadish Ramamurthy, Shweta Medhekar

Published 2026-06-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Chirag Parmar, Akshat Mehta, Henglin Wu, Jagadish Ramamurthy, Shweta Medhekar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, hardworking assistant (let's call them the Generator) whose job is to clean up a messy spreadsheet. They are great at their job, but sometimes they get confused or make up facts (hallucinations), like trying to fix a column that doesn't exist.

To help, you hire a Critic—another smart assistant whose only job is to double-check the first one's work before you hit "save." You might think, "Two heads are better than one! If they argue back and forth, the final result will be perfect."

This paper asks: Does this "two-person debate" actually help, or does it make things worse?

The answer is a surprising "It depends." In fact, the debate can actually hurt the work just as often as it helps, depending on the type of task.

Here is the breakdown of their findings using simple analogies:

1. The "Help vs. Hurt" Switch

The researchers tested this setup on over 6,000 different cleaning tasks. They found a strange flip-flop effect:

When it HURTS (The "Confused Chef" Scenario):
Imagine the Generator is a chef making a complex recipe. The Critic is a food critic who says, "I don't think you need salt," or "This ingredient doesn't exist."
- If the recipe is open-ended (like "make a new dish"), the Critic might just be guessing. The Chef, wanting to please the Critic, deletes the salt or changes the recipe based on bad advice.
- Result: The Chef ends up with a worse dish than if they had just cooked alone. The Critic's "bad guesses" confuse the Chef, causing them to throw away good ideas. The paper calls this Critique-Induced Confusion (CIC).
When it HELPS (The "Spot the Difference" Scenario):
Now imagine the Generator is playing a game of "Spot the Difference" between two pictures. The Critic's job is to point out the errors.
- Here, the answer is either "Yes, that's an error" or "No, it's fine." It's a fact.
- Result: The Critic can easily check the facts. If the Generator misses a spot, the Critic catches it. If the Generator points to a spot that isn't an error, the Critic says, "No, that's actually fine."
- Outcome: The debate works perfectly here, catching errors and removing false alarms.

2. The "Golden Rule" for Debates

The authors came up with a simple formula to decide when to use a debate team and when to let the worker fly solo. Think of it like a Rescue vs. Damage scale:

Use the Debate Team IF:
1. The Critic can easily check the facts (like looking at a map or running a code test).
2. If the Critic finds a mistake, it's easy to fix it.
3. The Generator isn't already perfect (there's room for improvement).
- Analogy: If the Critic is a detective with a magnifying glass and the Generator is a witness, the detective can help.
Skip the Debate Team IF:
1. The Critic has to guess or use "gut feeling" to check the work.
2. The Generator is already doing a great job.
- Analogy: If the Critic is just guessing and the Generator is already a master chef, the Critic will just mess up the chef's confidence and ruin the meal.

3. Why "Self-Checking" Doesn't Work

You might think, "Why not just have the Generator check its own work?"

The Paper's Finding: No. If the Generator checks its own work, it's like a student grading their own test. They tend to miss the same mistakes they made the first time.
The Fix: You need a separate person (the Critic) who looks at the work with fresh eyes. But, that Critic needs proof.

4. The Magic Fix: "Evidence-Gated" Debates

The paper found a way to make debates work even for the tricky "cooking" (generative) tasks.

The Problem: The Critic was making up reasons to change things.
The Solution: Force the Critic to show their work. The Critic must write a piece of code or point to a specific cell in the spreadsheet to prove why something is wrong.
The Result: When the Critic has to prove their point with evidence, and the Generator only listens to proven points, the debate suddenly becomes super effective. It's like a court case where the lawyer must show physical evidence, not just make speeches.

Summary

Debate is great for finding errors in data (like spotting a typo or a missing number) because facts are easy to check.
Debate is dangerous for creating new things (like writing a whole new cleaning plan) because the Critic might guess wrong, and the Generator will blindly follow the bad advice.
The Secret Sauce: If you must use a debate for creative tasks, the Critic must provide hard evidence (like code or data points) to back up their claims, or the whole process falls apart.

In short: Don't argue with your assistant unless they can show you the receipts.

Technical Summary: When Helping Hurts and How to Fix It: Multi-Agent Debate for Data Cleaning

Problem Statement

Data cleaning historically consumes 60–80% of a data engineer's time. While Large Language Models (LLMs) are increasingly deployed to automate this via workflow generation, anomaly detection, and pipeline orchestration, they remain prone to hallucination. A critical failure mode occurs when an LLM proposes cleaning non-existent columns or removing valid records, silently corrupting data.

Multi-agent debate, where an adversarial Critic challenges a Generator's proposals, has been proposed as a mitigation strategy. However, existing evaluations focus on constrained reasoning tasks (e.g., arithmetic, factual QA) with small answer spaces. Data cleaning is fundamentally different, requiring grounding in specific table schemas and producing executable operations. Prior studies have not determined whether debate helps or harms across the full spectrum of data cleaning subtasks, nor have they established a principled framework for predicting its effect. Without such a framework, practitioners face expensive trial-and-error, risking the deployment of debate on generative tasks where it may degrade output quality while increasing compute costs by 4–7×.

Methodology

The authors conducted a controlled experiment across three benchmarks, four model families (Claude 4 Sonnet, Gemini 3.1 Pro, Qwen3 235B, DeepSeek R1), and over 6,000 task-condition pairs.

Benchmarks:

AutoDCWorkflow: 142 tasks involving multi-step cleaning workflow generation (evaluated via Factual Consistency and cell-level accuracy).
MMTU: 200 questions spanning table understanding tasks (Error Detection, Data Imputation, Entity Matching, Schema Matching).
MaTElDa: 1,173 real-world tables with injected errors, evaluated for cell-level precision, recall, and F1.

Experimental Design:

Topology: A Generator-Critic architecture where the Generator proposes structured JSON outputs and an adversarial Critic challenges them.
Variables: The study manipulated adversarial intensity (6 Critic prompt variants), anonymity, task isolation, and output cardinality.
Grounding: A factorial experiment tested the impact of code-execution grounding (Critic runs Python/pandas queries in a sandbox) and evidence-gated generation (Generator only accepts feedback citing specific data evidence).
Controls: Comparisons included single-agent baselines, self-verification (single agent with code sandbox), and self-consistency (majority voting).

Key Contributions

1. The Debate Benefit Condition

The paper derives a formal condition predicting when debate improves output quality. Debate helps when the probability of rescuing a wrong output exceeds the probability of destroying a correct one. Formally, debate is beneficial when:
$\frac{p_c}{1 - p_c} \times p_r > \frac{p_g}{1 - p_g}$
Where:

$p_g$ : Generator's baseline accuracy.
$p_c$ : Critic's verification accuracy (ability to correctly judge an item).
$p_r$ : Repair probability (probability the Generator produces a correct fix given a correct critique).

The condition posits that debate helps when the Critic's verification odds (weighted by fixability) exceed the Generator's baseline accuracy odds.

2. Critique-Induced Confusion (CIC)

The authors identify Critique-Induced Confusion (CIC) as a structural failure mode in generative tasks. In these tasks, the Critic often generates hallucinated feedback that the compliant Generator accepts uncritically. This causes the Generator to retract valid operations and substitute hallucinated alternatives. The study confirms CIC is structural, not prompt-dependent, persisting across all 6 Critic prompt variants and 3 Generator variants.

3. The Fix: Grounding and Calibration

The paper demonstrates that CIC can be addressed through a specific architectural configuration:

Adversarial Separation: A separate Critic is essential; self-verification with identical tools fails because $p_c \approx p_g$ .
Code-Execution Grounding: The Critic must verify claims against executable evidence (e.g., running code against the data).
Evidence-Gated Generation: The Generator must only accept feedback that cites specific data evidence.
This combination yields the first debate configuration to significantly exceed single-agent performance on a generative task (+5.3pp Factual Consistency, $p<0.05$ ).

Key Results

Debate Reverses Sign

The effect of debate is not uniform; it reverses sign based on task type:

Detection Tasks (Helps): On MaTElDa (error detection), debate improved F1 by +27.4pp ( $d=1.0$ ). The Critic effectively pruned false positives.
Generation Tasks (Hurts): On AutoDCWorkflow (workflow generation), debate degraded performance across all models, with Factual Consistency dropping by −1.6 to −15.5pp.
- Mechanism: Debate acts as a variance amplifier. It increases the number of perfect-score outputs but also introduces complete failures (retracting valid operations).
- Scale: Token overhead increased 4–7×.

Task Structure is the Critical Moderator

The boundary between help and harm lies in per-item verifiability and answer-space constraint:

Constrained/Verifiable: Tasks where outputs can be independently checked against source data (e.g., "Is this cell an error?") benefit from debate.
Open-Ended/Dependent: Tasks requiring generation of new values or dependent chains of operations (e.g., "What value fills this cell?") suffer from CIC because a single hallucinated critique cascades through the workflow.

Factorial Experiment Findings

A 2x2 factorial design (Adversarial Separation $\times$ Tool Augmentation) revealed an epistatic interaction:

Tools Alone (Self-Verification): No improvement over single-agent ( $\Delta \approx -1.3$ pp).
Separation Alone (Text Debate): Significant degradation ( $\Delta \approx -7.0$ pp).
Separation + Tools + Evidence Gating: Significant improvement ( $\Delta \approx +5.3$ pp).
Neither component alone was sufficient; the combination was required to break the symmetry of self-verification and filter noise.

Cross-Domain Validation

The Debate Benefit Condition correctly predicted the sign of debate's effect for all nine task types studied and generalized with zero false positives across 19 published comparisons in seven domains (including code generation, math reasoning, and QA).

Significance and Claims

The paper claims to provide the first principled framework for predicting when multi-agent debate is beneficial in data cleaning and, by extension, other generative tasks.

Practical Decision Rule: Practitioners can use the Debate Benefit Condition to decide whether to invest in debate. If $p_g$ is high, $p_c$ is low, or $p_r$ is low (common in open-ended generation), single-agent approaches are more effective and 4–6× cheaper.
Structural Insight: The findings challenge the notion that debate is universally beneficial or that prompt engineering alone can fix debate failures. The failure mode (CIC) is structural to the interaction between non-verifiable tasks and adversarial pressure.
Architectural Path Forward: For generative tasks where debate is desired, the paper argues that adversarial separation combined with grounded verification and evidence-gated compliance is the necessary configuration to overcome CIC.

The authors conclude that multi-agent debate is not a universal improvement but a tool whose utility is strictly governed by the verifiability of the task and the relative capabilities of the Generator and Critic.