🤖 AI

Trace-Level Analysis of Information Contamination in Multi-Agent Systems

This paper investigates how uncertainty-induced information contamination in multi-agent workflows leads to diverse execution trajectories and output failures, proposing a trace-based measurement framework and a formal taxonomy to better detect, localize, and mitigate these risks across heterogeneous artifact reasoning tasks.

Original authors: Anna Mazhar, Huzaifa Suri, Sainyam Galhotra

Published 2026-05-01

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Anna Mazhar, Huzaifa Suri, Sainyam Galhotra

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: A Team of Specialists with a Broken Clipboard

Imagine you hire a team of specialists to solve a complex mystery. You have a Data Analyst who reads spreadsheets, a Fact Checker who verifies numbers, a Coder who runs calculations, and a Manager who decides who does what next. They work together in a strict sequence, passing notes and files back and forth.

This paper studies what happens when the initial notes the team receives are slightly corrupted. Maybe a spreadsheet cell is shifted by one column, a photo is slightly blurry, or a document has a weird watermark.

The researchers wanted to know: If the starting information is a little bit "wrong," does the whole team crash, do they get confused, or do they accidentally fix it themselves?

The Experiment: The "Controlled Mess"

Instead of waiting for real-world accidents, the researchers created a "controlled mess." They took 32 different complex tasks (like analyzing financial reports or reading medical charts) and intentionally introduced small errors into the files the agents were reading.

They ran every task twice:

The Clean Run: The team gets perfect files.
The "Contaminated" Run: The team gets files with specific, injected errors (like swapping two columns in a table or blurring an image).

They didn't just look at the final answer. They put a camera on the entire process, recording every single decision, tool used, and note passed between agents. This recording is called a "Trace."

The Big Surprise: The Answer vs. The Journey

The most shocking discovery is that the final answer and the journey to get there are completely disconnected.

The researchers found three main ways the team reacted to the bad data:

1. The "Silent Saboteur" (Silent Semantic Corruption)

What happens: The team follows the exact same script as the clean run. They talk to the same people, use the same tools, and take the same number of steps. But, because the starting data was slightly wrong (e.g., a number was off by one), the final answer is wrong.
The Analogy: Imagine a chef following a perfect recipe, chopping onions and boiling water exactly right. But because the bag of flour they started with had a typo on the label saying "500g" instead of "50g," the cake collapses. The chef did everything "correctly" according to the plan, but the result is a disaster.
Why it matters: Current safety checks often look for "chaos" (did the team stop? did they panic?). This type of error looks calm and orderly, so it slips right past the guards.

2. The "Detour with a Happy Ending" (Behavioral Detours with Recovery)

What happens: The team gets confused immediately. They try a tool, it fails, so the Manager says, "Okay, let's try a different tool!" They loop back, re-read the file, ask for a second opinion, and eventually figure out the right answer.
The Analogy: You are driving to a party. You take a wrong turn because the GPS was slightly off. Instead of giving up, you realize you're lost, pull over, check a map, take a scenic route, and still arrive at the party on time. You took a much longer, more expensive route (more gas, more time), but you got there.
Why it matters: This is the most common reaction (40% of the time). The system is "resilient," but it cost a lot of extra money (computing power) to fix the mistake.

3. The "Total Crash" (Combined Disruption)

What happens: The error is so bad that the team gets confused, tries to fix it, fails, and eventually gives up or produces a nonsensical answer.
The Analogy: The chef tries to bake the cake, but the oven is broken. They try to fix the oven, fail, and then try to bake it in a toaster. It explodes. The kitchen is a mess, and there is no cake.

Key Findings in Plain English

1. "Expensive" doesn't mean "Safe."
You might think, "If the team is working hard and taking a long time (high cost), they must be being careful and getting the right answer."
The paper says: No.

Low Cost = Dangerous: Sometimes the team works fast and cheap, but they are silently producing the wrong answer (The Silent Saboteur).
High Cost = Not a Guarantee: Sometimes the team works hard and takes a long detour, but they still fail to get the right answer.
The Lesson: You cannot judge if a system is working just by looking at how much it costs or how long it takes.

2. Different Files Break Differently.
The type of file matters:

Spreadsheets/Tables: When these get messed up, the team tends to get stuck in a loop, trying to re-calculate and re-check things over and over (Extended Execution).
Audio Files: When audio is garbled, the team usually just gives up immediately (Early Termination).
Images: If an image is blurry, the team might take a weird detour but sometimes still get the answer.

3. The "First Mistake" Tells a Story.
The researchers looked at when the team first realized something was wrong.

Early Divergence: If they get confused immediately, it means the initial reading of the file was broken.
Late Divergence: If they work fine for a while and then get confused, it means the error was subtle and only showed up when they started doing complex math or logic later in the process.

Why Current Safety Guards Fail

Most companies build safety nets that look like this: "If the system stops working, crashes, or takes too long, stop it."

This paper argues those safety nets are blind to the real danger.

They miss the Silent Saboteurs because the system looks calm and orderly.
They might punish the Detourers (who actually solved the problem) because the system took too long or used too many resources.

The Takeaway

When we build teams of AI agents to do complex work, we can't just check the final answer. We have to watch the movie of how they got there.

A calm, fast process might be a hidden failure.
A chaotic, expensive process might be a successful recovery.
To make these systems safe, we need new ways to watch the "trace" of their thinking, not just the final result.

The paper concludes that we need to design systems that can spot these "silent" errors and understand that "working hard" doesn't always mean "working right."

1. Problem Statement

Multi-agent systems (MAS) increasingly rely on heterogeneous external artifacts (PDFs, spreadsheets, slides, audio) to perform complex reasoning tasks. Information extracted from these sources is passed between specialized agents (e.g., extractors, analysts, validators). The core problem addressed is information contamination: errors introduced during the extraction or parsing of these artifacts (e.g., misaligned table columns, OCR noise, corrupted text) that propagate through the workflow.

Current evaluation practices focus primarily on endpoint accuracy (does the final answer match the ground truth?). The authors argue this is insufficient because:

Decoupling: Workflows can diverge structurally (change execution paths) yet recover the correct answer, or remain structurally stable while producing incorrect outputs.
Hidden Costs: Structural instability (rerouting, retries) incurs significant operational costs (tokens, time) even if the final answer is correct.
Guardrail Failure: Common verification mechanisms (schema checks, format validation) often fail to intercept "locally plausible but globally corrupting" data.

2. Methodology

The authors propose a trace-level analysis framework to quantify how uncertainty propagates through structured multi-agent workflows.

Experimental Setup

Benchmark: 32 tasks from the GAIA benchmark, selected for their reliance on file attachments (tabular, document, image, audio).
Architecture: A fixed multi-agent orchestration with specialized roles (Coordinator, Data Analyst, Fact Checker, Synthesizer, etc.) communicating via a shared workspace.
Models: Experiments conducted across three LLM backends: GPT-5-mini, LLaMA-3.1-70B, and Qwen3-235B.
Data: 614 paired runs (clean vs. perturbed) per task.

Perturbation Strategy

Instead of modifying raw files, the authors inject structured perturbations into the artifact-derived representations (e.g., the text span extracted by an OCR tool or the table parsed from a PDF).

Perturbation Types: Includes column swaps, unit changes, OCR noise, image blurring, encoding errors, and text redaction.
Control: All other variables (prompts, seeds, agent roles) are held constant to isolate the effect of the perturbation.

Measurement Framework

The core innovation is the use of Execution Traces ( $\tau$ ) rather than just final outputs.

Structural Signatures: Events (routing decisions, tool calls, memory reads) are abstracted into structural signatures to ignore lexical noise (e.g., different wordings) and focus on control flow.
Trace Divergence: Measured using Normalized Structural Edit Distance ( $d_{norm}$ ) between the clean trace and the perturbed trace.
First Divergence Point ( $t^\star$ ): Identifies the exact step where the perturbed execution first deviates from the baseline, helping to localize the failure mechanism.
Metrics:
- Outcome: Task success/failure.
- Cost: Token overhead (perturbed vs. clean).
- Control Flow: Rerouting, extended execution (loops), and early termination.

3. Key Contributions

Formal Taxonomy of Contamination Manifestations:
- Silent Semantic Corruption (15.3%): The execution trace remains structurally identical to the baseline ( $d_{norm} \approx 0$ ), but the final output is incorrect. The system "accepts" corrupted data without triggering error states.
- Behavioral Detours with Recovery (40.3%): The execution path diverges significantly (rerouting, retries, extra steps), but the system successfully compensates and produces the correct answer.
- Combined Disruption (39.9%): Both the execution path and the final output are corrupted.
Trace-Based Measurement Framework: A method to detect, localize, and quantify contamination using structural edit distance and first divergence points, moving beyond endpoint-only evaluation.
Empirical Evidence of Cost-Correctness Decoupling: Demonstrating that high operational cost does not guarantee correctness, and low cost does not guarantee safety.

4. Key Results & Findings

Decoupling of Structure and Outcome

Finding 1: Endpoint metrics are inadequate. 40.3% of runs with major structural divergence still recovered correct answers, while 15.3% of runs with stable traces failed silently.
Implication: Relying solely on answer accuracy hides internal instability; relying solely on trace divergence creates false positives for "failure" when the system is actually adapting.

Control-Flow Signatures

Rerouting (80.6% of divergent runs): The most common response to contamination. Agents detect inconsistencies and switch strategies.
Extended Execution (37.4%): Contamination triggers retry loops and multi-stage verification, leading to high token costs (median 1.5x–2.4x baseline).
Early Termination (25.3%): Common in audio modalities where transcription failure halts the entire pipeline.
Modality Specificity: Tabular data perturbations often trigger extended execution; audio perturbations favor early termination.

Temporal Localization

Early Divergence ( $t^\star < 0.1$ ): Indicates foundational extraction failures (e.g., parsing errors).
Late Divergence ( $t^\star > 0.3$ ): Indicates reasoning-stage sensitivity (e.g., arithmetic errors arising from subtle numeric corruption).

Operational Costs

Finding 4: Cost is a poor indicator of correctness.
- 76.2% of low-cost runs produced incorrect answers (Silent Corruption).
- Only 16.3% of high-cost runs (>2x tokens) produced correct answers.
Finding 5: High token overhead does not predict recovery. Some perturbations (e.g., watermarks) cause expensive detours that rarely succeed, while others (e.g., encoding errors) allow for moderate recovery.

Model Differences

LLaMA-3.1-70B exhibited earlier divergence (faster detection of errors) but lower recovery rates compared to GPT-5-mini and Qwen3-235B.
GPT-5-mini showed the highest rate of "Behavioral Detours with Recovery" (48.6%), suggesting a more robust adaptive strategy.

5. Significance and Implications

Critique of Current Guardrails: Standard guardrails (schema validation, retry budgets) fail because they assume contamination manifests as structural errors. Silent semantic corruption bypasses these checks entirely.
Design Recommendations:
- Targeted Verification: Verification strategies should be modality-specific (e.g., aggressive validation for audio, schema checks for tables).
- Cost-Aware Defense: Systems must recognize that high-cost runs are not necessarily "successful" adaptations, and low-cost runs are not necessarily "safe."
- Trace-Native Evaluation: Future benchmarks must evaluate execution traces, not just final answers, to capture the full spectrum of robustness.
Future Directions: The paper calls for provenance-based origin attribution (tracing back to the specific extraction error), learning contamination-resilient workflows (RL for robust routing), and adaptive verification budgets based on predicted contamination risk.

In summary, the paper establishes that information contamination in multi-agent systems is a complex, multi-dimensional phenomenon where structural stability and semantic correctness are often misaligned. It provides a rigorous framework to measure this misalignment, offering a path toward more robust, cost-effective, and verifiable agent systems.