🤖 AI

Why Retrying Fails: Context Contamination in LLM Agent Pipelines

This paper introduces the Context-Contaminated Restart Model (CCRM) to formally quantify how failed LLM agent attempts contaminate subsequent retries by elevating error rates, deriving theoretical bounds on success probabilities and optimal pipeline depths that are empirically validated against real-world SWE-bench data.

Original authors: Zhanfu Yang

Published 2026-05-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhanfu Yang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Core Problem: The "Bad Memory" Effect

Imagine you are trying to solve a complex puzzle, like fixing a broken piece of code. You ask an AI assistant to help.

Attempt 1: The AI tries to fix it but makes a mistake.
Attempt 2: You tell the AI, "That didn't work, try again."

In a perfect world, the AI would forget the mistake and start fresh. But in reality, the AI remembers the mistake. The failed attempt is still sitting in its "conversation history" (its context window). Because the AI sees its own previous error, it gets confused or stuck in a loop, making it more likely to fail the second time than the first.

The authors call this "Context Contamination." It's like trying to clean a muddy floor, but every time you wipe it, you leave a little more mud behind, making the next wipe harder.

The Solution: A New Math Model (CCRM)

The researchers created a new mathematical model called the Context-Contaminated Restart Model (CCRM) to explain exactly how this happens and how to fix it. They treat the AI's retry process like a game with specific rules:

The Pipeline: The AI doesn't just "fix" the code in one go; it breaks the job into a chain of small steps (like a pipeline). If any step fails, the whole attempt fails.
The Contamination:
- Attempt 1 (Clean): The AI has a standard chance of failing (let's say 10%).
- Attempt 2+ (Contaminated): If Attempt 1 fails, the AI is now "contaminated." Its chance of failing jumps up (maybe to 30% or higher) because it's carrying the baggage of the previous mistake.

The Five Big Discoveries

The paper proves five main things about this "bad memory" effect:

1. The Exact Formula for Success
They figured out a precise math equation to predict the odds of the AI finally succeeding within a certain number of tries.

Analogy: It's like knowing exactly how many times you need to roll a die to get a "6," but knowing that every time you fail, the die becomes slightly "loaded" (weighted) against you.

2. The "Cascade" Overhead
Because of the contamination, you need many more attempts to succeed than you would if the AI had a clean memory.

Analogy: If you were climbing a ladder, a clean restart is like stepping back down and starting over. A contaminated restart is like trying to climb the same ladder, but every time you slip, the rungs get slippery and harder to grip. You end up slipping many more times than necessary.

3. The Sweet Spot for Task Size
The paper asks: "If we have a limited budget of 'tries' (computing power), how many steps should we break the task into?"

The Finding: There is a perfect middle ground. If the task is too long (too many steps), one slip ruins the whole thing. If it's too short, you waste time restarting too often. The math tells you the exact "sweet spot" length for the task to maximize success.

4. The Theoretical Limit
They proved that no matter how smart your strategy is, you cannot beat the limits set by this contamination. You simply cannot succeed with fewer attempts than their model predicts without clearing the memory.

5. The Power of "Clearing the Slate"
The most practical finding: If you wipe the AI's memory before it tries again, it works much better.

Analogy: If you are stuck in a traffic jam, trying to drive through the same jam again won't help. But if you take a different route (clearing the context), you avoid the traffic entirely. The math shows that clearing the context before a retry is always the best move.

Real-World Proof: The "SWE-bench" Test

The researchers tested their theory on real data from a benchmark called SWE-bench (where AI tries to fix software bugs).

The Old Way (IID Model): Previous theories assumed that if an AI failed once, it had the same chance of failing the next time as it did the first time (like flipping a fair coin).
The Reality: The old theories were way too optimistic. They predicted the AI would succeed about 98.6% of the time after three tries. In reality, it only succeeded 81.2% of the time.
The New Way (CCRM): The new model predicted the success rate with almost perfect accuracy (error less than 0.1%).

The Takeaway

When an AI fails a task and you ask it to "try again," it often fails again because it's haunted by its own previous mistakes.

Don't just retry blindly.
Do clear the context. The paper proves that resetting the AI's memory before a retry is the single most effective way to save computing power and get better results.

The authors also note that if you can train the AI to make fewer initial mistakes, the benefit is huge because it prevents the "contamination cascade" from ever starting.

Technical Summary: Why Retrying Fails: Context Contamination in LLM Agent Pipelines

1. Problem Statement

When Large Language Model (LLM) agents fail at multi-step, tool-augmented tasks, standard practice involves retrying the task. However, the failed attempt remains in the model's context window as history. This "context contamination" causes the agent to carry forward erroneous reasoning or incorrect tool outputs, effectively elevating the error rate for subsequent attempts.

While this phenomenon is widely observed in practice (e.g., agents repeatedly referencing bad endpoints learned from prior mistakes), there is no formal theoretical framework quantifying how this contamination scales with task depth, error rates, or computational budgets. Existing literature focuses on single-run reliability or empirical failure catalogues but fails to model the cross-attempt dynamics that define real-world retry behavior. Consequently, naive retry strategies often waste significant compute resources, and standard Independent and Identically Distributed (IID) assumptions grossly overestimate success probabilities in multi-attempt scenarios.

2. Methodology: The Context-Contaminated Restart Model (CCRM)

The authors introduce the Context-Contaminated Restart Model (CCRM) to formalize this dynamic. The model defines a pipeline of depth $T$ (a sequence of $T$ tool invocations) where the agent makes up to $K$ attempts within a total budget $B = KT$.

Core Mechanisms:

Contamination State ( $Z_k$ ): A binary state $Z_k \in \{0, 1\}$ representing the context condition of the $k$ -th attempt. $Z_1 = 0$ (clean).
Error Rates:
- $\epsilon_0$ : Base per-step error rate when the context is clean ( $Z_k=0$ ).
- $\epsilon_1$ : Elevated per-step error rate when the context is contaminated ( $Z_k=1$ ), where $\epsilon_1 > \epsilon_0$ .
Transition Dynamics: If an attempt $k$ fails ( $S_k=0$ ), the state flips to $Z_{k+1}=1$ , contaminating all subsequent attempts. If an attempt succeeds, the process terminates.
Success Probability: The probability of a single attempt succeeding is $p_i = (1-\epsilon_i)^T$ . Thus, $p_1 < p_0$ .

The model treats the sequence of attempts as a Markov chain where the transition to a "contaminated" state is absorbing for the purpose of the current task (until a success occurs, which ends the process, or the budget is exhausted).

3. Key Contributions and Theoretical Results

The paper derives five main theoretical results under the CCRM framework:

R1: Exact Success Formula (Theorem 1)

The authors derive a closed-form formula for the probability of succeeding within $K$ attempts:
$P(\text{succeed in } \le K) = p_0 + (1-p_0)[1 - (1-p_1)^{K-1}]$
This formula accounts for the first attempt having probability $p_0$ and all subsequent attempts (given a failure) having the degraded probability $p_1$ . This structure is isomorphic to a modified geometric distribution found in reliability engineering but applied here to cross-attempt contamination.

R2: Cascade Overhead and Phase Transition (Theorem 2)

The paper quantifies the "cascade overhead" ( $\Delta K$ ), the additional attempts required by CCRM compared to a clean-restart (IID) baseline to achieve a target success probability $1-\delta$ .

Universal Lower Bound: Contamination always incurs a positive overhead ( $\Delta K > 0$ ).
Phase Transition: As the contamination ratio $\epsilon_1/\epsilon_0$ increases (or pipeline depth $T$ increases), the system undergoes a phase transition. In the "super-critical" regime, the required attempts $\Delta K$ diverge to infinity, whereas the IID baseline remains bounded. This explains why deep pipelines with high contamination ratios become practically unsolvable with simple retries.

R3: Optimal Budget Allocation (Theorem 3)

For a fixed total budget $B$ , the paper identifies the optimal pipeline depth $T^*$ that maximizes the success probability. The closed-form solution is:
$T^* = \sqrt{B \cdot \frac{\log(1/(1-\epsilon_1))}{\log(1/(1-\epsilon_0))}}$
The result shows that stronger contamination (larger $\epsilon_1/\epsilon_0$ ) dictates a proportionally smaller optimal depth $T^*$ , effectively recommending shorter, shallower pipelines to minimize the risk of entering the contaminated state.

R4: Information-Theoretic Lower Bound (Theorem 4)

Using Le Cam's method and a two-hypothesis construction (distinguishing between an IID model and the CCRM), the authors prove that no policy can achieve a success probability of $1-\delta$ with fewer than $K_{CCRM} - O(1)$ attempts. This establishes that the derived formula is tight up to a constant factor.

R5: Clean-Restart Dominance (Theorem 5)

The paper proves that clearing the context window before a retry (resetting $Z_k=0$ ) strictly dominates the standard "contaminated" retry strategy. The improvement ratio is derived explicitly, showing that context clearing significantly reduces the number of attempts required.

4. Empirical Validation

The authors validate the CCRM model using both synthetic Monte Carlo experiments and real-world data from SWE-bench Verified.

Synthetic Validation: Experiments with 30,000 trials confirm that the theoretical success formula matches simulation data with an error $\le 0.003$ . The phase transition in cascade overhead is observed as predicted.
Real-World Data (SWE-bench):
- Using data from Verdent (SWE-bench Verified), where agents make consecutive attempts on the same issue, the authors fit the CCRM model.
- Result: The standard IID model overestimates the pass@3 rate by 17.4 percentage points (predicting 98.6% vs. the actual 81.2%).
- Fit: The CCRM model fits the data with an error of less than 0.001.
- Inference: The fitted parameters imply a cascade ratio of $\epsilon_1/\epsilon_0 = 7.1$ , indicating that a failed attempt increases the per-step error rate by a factor of 7.1.
- Comparison: In contrast, benchmarks using independent fresh-start retries (e.g., USEagent, OpenHands) show large gaps between IID predictions and actual performance, but the authors attribute this to correlated task difficulty rather than context contamination, a distinct phenomenon outside the CCRM scope.

5. Significance and Claims

The paper claims to provide the first formal framework for analyzing cross-attempt context contamination in LLM pipelines. Its significance lies in:

Correcting Misconceptions: Demonstrating that the IID assumption is fundamentally flawed for consecutive retries in tool-augmented agents, leading to severe overestimation of reliability.
Actionable Principles: Providing closed-form solutions for optimal budget allocation ( $T^*$ ) and proving that context clearing is strictly superior to retaining history.
Theoretical Rigor: Establishing information-theoretic lower bounds and identifying phase transitions where retry strategies become ineffective.
Practical Guidance: Suggesting that reducing the base error rate $\epsilon_0$ (via fine-tuning or better reasoning) has a superlinear benefit when the contamination ratio is high, as it simultaneously reduces the elevated error rate $\epsilon_1$ .

The authors maintain a modest scope, acknowledging limitations such as the binary nature of the contamination state, the assumption of per-step independence within an attempt, and the exclusion of correlated task difficulty (which affects independent retries). They position CCRM as a foundational step, noting that combining cross-attempt contamination with within-attempt Markov dynamics remains an open problem.