💻 computer science

A Systematic Methodology for Evaluating Failure Independence in LLM-Generated Code

This paper introduces a systematic methodology to evaluate failure independence in LLM-generated code, revealing that while heterogeneous models offer some diversity, their implementations frequently share root causes and fail on the same test cases, thereby violating the independence assumption required for effective N-Version Programming.

Original authors: Rodrigo Pato Nogueira, Karthik Pattabiraman, Marco Vieira, João R. Campos

Published 2026-07-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Rodrigo Pato Nogueira, Karthik Pattabiraman, Marco Vieira, João R. Campos

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are building a critical bridge. To make sure it doesn't collapse, you decide to hire five different engineering teams to design the exact same support beam. The idea is that if Team A makes a mistake, Team B might get it right, and if Team C also gets it right, you can just go with the majority opinion. This is called N-Version Programming. It's a safety net that relies on one big assumption: that the teams are independent enough that they won't all make the exact same mistake.

For decades, we've known that human teams often fail this test. They read the same blueprints, use the same textbooks, and end up making the same errors. But now, we have Large Language Models (LLMs)—AI systems that can write code instantly and cheaply. The hope was that since these AIs are "different" from each other, they might act like a perfect team of independent engineers, failing in different ways so we could always trust the majority.

This paper asks a simple, crucial question: Is that hope real? Do these AI models actually fail independently, or do they all trip over the same invisible banana peel?

The Experiment: A "Code Olympics"

The researchers set up a massive experiment, like a coding Olympics.

The Athletes: They used 12 different AI models (some from big tech companies, some open-source, some very smart, some less so).
The Events: They gave them 224 different coding problems to solve.
The Rules: They asked the AIs to write solutions in 5 different programming languages (like Python, C++, Java) and tried 3 different ways of asking (simple prompts, step-by-step reasoning, or "be creative" prompts).
The Judges: They didn't just look at the code; they ran the code against thousands of test cases to see where it broke.

The Findings: The "Echo Chamber" Effect

Here is what they discovered, broken down into simple concepts:

1. The "Same Brain" Problem (Structural Diversity)

When the researchers looked at the code itself, they found that if you ask one specific AI model to write the same solution five times, it writes almost the exact same code every time. It's like asking a single person to write five different essays on the same topic; they will likely use the same vocabulary and sentence structure.

The Metaphor: If you ask five different people to draw a cat, they might draw different breeds. But if you ask the same person to draw a cat five times, they will draw the same cat five times.
The Result: To get different-looking code, you must use different AI models. Using the same model with different "prompts" (instructions) didn't help much.

2. The "Shared Blind Spot" (Behavioral Diversity)

This is the most important part. Even when the code looked different, the researchers checked where the code failed.

The Metaphor: Imagine five different drivers taking a test drive. Driver A and Driver B might drive different cars and take slightly different routes, but they both hit the same pothole at the exact same time.
The Result: The AIs were not independent. They frequently failed on the exact same test cases. Even when using 12 different models, they tended to trip over the same logical traps. If one AI failed a specific math problem, a different AI was very likely to fail that same problem too.

3. The Safety Net Has Holes (Reliability Gains)

The researchers tried to use the "Majority Vote" strategy (if 3 out of 5 AIs say the answer is X, we go with X).

The Result: It helped a little bit, but not nearly as much as the theory predicted.
- If the AIs were truly independent, combining five of them should have made the system almost perfect.
- In reality, the improvement was less than half of what was expected.
- The Hard Truth: No combination of multiple AIs was ever more reliable than just using the single best AI model on its own. The "safety net" had too many holes because everyone was falling through the same holes.

4. Why Did They Fail? (Fault Analysis)

The researchers dug deep into why the AIs failed. They found that even when the errors looked different on the surface, they often came from the same root cause.

The Metaphor: One AI might fail because it forgot to check if a number was negative, while another fails because it got confused by a large number. But both are actually failing because they don't understand the concept of "boundaries" in the same way. They share the same "reasoning weakness."
The Result: The AIs aren't just making random mistakes; they are making systematic mistakes based on how they were trained.

The Bottom Line

The paper concludes that current AI models are not independent enough to be used in a "safety net" system where you rely on a majority vote to catch errors.

Using different models helps a little: It's better to mix models than to use the same one five times.
Changing the language or the prompt doesn't help much: Asking the AI to "be creative" or writing in Python instead of Java didn't stop them from making the same mistakes.
The "Independence" Assumption is Broken: The idea that "different AIs will fail in different ways" is currently a myth. They tend to fail together.

In short: If you are building a critical system and thinking, "I'll just ask five different AIs and go with the majority," this paper warns you: Don't do it. They are likely to all get the same thing wrong, and you won't be any safer than if you had just asked the smartest single AI once.

Technical Summary: A Systematic Methodology for Evaluating Failure Independence in LLM-Generated Code

Problem Statement

N-Version Programming (NVP) is a fault-tolerance technique that improves software reliability by executing multiple independent implementations of the same specification and combining their outputs (e.g., via majority voting). Its efficacy relies on the assumption that independently developed versions fail independently. However, empirical studies have shown that human-developed versions often fail on the same inputs due to shared interpretations of ambiguous specifications and common design strategies, significantly reducing reliability gains. Furthermore, the high cost of developing and maintaining multiple human teams has limited NVP adoption.

Large Language Models (LLMs) offer a potential solution to the cost barrier by generating multiple implementations at scale. However, it remains unclear whether LLM-generated solutions satisfy the critical assumption of failure independence. Current research on LLM code generation focuses on functional correctness, syntactic diversity, and code quality, but lacks a methodology to determine if independently generated implementations fail independently. Without such a methodology, it is impossible to assess whether NVP is viable with LLMs or to quantify the reliability gains achievable through redundancy.

Methodology

The authors propose the first systematic methodology to assess failure independence in LLM-generated code. The framework evaluates implementations across three complementary dimensions:

Structural Diversity: Measures differences in the static structure of code (organization, control flow, syntax) using CodeBLEU similarity. This captures surface-level differences in how solutions are written.
Behavioral Diversity: Measures whether implementations fail on the same inputs more often than expected by chance. The authors adapt the hypergeometric distribution model (originally used by Knight and Leveson) to calculate a $Z$ -score ( $Z_{ij}$ ) for pairs of solutions. This score quantifies the deviation of observed co-failures from the expected overlap under the null hypothesis of independence, controlling for the overall failure rate of each solution.
Reliability Gains: Simulates N-version majority voting ( $N \in \{3, 5\}$ ) to quantify the practical reliability improvement. The study introduces a metric called redundancy effectiveness, defined as the fraction of the theoretical reliability gain (achievable under perfect independence) that is actually realized by the ensemble.

Experimental Setup:

Dataset: Derived from the PROBE benchmark (1,651 contest-style problems). The authors curated a subset of 224 problems by filtering out trivial or intractable tasks and augmenting test suites via fuzzing and multi-agent generation to ensure high coverage (96.7% line, 94.8% branch) and robustness against edge cases.
Systems: Evaluated 12 diverse LLMs (proprietary and open-source, varying sizes and architectures) across 5 programming languages (Python, C++, Java, C, Rust) and 3 prompting strategies (Baseline, Chain-of-Thought, Out-of-the-box).
Procedure: Generated solutions were filtered for basic correctness, evaluated against the augmented test suites, and subjected to structural/behavioral analysis. A manual fault analysis was conducted on a subset of partially correct implementations to identify root causes.

Key Contributions

A Novel Methodology: The paper introduces the first systematic framework for evaluating failure independence in LLM-generated code, moving beyond correctness metrics to directly measure failure correlation and redundancy effectiveness.
Comprehensive Empirical Study: The study applies this methodology to a large-scale dataset (224 problems, 12 models, 5 languages, 3 strategies), providing the most extensive evaluation of failure correlation in LLM code to date.
Public Dataset: The authors release a dataset containing prompts, generated code, unit tests, and metrics to enable reproducibility and future research.
Detailed Analysis: The study provides granular insights into structural vs. behavioral diversity, reliability gains, and fault-level similarities, distinguishing between surface-level code differences and underlying failure modes.

Results

Structural Diversity (RQ1)

Implementations from the same model are highly similar structurally (e.g., CodeBLEU scores of 0.76 for C++ within the same model vs. 0.55 across different models).
Different models produce more distinct solutions, but the diversity is largely concentrated at the surface level (n-grams, syntax). Deeper structural properties (AST, dataflow) show less variation.
Prompting strategies have a negligible impact on structural diversity compared to model choice.

Behavioral Diversity (RQ2)

High Correlation: Failures are strongly correlated. The average pairwise similarity score ( $Z_{ij}$ ) is 2.04, indicating a significant deviation from independence. 65.9% of implementation pairs show statistically significant positive deviation.
Model Impact: Model choice is the strongest factor. Solutions from the same model fail on the same inputs far more often than expected. While stronger models exhibit more distinct failure patterns relative to other models, they remain highly correlated internally.
Edge Cases: Correlation increases on edge-case inputs (similarity score rises to 3.39), suggesting that difficult conditions align failure modes even more strongly.
Language/Prompting: Language has a moderate effect (Python shows higher internal alignment), while prompting strategies have negligible impact.

Reliability Gains (RQ3)

Modest Gains: N-version majority voting yields only modest reliability improvements. Reliability increases from 0.88 (single version) to 0.90 ( $N=3$ ) and 0.91 ( $N=5$ ).
Low Effectiveness: Redundancy effectiveness is low (0.43 for $N=3$ , 0.44 for $N=5$ ), meaning ensembles capture less than half of the reliability gain that would be possible under independence.
Model Selection: Heterogeneous ensembles (different models) perform significantly better than homogeneous ones, but even the best heterogeneous combinations do not surpass the reliability of the single strongest model (GPT-OSS).
Diminishing Returns: Adding more versions yields diminishing returns, failing to approach the theoretical limits of independence.

Fault Analysis (RQ4)

Shared Root Causes: Manual inspection reveals that even when failure patterns appear different, they often stem from the same underlying faults (e.g., reliance on global heuristics, integer overflows, or suboptimal algorithmic complexity).
Model-Specific Patterns: Specific models consistently reproduce the same faulty reasoning patterns. Stronger models tend to avoid simple errors but still exhibit consistent internal error patterns on complex inputs.

Significance and Claims

The paper claims that the failure independence assumption underlying NVP does not hold for current LLMs. Despite the ability of LLMs to generate code at low cost, the resulting implementations share correlated failure modes that severely limit the benefits of redundancy.

Practical Implication: Simply generating multiple versions of code using the same model (or even different models) is insufficient to achieve the fault masking benefits of NVP. The reliability gains are modest, and no ensemble surpasses the best single model.
Methodological Value: The proposed methodology serves as a critical tool for systematically evaluating failure independence as models evolve. It highlights that structural diversity is a poor proxy for failure independence and that behavioral diversity and fault-level analysis are necessary to assess the true robustness of LLM-generated code.
Future Directions: The authors suggest that future work must focus on techniques to explicitly optimize for behavioral diversity and independence (e.g., diversity-oriented prompting or fine-tuning) rather than relying on ad-hoc model combinations.

The study concludes that while LLMs reduce the cost of NVP, the fundamental barrier of failure correlation remains, necessitating new strategies to ensure true diversity in generated solutions.