Effectiveness of LLM-based Software Diversity for Reliability Improvement -- an Empirical Study

Title: Effectiveness of LLM-based Software Diversity for Reliability Improvement – an Empirical Study

Problem Statement
Software diversity is a established strategy for reducing common-mode failures in fault-tolerant systems, notably through N-Version Programming (NVP). However, traditional diversity is costly, requiring multiple independent human implementations, validation, and maintenance. While Large Language Models (LLMs) offer a scalable, low-cost mechanism to generate diverse code implementations, it remains unclear whether LLM-generated programs exhibit the necessary failure diversity to improve system reliability. Specifically, it is unknown if LLMs introduce correlated failures similar to human developers or if they provide complementary failure profiles that can be leveraged for reliability gains.

Methodology
The authors conducted an empirical study comparing historical human-written programs against large pools of LLM-generated code across three specifications: 3n+1, Factors and Factorials, and Factovisors.

• Datasets:
  • Human Baseline: Programs sourced from the UVa Online Judge repository, developed prior to the ubiquity of LLMs to ensure a contamination-free baseline.
  • LLM Generation: Large pools of code generated using 14 modern LLMs (5 Commercial APIs and 9 Local open-weight models) across four programming languages (C, C++, Java, Pascal) and five temperature settings (0.5 to 1.5).
• Evaluation Framework:
  • Programs were compiled and executed in a sandboxed environment against exhaustive test suites.
  • Filtering: Programs that failed to compile, crashed, or failed all test cases (PFD=1) were excluded. Only valid programs (Perfect or Partial) were retained.
  • Metrics: Reliability improvement was measured using the reliability improvement ratio ($R$), defined as the average Probability of Failure on Demand (PFD) of individual programs divided by the average PFD of 1-out-of-2 pairs.
  • Configurations: The study analyzed:
    1. Homogeneous pairs: Drawn from the same population (e.g., human-human or LLM-LLM).
    2. Heterogeneous pairs: Drawn from different populations (e.g., different languages, or human-LLM).
    3. Cross-source: Specifically pairing human-written programs with LLM-generated programs.

Key Contributions

1. First Application of Classical Frameworks to LLM Code: This is the first study to apply the classical Eckhardt-Lee (EL) and Littlewood-Miller (LM) reliability analysis frameworks to large populations of LLM-generated code, comparing them directly with historical human-written baselines.
2. Systematic Characterization of LLM Diversity: The study characterizes LLM-generated diversity across controllable dimensions (model family, temperature, programming language) and quantifies their impact on reliability gains.
3. Analysis of Human-LLM Heterogeneity: It provides an initial empirical view of whether LLMs reproduce existing human failure patterns or offer complementary diversity when paired with human code.

Results

• Homogeneous Reliability: LLM-generated programs generally yield higher reliability improvements than human-written programs in the highly reliable spectrum (e.g., for 3n+1, LLMs achieved ~11,500x improvement at PFD $10^{-7}$ vs. ~1,900x for humans). However, like human programs, LLM programs are not failure-independent; their reliability curves remain bounded by the theoretical independence line, indicating inherent failure correlation.
• Language Diversity: Forcing LLMs to generate code in different languages significantly increases reliability gains. Heterogeneous LLM pairings (e.g., C++ and Java) often outperform human language diversity, pushing empirical measurements closer to the strict independence bound.
• Cross-Source (Human-LLM) Diversity:
  • Enforced Heterogeneity: The most significant reliability gains were achieved by enforcing a strict heterogeneous architecture (one human program paired with one LLM program). This configuration consistently outperformed both pure human and pure LLM baselines.
  • Negative Correlation: In specific cases (e.g., 3n+1 at certain thresholds), the enforced human-LLM pairing crossed the theoretical independence line, indicating negative failure correlation where the two sources actively compensate for each other's weaknesses.
  • Model Tier Differences: Pairing humans with Commercial LLMs generally provided robust reliability improvements. Pairing humans with smaller Local models sometimes yielded mathematically extreme improvement factors (e.g., >2.9 million), but the authors note this is often deceptive, driven by the high failure rate of the local model rather than true complementary diversity.
• Temperature Effects: Increasing generation temperature increased diversity within the LLM pool (improving homogeneous LLM pairings) but had no impact on heterogeneous cross-source reliability, suggesting temperature alters the distribution of AI errors but does not bridge the fundamental cognitive gap between AI and human reasoning.

Significance and Claims
The paper claims that LLMs provide a scalable, low-cost source of heterogeneous programs whose diversity can be leveraged for reliability improvement. However, the authors emphasize that LLMs do not eliminate the classical challenge of correlated software failures. Instead, they offer a new mechanism to induce diversity.

The study concludes that the most effective architectural strategy for reliability is not simply pooling human and AI outputs, but enforcing structural heterogeneity (e.g., pairing a human-written component with an LLM-generated component). This approach can yield reliability gains that surpass both pure human and pure AI baselines, and in some instances, achieve negative failure correlation. The findings suggest that LLMs can support multi-version software development by making the systematic generation and evaluation of heterogeneous programs practical.

Threats to Validity
The authors acknowledge that the human corpus consists of competitive programming solutions from a pre-LLM era, which may not fully represent modern "augmented" development. Additionally, the test suites are specific to competitive programming tasks, and the results are conditional on these specific difficulty landscapes. The study also notes that LLMs may have been exposed to similar specifications during training, though the specific human corpus used was not public in the analyzed form.