🤖 AI

Adversarial Test-Hardening for AI-Written Code: An Instrument Autopsy and a Pre-Registered Causal Estimate of the Critic Loop

This paper presents a pre-registered causal study of an adversarial test-hardening loop that uses mechanical oracles to validate AI-generated code, revealing that a previously reported statistical breakthrough was an instrument artifact while demonstrating that a same-lineage critic model significantly improves mutant killing rates compared to a cross-provider configuration, with findings that highlight how harness asymmetries and operational failures can distort cross-model evaluations.

Original authors: Jeff Otterson

Published 2026-07-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jeff Otterson

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where computers are learning to write their own homework. In the field of software engineering, this is a rapidly growing corner of science where Artificial Intelligence (AI) is asked to write code and then write the tests to check if that code works. Think of it like a student who not only writes an essay but also creates the answer key for the teacher. The problem is, these AI students are often too nice to themselves. They write tests that only check if the code runs without crashing (the "happy path") but fail to catch the sneaky bugs hidden inside. To measure how good a test suite really is, scientists use a trick called "mutation testing." Imagine taking a perfectly good essay and secretly swapping out a few words to create nonsense. If the teacher's answer key (the test) can spot the nonsense and mark it wrong, the test is good. If the test lets the nonsense slide, the test is weak. The big question researchers are asking is: Can we build a system where one AI writes the code, a second AI writes the tests, and a third AI acts as a strict critic to find the mistakes the second one missed? And if we use different AI companies for the critic, does that make the tests better?

This paper tells the story of a scientific experiment that tried to answer that question, but with a twist: the researchers accidentally discovered a massive glitch in their own measuring tape. They set up a "test-hardening loop" where an AI "Tester" writes code, and an AI "Critic" writes new tests specifically to kill the bugs the first round missed. They ran this loop using two different setups: one where the Critic was from the same company as the Tester, and another where the Critic was from a different company.

At first, the results looked like a huge victory for the "different company" Critic. The data suggested it was vastly superior, finding bugs the other one missed with a statistical certainty so high it looked like a miracle ( $p = 9.5 \times 10^{-66}$ ). But then, the researchers did something rare and brave: they tore their own experiment apart to perform an "instrument autopsy." They found that the "different company" Critic wasn't actually smarter. Instead, the "same company" Critic was getting its answers silently chopped off by a hidden limit in the computer system they were using. Because the "same company" model tended to write longer, more detailed answers, the system cut them off before they finished, making it look like the model failed. The "different company" model wrote shorter answers, so it never hit the limit and looked perfect.

Once the researchers fixed this glitch, the "miracle" disappeared. The "different company" Critic wasn't a superhero; it was just the only one that didn't get its homework cut in half. However, there was a second, deeper issue: the initial experiment (Experiment 1) had a design flaw where each setup generated its own fresh starting test suite from scratch. This meant the comparison wasn't just about the Critic's skill, but also about the random luck of the initial test draw, making it impossible to tell if the "different company" was truly better or just got a lucky start. To fix this, the researchers ran a second experiment (Experiment 2) where they froze the initial test suite and forced both setups to start from the exact same point.

The real, honest finding from this corrected design was that the loop itself is powerful: when the Critic is allowed to keep trying, it can kill about 78% of the bugs that the first round of testing missed. However, comparing two different AI companies is tricky because the tools running them can be unfair. The paper concludes that while using a strict, mechanical referee (mutation testing) is great, you still have to make sure the referee's arena is fair for everyone, or you might end up praising the wrong winner. The researchers also found that the "different company" setup was cheaper to run, but this wasn't just because it wrote shorter answers; the cost gap was largely driven by the "same company" setup suffering from repeated operational failures, such as its verbose answers hitting the system limits and getting rejected, which forced the system to spend more money on retries and failed attempts.

Technical Summary: Adversarial Test-Hardening for AI-Written Code

Problem Statement
Large Language Models (LLMs) are increasingly tasked with generating both production code and the test suites intended to verify it. However, standard coverage metrics record which lines executed, not whether the behavior was rigorously verified. LLMs tend to generate "happy path" tests with weak assertions, resulting in suites that are fully covered but lack fault-detection power. Furthermore, evaluating these generated tests often relies on "LLM-as-judge" protocols, which suffer from self-preference bias, verbosity bias, and the inability of a model to objectively judge its own or its lineage's output. While mutation testing offers a mechanical oracle to measure fault detection, previous adversarial loops (e.g., MuTAP, AdverTest) have not rigorously isolated the causal effect of a "Critic" model hardening a test suite, nor have they controlled for instrument asymmetries in cross-model comparisons.

Methodology
The study employs a two-role adversarial loop under a mechanical mutation-kill oracle (using mutmut and pytest).

Tester Role: An LLM generates an initial test suite (Round 0).
Critic Role: A second LLM is prompted with the specific "surviving mutants" (defects the initial suite missed) and generates new tests specifically designed to kill those mutants.
Mechanical Verdict: The loop repeats until no new mutants are killed. Crucially, the verdict is entirely mechanical: a generated test either kills a mutant under the test runner or it does not. No model judges another model's output.

The research is structured around two pre-registered experiments:

Experiment 1: A pipeline-level comparison across five Python subjects. It compared a one-shot generation baseline against an adversarial loop (same-lineage Critic) and a cross-lineage Critic (different provider). However, this design suffered from a confound: each arm independently resampled its own Round 0 test suite, meaning differences could be attributed to initial sampling variance rather than the loop or critic lineage.
Experiment 2: A causal, within-replicate design. To remove the initial-draw confound, the Round 0 suite was frozen and shared across all arms (no-critic, same-lineage loop, cross-lineage loop) for five replicates per subject. This isolated the incremental effect of the Critic continuation.

Key Findings and Results

H1 (Replication): The adversarial loop kills significantly more mutants than one-shot generation. In Experiment 1, the loop killed 105 mutants that one-shot generation missed and lost none. This finding held even after correcting for instrument defects.
H2 (Cross-Lineage Effect): The hypothesis that a cross-lineage Critic (different provider) kills more survivors than a same-lineage Critic was not supported in Experiment 1. The initial analysis showed a massive, statistically significant effect ( $p = 9.5 \times 10^{-66}$ ), but this was later identified as an instrument artifact.
The Instrument Autopsy: A critical finding of the paper is the discovery that the initial H2 result was manufactured by a lineage-correlated instrument failure. The same-provider model (Anthropic) had an output token cap that silently truncated verbose responses from the same-lineage Critic, causing those rounds to be rejected or credited with zero kills. The cross-provider model (OpenAI) did not have this specific cap active in the same way, leading to a false appearance of superiority. Once the instrument was corrected to detect truncation and fail loudly, the significant effect collapsed to a non-significant result ( $p = 0.0625$ ).
Experiment 2 Causal Estimate: Under the corrected, frozen-shared-round-0 design, the same-lineage Critic loop killed a mean of 78.3% of the survivors left by the frozen initial suite (95% cluster-bootstrap interval [0.592, 0.935]).
Cross-Configuration Pilot: The cross-provider configuration showed a positive pilot difference in kill rate (0.178 gap) at 5.5x lower cost. However, the paper notes this gap is heavily dominated by a single replicate where the same-provider configuration suffered an operational failure (truncation) that was now detected and scored, rather than laundered. The authors emphasize this is a comparison of specific configurations (model + provider API + harness), not an isolated "lineage" effect.
Cost Accounting: The study introduces cost-per-mutant-kill as a normalization metric. Experiment 2 showed the cross-provider arm was 6.4x cheaper per incremental kill, primarily due to lower output token volume and fewer rejected rounds, rather than per-token price differences.

Significance and Claims
The paper makes three primary contributions:

Causal Estimation: It provides a pre-registered, within-replicate causal estimate of the incremental effect of a guarded, add-only Critic loop over a frozen initial suite, isolating the loop's value from initial sampling variance.
Configuration Comparison: It offers a pilot comparison of cross-provider vs. same-provider critic configurations under a mechanical oracle, demonstrating that such comparisons are sensitive to the specific harness and API behaviors, not just model lineage.
Instrument Autopsy: The most durable contribution is the "receipted autopsy" of a provider-correlated instrument artifact. The paper demonstrates that even with model judgment removed from the verdict, asymmetries in the execution harness (e.g., output caps) can manufacture large, false cross-model differences.

Conclusion
The study concludes that cross-model comparisons inherit the asymmetries of the harness running them. Closing the "model-judgment" channel is necessary but insufficient for neutrality; the apparatus feeding the models must also be symmetric. The paper advocates for pre-registration to prevent post-hoc reinterpretation and fail-closed instrumentation (where the system halts and reports errors rather than producing plausible but wrong numbers) to catch silent corruptions. The authors release all protocols, receipts, and analysis code to ensure reproducibility, framing the "autopsy" of their own flawed instrument as a core part of the scientific contribution.

Technical Summary: Adversarial Test-Hardening for AI-Written Code

More like this