Technical Summary: Beyond Fail-to-Pass: Iterative Hardening of Co-Generated Bug Reproduction Tests and Fixes

Problem Statement

Automated Program Repair (APR) driven by Large Language Models (LLMs) has advanced significantly, yet repairing bugs directly from natural language reports remains fundamentally underconstrained. Bug reports are often incomplete, and many candidate patches appear superficially reasonable without actually fixing the root cause. While Bug Reproduction Tests (BRTs) are increasingly used to guide repair by converting issue descriptions into executable signals, the standard evaluation criterion for these tests—Fail-to-Pass (F→P)—is insufficient.

The F→P criterion requires a generated test to fail on the buggy code and pass on the "golden" (correct) fix. However, the authors demonstrate that F→P alone does not guarantee a test is effective for downstream repair. Specifically:

1. Lax Tests: Some F→P tests are "lax," meaning they detect the observed symptom but admit plausible-but-incorrect patches that do not address the root cause.
2. Misaligned Tests: Some tests reject the correct fix while accepting incorrect ones.
3. Test-Fix Coupling: In co-generation (where an agent generates both the test and fix simultaneously), errors in the test and fix are correlated. An agent may generate a weak (lax) test and a corresponding incorrect fix that passes the test, leading the agent to stop prematurely under the false assumption of success.

Methodology

The paper proposes a two-phase framework called COHARDEN to address these limitations. The methodology relies on a new evaluation metric and an iterative hardening loop.

1. Mutation Patch Evaluation (MPE)

To distinguish between effective and ineffective tests, the authors introduce Mutation Patch Evaluation (MPE). Instead of relying solely on the golden fix, MPE evaluates a generated test against a pool of plausible-but-incorrect mutation patches. These patches are generated by applying semantic operators (e.g., symptom suppression, incomplete fixes, input-specific shortcuts, behavior substitution) to the golden fix.

Based on how a test reacts to these mutations, it is classified into three categories:

• Rigorous: The test fails on the buggy code, passes on the golden fix, and rejects plausible-but-incorrect mutations.
• Lax: The test fails on the buggy code and passes on the golden fix, but accepts plausible-but-incorrect mutations (under-constraining the repair space).
• Misaligned: The test rejects the golden fix (conflicting with the correctness specification).

2. The COHARDEN Framework

COHARDEN is a co-generation loop designed to produce Rigorous tests and correct fixes by avoiding the pitfalls of standard co-generation. It operates in two phases:

• Phase 1: Lax-init (Test-First Generation):
  Before attempting any code fix, the agent generates a reproduction test ($t_0$) targeting the observable failure (e.g., exception, output discrepancy). The agent is explicitly forbidden from modifying source code at this stage. This decouples the initial test from any specific (potentially incorrect) fix hypothesis, reducing the risk of generating a "Misaligned" test that hard-codes a wrong solution.

• Phase 2: Iterative Hardening:
  Starting from $t_0$, the agent enters a loop to jointly refine the test ($t_k$) and fix ($c_k$). In each round:

  1. Mutate: The current fix $c_k$ is mutated to create a pool of adversarial patches ($C_k$).
  2. Evaluate: The current test $t_k$ and the previous round's test $t_{k-1}$ are run against $C_k$. This populates a Temporal Matrix (analogous to the MPE confusion matrix) to track how many mutations are accepted or rejected.
  3. Harden: The agent receives feedback on "surviving" mutants (patches that passed the test). The goal is to tighten the test so it rejects these mutants.

  Convergence Criterion: The loop continues until the test is "non-trivially changed" and the Laxity Rate (the fraction of mutations accepted by the new test but rejected by the reference test) drops below a threshold ($\tau = 0.2$). This ensures the test evolves from Lax to Rigorous without requiring access to the golden fix during the loop.

Key Contributions

1. Mutation Patch Evaluation (MPE): A finer-grained evaluation framework that partitions BRTs into Rigorous, Lax, and Misaligned categories. This exposes a quality dimension orthogonal to F→P, revealing that F→P is too coarse to predict repair utility.
2. Empirical Analysis of BRT Utility:
   • The authors show that the repair success gain from F→P-passing tests comes entirely from the Rigorous tier; Lax tests provide zero gain ($\Delta = 0.0$).
   • They identify that standard co-generation produces Lax tests where the test and fix errors are correlated (joint failure rate is 1.87$\times$ higher than independent prediction), causing agents to stop on incorrect solutions.
3. COHARDEN Framework: A two-phase co-generation loop that uses the Laxity signal as an in-loop convergence criterion. By generating the test first and iteratively hardening it against mutation pools, COHARDEN produces stronger reproduction tests and higher-quality fixes.

Experimental Results

The authors evaluated COHARDEN on the SWE-bench Verified $\cap$ SWT-bench Verified subset (433 instances) using OpenHands with GPT-5-mini as the backbone.

• Performance: COHARDEN achieved 69.4% Resolved (fixes passing golden tests) and 78.9% F→P (tests passing golden fixes).
• Comparison:
  • Outperformed the strongest fix-only baseline (SWE-Agent) by +9.6 percentage points in Resolved.
  • Outperformed the strongest co-generation baseline (InfCode) by +7.9 percentage points in Resolved.
  • Outperformed the vanilla co-generation baseline (OpenHands + COGEN) by +8.0 percentage points in Resolved.
• Generalization: Consistent gains were observed across different LLM backbones (GPT-5.4, Claude Opus 4.5) and on the SWE-bench Lite benchmark.
• Cost: The average cost per instance was $0.84, comparable to existing co-generation methods, with the majority of gains achieved within the first few hardening rounds.

Significance

The paper argues that the field of automated program repair must move beyond the Fail-to-Pass criterion. The authors demonstrate that a test's ability to pass a specific reference fix is not a proxy for its ability to constrain the search space toward the correct root cause. By formalizing the distinction between Rigorous and Lax tests and using mutation-based feedback to iteratively harden tests, COHARDEN bridges the gap between test generation and effective repair. The work suggests that future APR systems should treat test quality not as a binary pass/fail metric, but as a dynamic property that must be actively strengthened against plausible incorrect variations to ensure robust bug fixing.