Heterogeneous Prompting and Execution Feedback for SWE Issue Test Generation and Selection

Technical Summary: Heterogeneous Prompting and Execution Feedback for SWE Issue Test Generation and Selection

Problem Statement
Software Engineering (SWE) issues, which describe bugs or missing features, are often accompanied by informal descriptions lacking automated reproduction tests. The absence of such tests hinders the verification of issues and the acceptance of code patches. The specific challenge addressed in this paper is the automatic generation of "fail-to-pass" ($F \to P$) reproduction tests directly from issue descriptions and the existing (pre-patch) codebase ($c_{old}$), without access to the resolving code patch ($c_{new}$). A valid $F \to P$ test must fail when executed on $c_{old}$ (reproducing the issue) and pass when executed on $c_{new}$ (confirming the fix). This task is difficult because issue descriptions are informal, and the target code for the test to pass does not yet exist.

Methodology: e-Otter++
The authors introduce e-Otter++, a system that extends the prior Otter++ framework. The system operates through three primary components: a test generator, a code patch generator, and a test selector.

1. Test Generation with Heterogeneous Prompting:
   Instead of relying solely on increasing LLM decoding temperatures to generate diverse candidates (a common practice that often lacks diversity), e-Otter++ employs heterogeneous prompting. This involves two systematic variations:

   • Issue Description Morphs: An LLM rewrites the original issue description in five distinct ways before test generation: standard (formatting), simple (simplifying jargon), dropCode (removing potentially misleading code snippets), initTest (incorporating an initial test), and initPatch (incorporating an initial patch proposal).
   • Context Masks: The test generation prompt is modified to include or exclude specific localization contexts (e.g., planner outputs, test localization, patch localization), creating five distinct prompt variations.
     By combining these, the system generates a diverse pool of 10 candidate tests per issue.

2. Execution-Augmented Test Repair:
   Generated candidate tests are executed on $c_{old}$. While they are expected to fail, they may fail for incorrect reasons (e.g., syntax errors or unrelated exceptions) rather than reproducing the specific issue described. e-Otter++ introduces a critic mechanism:

   • An LLM analyzes the issue description and the execution log to determine if the test fails for the "right reason."
   • If the failure is incorrect, the critic identifies necessary context (e.g., specific buggy lines, relevant code snippets, or function signatures) and augments the repair prompt.
   • The system iteratively repairs the test (up to 10 attempts) using this enriched context until the test fails for the correct reason or the iteration limit is reached.

3. Test Selection via Surrogate Patches:
   Since $c_{new}$ is unavailable, e-Otter++ cannot directly verify if a test passes on the true fix. To solve this, the system uses Agentless to generate 40 candidate code patches (surrogates for $c_{new}$) and filters them down to the 5 most frequent ones.

   • The test selector executes the candidate tests on these surrogate patches.
   • Tests that are $F \to P$ on the surrogate patches are prioritized.
   • A rule-based selector classifies failures (assertion failures vs. errors) and uses code coverage (the proportion of lines added/removed by the patch that the test covers) as a tie-breaker to select the single final test.

Key Contributions
The paper makes three novel contributions:

1. Execution-Augmented Test Repair: A method to increase test correctness by using execution feedback to retrieve related code and augment the repair prompt, ensuring tests fail for the reasons specified in the issue.
2. Heterogeneous Prompting for Diversity: Increasing the diversity of test candidates through issue description morphs and context masks, rather than relying solely on high-temperature sampling.
3. Surrogate Patch-Based Selection: A technique for selecting the final test using execution feedback from imperfect, generated code patches as a proxy for the ground-truth fix.

Results
The system was evaluated on TDD-Bench Verified (449 samples) and SWT-bench Lite (276 samples), both derived from SWE-bench.

• Performance: e-Otter++ achieved a state-of-the-art Fail-to-Pass ($F \to P$) rate of 63.0% on TDD-Bench Verified and 52.5% on SWT-bench Lite (using the Claude-3.7-Sonnet model). This significantly outperforms the previous best published results (Otter++ at 37.0% and 29.0%, respectively).
• Component Efficacy:
  • Execution-augmented repair improved $F \to P$ rates by 11.1% to 74.3% across different configurations.
  • Heterogeneous prompting (morphs + masks) increased the $F \to P@N$ (rate of at least one successful test among $N$ candidates) by over 13% compared to using masks alone.
  • Using generated code patches as a pre-filter for test selection provided a significant performance boost (e.g., a 32.9% increase in $F \to P$ on TDD-Bench Verified).
• Generalizability: The approach was validated on SWE-rebench, a contamination-free benchmark, where it improved the $F \to P$ rate by a relative 75% over the base Otter system, demonstrating robustness against data contamination.
• Validation Utility: The generated tests were shown to effectively validate code patches from other SWE agents, significantly improving the precision of solutions proposed by top leaderboard systems.

Significance and Claims
The authors claim that e-Otter++ represents a substantial leap in the state-of-the-art for generating reproduction tests from SWE issues. The paper argues that the combination of heterogeneous prompting and execution feedback—specifically the novel use of surrogate patches for selection and context-augmented repair—overcomes the limitations of previous methods that struggled with test diversity and the lack of a target codebase for verification.

The work aims to cement the practice of Test-Driven Development (TDD) by automating the tedious creation of reproduction tests, thereby aiding both human engineers and automated SWE agents in issue resolution and long-term quality assurance. The authors remain modest regarding the scope, noting that the system currently handles single test files and is evaluated primarily on Python, but emphasize that the relative improvements suggest the techniques are effective even on non-contaminated, diverse datasets.