Technical Summary: EvoOtter

Problem Statement

The generation of Bug Reproduction Tests (BRTs) is a critical step in software engineering, serving to confirm the presence of a bug before a fix is applied. Unlike traditional test generation, which aims for passing tests, a BRT must be "fail-to-pass" (F2P): it must fail on the current buggy code ($c_{old}$) but pass on the future resolved code ($c_{new}$).

The primary challenge in BRT generation is the lack of reliable feedback. Since the resolved code ($c_{new}$) is unavailable during generation, it is difficult to determine if a candidate test fails for the correct reason or to select the best candidate among many. Prior approaches have relied on "inference scaling"—generating many candidates and using Large Language Models (LLMs) or execution feedback to select and improve them. However, these methods are often expensive due to high LLM call counts and suffer from unreliable feedback or test overfitting.

Methodology: EvoOtter

EvoOtter addresses these challenges by combining evolutionary programming with LLMs, specifically tailored to reduce costs and sharpen feedback signals. The workflow operates on a population of candidate tests and a population of code mutants.

1. Core Evolutionary Loop

The system maintains two populations:

• Predators: Candidate BRTs.
• Prey: Buggy variants of the code (mutants).

The evolutionary loop proceeds as follows:

1. Initialization: A base generator creates $n$ initial diverse tests (typically 8) using heterogeneous prompting to vary issue descriptions and context.
2. Mutant Generation: A rule-based generator creates $m$ code mutants by applying specific mutation operators (e.g., arithmetic swaps, constant replacements, force failures) to localized focal functions suspected of containing the bug. No LLM is used for this step.
3. Fitness Evaluation: Each test is executed against the current population of mutants. The fitness score is defined as the number of mutants "killed" by the test.
   • Novelty in Killing Definition: Since the original code is already buggy, a test killing a mutant is not defined simply by a test failure. Instead, the system analyzes execution logs. A mutant is considered "killed" if the test fails on the mutant for a different reason than it failed on the original code (indicated by a change in the log output).
4. Selection (Successive Halving): Tests are sorted by fitness, and the lower half are discarded. To maintain a strong feedback signal while keeping execution costs constant, the system doubles the number of mutants in the next generation (sampling from a pre-generated pool) while halving the test population. This process repeats until a single test remains.
5. Crossover (Batched Repair): The surviving tests are used as parents. A single LLM call is used to generate the entire next generation of offspring. The LLM is prompted to interchange, add, or remove statements between the parent tests and their execution logs to improve them. This "batched crossover" avoids the high cost of individual LLM calls per test.

2. Cost Control Mechanisms

• Successive Halving: Reduces the test population by half each generation while doubling the mutant population, keeping the total number of test executions per iteration constant ($2m$).
• Batched Crossover: Generates all offspring for a generation in a single LLM call rather than one call per candidate.
• Rule-Based Mutants: Uses deterministic code rewrite rules for mutants instead of LLMs, reserving LLMs strictly for test generation and evolution.
• Single-Prompting Option: While the default uses heterogeneous prompting for initial diversity, the system supports a "single-prompt" mode where one LLM call generates all initial candidates. This significantly reduces costs and allows the use of more powerful (but expensive) reasoning models like Claude-Opus-4.7.

Key Contributions

1. EvoOtter Workflow: The first BRT generation workflow based on evolutionary programming, integrating mutation testing and successive halving.
2. Mutation-Based Fitness Score: A novel fitness metric that counts the number of rule-based code mutants killed by a test, utilizing log differences to handle the "buggy original code" scenario.
3. Batched Crossover: A technique that leverages modern LLM capabilities to evolve an entire generation of tests in a single call, balancing performance with cost.

Experimental Results

The authors evaluated EvoOtter on three benchmarks: TDD-Bench-Verified, SWE-Rebench, and SWT-Bench-Verified.

• Performance: Using Claude-Sonnet-4.5 with heterogeneous prompting, EvoOtter achieved an F2P rate of 65.9% on TDD-Bench-Verified and 51.9% on SWE-Rebench, outperforming prior baselines that rely solely on selection or repair (e.g., e-Otter).
• State-of-the-Art: By utilizing single prompting with the stronger reasoning model Claude-Opus-4.7, EvoOtter achieved 75.3% on TDD-Bench-Verified and 66.3% on SWE-Rebench. On the SWT-Bench-Verified leaderboard, it reached 77.8%.
• Cost Efficiency: EvoOtter achieves these results at a fraction of the cost of prior inference-scaling approaches. The average cost per instance is approximately $0.17 (excluding initial generation) for the standard workflow, and $0.88 for the high-performance single-prompt variant, compared to significantly higher costs for methods requiring individual repair loops.
• Component Analysis:
  • Selection: Mutation-based selection outperformed LLM-based and random selection in later evolutionary stages, though LLMs performed well in the initial phase.
  • Crossover: Removing crossover caused a performance drop of over 10%, highlighting its critical role.
  • Diversity: Single prompting reduced diversity slightly but did not prevent the evolutionary operators from effectively selecting and improving tests.

Significance and Claims

The paper claims that EvoOtter demonstrates how to efficiently and effectively combine evolutionary programming with LLMs for software engineering tasks. Its primary significance lies in:

• Robustness: Providing more reliable feedback signals through mutation testing rather than relying solely on potentially noisy LLM judgments or execution on the buggy code alone.
• Efficiency: Drastically reducing the cost of BRT generation by minimizing LLM calls through batched operations and successive halving, making the approach scalable.
• Practicality: Achieving state-of-the-art results on verified benchmarks while maintaining a low inference cost, suggesting that evolutionary strategies can be a viable alternative to pure inference scaling for complex software engineering tasks.

The authors note that their findings are currently limited to Python repositories and that the design choices (like single-prompting) are tailored to the capabilities of recent frontier models.