💻 computer science

Reproduction Test Generation for Java SWE Issues

This paper addresses the lack of reproduction test generation tools for Java by introducing TDD-Bench-Java, the first benchmark for this task with 250 instances from open-source repositories, and e-Otter++, an adapted solution that demonstrates high performance on both this benchmark and a proprietary industry dataset.

Original authors: Toufique Ahmed, Jatin Ganhotra, Avraham Shinnar, Martin Hirzel

Published 2026-05-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Toufique Ahmed, Jatin Ganhotra, Avraham Shinnar, Martin Hirzel

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a software detective working for a massive company. A user reports a bug: "Hey, when I click this button, the app crashes!" Before you can fix the code, you need to prove the bug actually exists. You write a small, automated test script that tries to click that button. If the script crashes, you've confirmed the bug. Once you fix the code, you run the script again; if it now works perfectly, you know the fix is real.

This paper is about teaching an AI to write those specific "bug-hunting" scripts automatically, but with a twist: it's doing it for Java, a programming language used by huge companies, whereas previous AI tools mostly only worked well for Python.

Here is the breakdown of their work using some everyday analogies:

1. The Problem: The Missing "Bug Hunter"

In the world of software, writing these bug-hunting tests is tedious and often skipped. Recently, AI has gotten good at writing them for Python (a popular language for startups and data science). But Java is the "heavy machinery" of the corporate world (banks, airlines, big tech). The AI struggled with Java because it's more rigid and complex.

The authors say, "We need a better way to teach AI to hunt bugs in Java."

2. The New Map: TDD-Bench-Java

To train and test their AI, they needed a map. They created a new benchmark called TDD-Bench-Java.

The Analogy: Think of this as a giant "Gym for AI." It contains 250 real-world bug reports from famous open-source Java projects. Each "workout" consists of a bug description and the code before the fix. The AI's job is to write a test that fails on the broken code and passes on the fixed code.
Why it matters: Before this, there wasn't a standardized way to see if AI could actually do this for Java. This benchmark is the first of its kind.

3. The Solution: e-Otter++ (The Smart Detective)

They took an existing AI detective named e-Otter (which was great at Python) and gave it a Java makeover, calling the new version e-Otter++.

Here is how this AI detective solves a case, step-by-step:

Step 1: The Localizer (Finding the Crime Scene)
The AI looks at the bug report and the massive codebase. It has to guess where the problem is hiding. It's like a detective looking at a city map and a vague description of a crime to guess which specific building and room to investigate.
- Java Twist: In Java, you often have to build a whole new file for a test. The AI has to figure out exactly where to put this new file so it doesn't break the building's structure.
Step 2: The Contextualizer (Gathering Clues)
Once it knows the location, it gathers the right tools (imports) and sets up the scene (package names). It's like a detective making sure they have the right badge and the right floor plan before entering the room.
Step 3: The Initial Test Generator (Making the First Attempt)
The AI writes a draft test script. It's a rough sketch.
Step 4: The Refiner (The Feedback Loop)
This is the secret sauce. The AI runs its own test on the broken code.
- Scenario A: The test crashes, but for the wrong reason (e.g., it crashed because of a typo, not the bug).
- The Fix: The AI looks at the error message, realizes its mistake, rewrites the test, and tries again. It does this up to 10 times, learning from every failure, until it finds a test that crashes exactly because of the reported bug.
Step 5: Heterogeneous Prompting (Asking the Same Question 6 Ways)
To make sure they don't miss the solution, the AI rewrites the bug report in six different ways (simplifying it, removing confusing code, adding a "hint," etc.) and generates six different test candidates. It's like asking six different detectives to solve the same case using different angles.
Step 6: The Selector (Picking the Winner)
Finally, a "Judge" AI looks at all six candidates and picks the single best test to submit.

4. The Results: How Good is it?

On the Public Gym (TDD-Bench-Java): The AI succeeded about 44% to 46% of the time. This means it successfully wrote a test that caught the bug and confirmed the fix in nearly half the cases. This is considered a strong result for such a difficult task.
On the "Real World" (Proprietary Data): The authors also tested this on 150 bugs from their own private company (IBM).
- The Challenge: These bugs were harder. The descriptions were shorter, vaguer, and often involved creating brand-new files that didn't exist yet.
- The Result: Without help, the AI only succeeded 4% of the time.
- The Fix: When they gave the AI a "hint" (telling it the names of the new files it needed to create), the success rate jumped to 20%.

5. The Takeaway

The paper concludes that while AI is getting better at writing bug-hunting tests for Java, it still struggles with the messy, vague reality of corporate software compared to the cleaner data found in open-source projects.

In short: They built a new training ground (TDD-Bench-Java) and a smarter detective (e-Otter++) that can now hunt bugs in Java code. It works well on standard problems but still needs a little human help (hints) when the clues are vague or the code is brand new.

Technical Summary: Reproduction Test Generation for Java SWE Issues

Problem Statement

The paper addresses the critical gap in automated software engineering (SWE) for Java: the generation of reproduction tests. A reproduction test is defined as a test that fails on the current codebase (confirming the presence of an issue) and passes after the issue is fixed (confirming its absence). While recent advances have produced benchmarks and solutions for Python, Java remains underserved despite its dominance in enterprise software. Existing Java-focused tools either lack fault localization capabilities, rely on simplified problem variants, or suffer from dataset contamination and overfitting. Furthermore, enterprise Java bugs often differ in distribution and complexity from open-source Python bugs, necessitating specialized approaches that account for Java's static type system and object-oriented nature.

Methodology

The authors propose e-Otter++ for Java, an adaptation of a state-of-the-art Python reproduction test generator. The methodology is a multi-stage pipeline designed to handle repository-level complexity without requiring access to the fix code ( $c_{new}$ ) during generation.

1. Pipeline Architecture

The workflow consists of four primary components:

Localizer: Identifies "focal functions" (likely to be modified) and related regression tests. Unlike the Python version which separated these tasks, the Java version merges them into a single step to handle the higher frequency of new test files in Java projects. It uses a three-step LLM process: selecting relevant files by name, selecting functions by name, and finally filtering based on function bodies.
Contextualizer: Constructs the necessary scaffolding for a new test file, which is common in Java. It determines the correct package name, imports, and file path by querying the repository and selecting from existing options to minimize hallucination.
Initial Test Generator: Generates an initial test candidate ( $t_{ot}$ ) as a new class in a new file, prompted with the issue description and localized context.
Refiner: Iteratively improves the test using execution feedback. It executes the test on the old code ( $c_{old}$ ), analyzes the logs via an LLM-based critic, and rewrites the test if it fails for the wrong reason. This loop runs up to 10 times, accumulating changes to prevent repetitive errors.

2. Inference Scaling via Heterogeneous Prompting

To increase the diversity of test candidates, the system employs heterogeneous prompting rather than temperature-based sampling. The issue description is rewritten into six distinct variants (morphs):

Standard: Rewriting into a structured format.
Simple: Removing jargon and simplifying the text.
dropCode: Removing potentially misleading code snippets.
initTest: Incorporating a generated initial test into the prompt.
initPatch: Incorporating a suggested code patch into the prompt.
Default: Using the original issue description.

A Test Selector then evaluates the six resulting candidates (along with their execution logs) and selects the single best test for submission.

Key Contributions

TDD-Bench-Java: The first repository-level benchmark specifically for Java reproduction test generation. It comprises 250 instances sourced from popular open-source repositories (e.g., Trino, Jackson, RocketMQ), filtered to ensure "fail-to-pass" behavior. The dataset includes diverse build tools (Maven, Gradle) and project structures.
e-Otter++ for Java: A robust solution adapted from Python workflows, achieving high performance on the new benchmark. It is the first complete solution for repository-level Java reproduction test generation that includes fault localization.
Proprietary Industry Evaluation: The paper evaluates the system on a contamination-free proprietary dataset of 150 issues from an internal IBM Java project. This provides insights into the challenges of applying these models to real-world enterprise code, which often differs significantly from open-source data.

Results

Public Benchmark (TDD-Bench-Java)

Using the Claude-Sonnet-4.5 and GPT-5.2 models, e-Otter++ achieved:

43.6% fail-to-pass rate with Claude-Sonnet-4.5.
46.4% fail-to-pass rate with GPT-5.2.

Key Findings:

Refinement Impact: The refinement step (e-Otter) significantly improved performance over the initial generator (Otter), with relative improvements ranging from 26.2% to 64.3% depending on the model and morph.
Heterogeneous Prompting: Using six morphs increased the fail-to-pass rate at N=6 (at least one candidate succeeds) by over 32% compared to single-variant approaches.
Statistical Significance: McNemar's tests confirmed the improvements from Otter to e-Otter were statistically significant ( $p < 0.01$ ).

Proprietary Dataset Results

Performance on the internal IBM dataset was lower, highlighting the difficulty of enterprise code:

Default Performance: With default issue descriptions, the fail-to-pass rate was only 4% (6/150).
With Hints: When the system was provided with "hints" (new class names and function signatures derived from the code patch), the rate improved to 20% (30/150).
Data Characteristics: Proprietary issues were shorter, lacked code snippets, and involved significantly larger code patches (avg. 362 lines vs. 55 lines in public data). Furthermore, 84% of proprietary issues were feature requests rather than bug fixes, often introducing new files that models struggle to anticipate.

Significance and Claims

The paper claims to make a significant contribution to the field by:

Bridging the Language Gap: Providing the first dedicated benchmark and solution for Java, a language critical for enterprise software but previously neglected in reproduction test generation research.
Industry Relevance: Moving beyond open-source benchmarks to evaluate performance on proprietary data, revealing that while models perform well on curated public data, they face substantial challenges with the "noise" and complexity of real-world enterprise issues (e.g., short descriptions, new file generation).
Methodological Rigor: Demonstrating that inference scaling via heterogeneous prompting and execution-driven refinement are effective strategies for Java, even when the static type system introduces build-time errors that differ from Python's runtime errors.

The authors conclude that while current LLM-based approaches show promise, bringing reliable diagnosis and validation to Java software development requires further innovation, particularly in handling poorly specified issues and generating accurate context for new code structures.