💻 computer science

Foundation Models as Oracles for Refactoring Correctness Detection

This study demonstrates that foundation models can effectively serve as adaptable, zero-shot oracles for detecting refactoring bugs in Java programs, achieving high accuracy across diverse IDEs and refactoring types while providing explanatory insights to complement traditional static and dynamic analysis tools.

Original authors: Rohit Gheyi, Rian Melo, Jonhnanthan Oliveira, Marcio Ribeiro, Baldoino Fonseca

Published 2026-07-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Rohit Gheyi, Rian Melo, Jonhnanthan Oliveira, Marcio Ribeiro, Baldoino Fonseca

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a master carpenter who has spent years building beautiful, sturdy tables. You have a set of automated machines (like IDEs) that can help you move a table leg from one side to another or swap out a drawer. These machines are supposed to do the job perfectly without changing how the table functions.

However, sometimes these machines make a mistake. They might move a leg in a way that makes the table wobble, or they might cut a piece off that causes the whole thing to collapse. In the world of software, these mistakes are called refactoring bugs. They can be subtle (the table wobbles) or obvious (the table falls apart and won't even stand up).

For a long time, to catch these mistakes, developers had to write very specific, manual rules for every possible way a machine could mess up. It was like trying to write a rulebook for every single way a table could break. It was hard, slow, and the machines kept finding new ways to break things that the rulebook didn't cover.

The New Idea: The "Super-Inspector"

This paper asks a simple question: Can a "Super-Inspector" (a Foundation Model, which is a type of advanced AI) look at the "Before" and "After" of a table being moved and tell us if it's broken?

The researchers didn't teach the AI specific rules about tables. Instead, they just showed it examples of broken tables and asked, "Is this broken?" This is called zero-shot prompting—like handing a smart person a broken chair and asking, "Is this safe to sit on?" without giving them a manual first.

The Experiment

The researchers gathered 226 real-life examples of broken tables (software bugs) that had happened over the last decade in popular software tools like IntelliJ, Eclipse, and NetBeans. These bugs fell into two categories:

The "Collapse" (Compilation Errors): The code is so messed up it won't even run.
The "Wobble" (Behavioral Changes): The code runs, but it does something different than it should (like printing the wrong number or crashing when it shouldn't).

They asked several different "Super-Inspectors" (AI models) to look at these 226 cases and say, "Is this broken?"

What They Found

1. The AI is surprisingly good at spotting mistakes.

The best AI inspectors (like Gemini-3.1-Pro-Preview and GPT-5.4) got it right almost every time (around 94% to 99% accuracy).
Even a smaller, free-to-use AI (GPT-OSS-20B) did a pretty good job (around 80% accuracy).
The AI didn't just say "Yes/No." It could explain why the table was wobbly, which helps the human carpenter understand the problem.

2. Some mistakes are harder than others.

The AI was great at spotting the "Wobbles" (behavioral changes).
It was slightly worse at spotting the "Collapses" (compilation errors), especially when the error was very subtle or involved complex rules about how the table parts connect.
Interestingly, the AI sometimes got confused by the way the table was described. If you added a fake screw or changed the color of the wood (metamorphic testing) without actually changing the structure, the AI still spotted the break. This suggests it's looking at the structure, not just memorizing the picture.

3. The "Big Project" Problem.

When the researchers tried to use the AI on huge, real-world projects where they only showed it a "diff" (a list of what changed, without the whole picture), the AI often said, "I don't know."
This happened in about 40% of the cases. The AI realized that without seeing the whole workshop (the full codebase), it couldn't be sure if moving a part would break something hidden in another room.

The Verdict

The paper concludes that these AI "Super-Inspectors" are not perfect replacements for the old, strict rulebooks (traditional tools). The old tools are like a laser level: they are fast, cheap, and 100% accurate for things they can measure.

However, the AI is like a wise, experienced carpenter.

It can spot tricky problems that the laser level misses.
It can explain why something is wrong in plain English.
It can handle new types of wood (new programming languages) without needing a new manual written for them.

The Best Strategy: Use the fast, cheap laser level first. If that doesn't catch everything, or if the problem is weird, ask the AI "Super-Inspector" to take a second look. They work best as a team, not as rivals.

Important Note: The paper only tested this on Java code (a specific type of programming language). It does not claim this works for all languages or that it can replace human judgment entirely. The AI is a helpful assistant, not a magic wand that fixes everything automatically.

Technical Summary: Foundation Models as Oracles for Refactoring Correctness Detection

1. Problem Statement

Automated refactoring tools in Integrated Development Environments (IDEs) are essential for maintaining code quality, yet they remain imperfect. Despite decades of research, these tools can introduce unintended behavioral changes or compilation errors, undermining developer trust. Traditional detection mechanisms rely on handcrafted preconditions, static analysis, and dynamic testing (e.g., SafeRefactor). However, these approaches face significant limitations:

Adaptability: Defining sound and complete preconditions for the combinatorial explosion of language constructs and edge cases is challenging.
Maintenance: Implementing new analyses for emerging refactoring types or language features requires substantial engineering effort.
Coverage: Existing test suites are often insufficient to reveal subtle behavioral deviations, and static analyses may miss complex interactions.

The paper posits that foundation models (general-purpose models trained on broad data) may serve as complementary, oracle-like components to detect these bugs. Unlike rule-based systems, foundation models can reason across diverse scenarios without task-specific training, potentially identifying subtle semantic shifts that traditional tools miss.

2. Methodology

The authors conducted an empirical study evaluating the ability of foundation models to detect refactoring bugs in Java programs.

2.1 Dataset Construction

The study utilized a dataset of 226 real-world refactoring bugs collected over more than a decade from three major Java IDEs: IntelliJ-IDEA, Eclipse, and NetBeans.

Composition: The dataset includes 185 compilation errors (CE) and 41 behavioral changes (BC).
Scope: It spans 47 distinct refactoring types, with the most frequent being Move Method, Inline Method, and Pull Up Method.
Ground Truth: The authors reconstructed executable Java programs for each bug report. For BC instances, they generated specific JUnit test cases that pass on the original program but fail on the refactored version, providing executable evidence of the behavioral deviation.

2.2 Experimental Setup

Models Evaluated:
- Open-Weight: GPT-OSS-20B (local execution), Llama-3.2-3B, Phi-4-14B, Gemma-4-31B, and Qwen-3.6-35B.
- Proprietary: GPT-5.4, Gemini-3.1-Pro-Preview, and Claude-4.6-Sonnet.
Prompting Strategy: The study employed zero-shot prompting without task-specific fine-tuning. The models were asked to analyze the original code (code1) and the refactored code (code2) to determine if the transformation introduced a compilation error or a behavioral change.
Output Requirements: Models were required to return structured JSON containing a verdict (YES, NO - COMPILATION ERROR, or NO - BEHAVIOR CHANGE), a brief explanation, and, if applicable, a generated JUnit test to prove the behavioral difference.
Evaluation Metrics:
- Accuracy: Mean accuracy and pass@k (success within $k$ attempts).
- Stability: Accuracy spread, Total Agreement Rate (tar@k), and Consensus accuracy (cons@k).
- Robustness: Metamorphic testing was applied to assess stability under behavior-preserving perturbations (e.g., adding comments, unused fields, or imports) to gauge sensitivity to data contamination or memorization.

2.3 Large-Project Feasibility

A secondary feasibility study evaluated the models on 44 real-world refactoring diffs from large open-source projects using a diff-only prompt. In this setting, models lacked full project context and were allowed to return an UNKNOWN verdict if the diff lacked sufficient information.

3. Key Results

3.1 Detection Accuracy

Proprietary Models: Gemini-3.1-Pro-Preview achieved the highest overall accuracy (99.6%), correctly classifying 225 out of 226 instances. GPT-5.4 and Claude-4.6-Sonnet followed with 93.8% and 94.7% accuracy, respectively.
Open-Weight Models: Gemma-4-31B outperformed other open models with 96.5% accuracy, surpassing the first-run performance of GPT-5.4. GPT-OSS-20B achieved 80.5% accuracy.
Error Types:
- Compilation Errors (CE): Proprietary models generally excelled here. Gemini-3.1-Pro-Preview achieved 100% CE accuracy. GPT-OSS-20B struggled more with CE (77.8%), often misclassifying uncompilable code as behavior-preserving or behavior-changing.
- Behavioral Changes (BC): All top models performed well (92.7%–97.6%). A common failure mode was the inability to generate a valid JUnit test that successfully compiled and exposed the difference.

3.2 Stability and Robustness

Repeated Sampling: For GPT-OSS-20B, repeated sampling (up to 5 attempts) increased cumulative coverage from 80.5% to 92.9%. GPT-5.4 showed high stability, with only marginal gains from repeated attempts (93.8% to 97.3%).
Metamorphic Testing: Both GPT-OSS-20B and GPT-5.4 showed high stability under behavior-preserving perturbations (e.g., adding comments or unused variables), with accuracy changes of less than 1%. However, the authors note this indicates robustness to specific perturbations, not necessarily immunity to data contamination.
Temperature Sensitivity: GPT-OSS-20B showed varying sensitivity to temperature settings; lower temperatures (0.1–0.5) favored behavioral change detection, while higher temperatures (0.8) favored compilation error detection. GPT-5.4's behavioral change detection dropped significantly under deterministic decoding ( $T=0$ ).

3.3 Large-Project Feasibility

In the diff-only evaluation, 40.9% of cases were classified as UNKNOWN by GPT-5.4. The authors attribute this to insufficient context in the diff (e.g., missing references to moved methods or fields in other parts of the codebase), highlighting a limitation of using foundation models as standalone oracles for large-scale projects without full context.

3.4 Comparison with Traditional Baselines

SafeRefactor: Achieved 97.8% accuracy on the dataset but failed on 5 cases due to version incompatibilities (the tool relies on Java 8, while the dataset includes newer Java features).
Foundation Models: The strongest foundation model (Gemini-3.1-Pro-Preview) matched or slightly exceeded SafeRefactor's accuracy (99.6%) while handling newer Java features natively.
Trade-off: Traditional tools provide deterministic, executable evidence. Foundation models provide probabilistic assessments with natural language explanations but lack formal guarantees.

4. Key Contributions

Empirical Evaluation: The first comprehensive study evaluating foundation models as zero-shot oracles for detecting refactoring bugs in real-world IDE scenarios.
Dataset Curation: The extension and reconstruction of a dataset of 226 refactoring bugs into executable artifacts, including specific JUnit tests for behavioral changes.
Metamorphic Robustness Analysis: An evaluation of model stability under behavior-preserving source code perturbations to assess robustness against potential data leakage.
Cost and Latency Analysis: A comparative analysis of the monetary cost and execution time of local open-weight models versus proprietary API-based models, highlighting trade-offs for deployment.
Diff-Only Feasibility Study: An investigation into the limitations of using foundation models for refactoring validation in large projects where full context is unavailable.

5. Significance and Claims

The paper claims that foundation models are promising complementary tools for refactoring correctness detection, rather than replacements for traditional verification techniques.

Complementary Role: The authors argue that foundation models can flag suspicious transformations and provide explanatory rationales, aiding developer inspection. They are particularly useful in scenarios where traditional tools are incomplete, costly, or difficult to configure for new language features.
Adaptability: Unlike rule-based systems that require manual updates for new language constructs, foundation models demonstrated the ability to analyze refactorings involving newer Java features (e.g., var, switch expressions) without specific training.
Limitations: The paper modestly concludes that foundation models do not provide formal guarantees. Their effectiveness depends on prompt design, context availability, and decoding configurations. The study emphasizes that human oversight remains crucial, especially when model explanations are plausible but insufficiently grounded.
Future Direction: The authors suggest hybrid workflows where deterministic tools handle compilation checks and foundation models assist with semantic equivalence assessment and triage, particularly in AI-augmented IDEs.

The study does not claim that foundation models have solved the problem of refactoring correctness; rather, it provides evidence that they can serve as effective, high-accuracy oracles that complement existing engineering practices.