💻 computer science

A Differential Fuzzing-Based Evaluation of Functional Equivalence in LLM-Generated Code Refactorings

This paper introduces a differential fuzzing-based approach to evaluate functional equivalence in LLM-generated code refactorings without predefined test cases, revealing that current models frequently produce semantically divergent code and that existing test suites often fail to detect these critical errors.

Original authors: Simantika Bhattacharjee Dristi, Matthew B. Dwyer

Published 2026-02-18

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Simantika Bhattacharjee Dristi, Matthew B. Dwyer

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very talented, super-fast chef (the Large Language Model or LLM) who is great at cooking. You ask this chef to take an existing recipe for a chocolate cake and "refactor" it. In the world of coding, refactoring means rewriting the recipe to make it faster, simpler, or cleaner, but with one golden rule: The cake must taste exactly the same.

This paper is a report card on how well these AI chefs are actually following that rule.

The Problem: The "Taste Test" Trap

Usually, when we check if a chef did a good job, we ask them to bake the cake and then taste it using a pre-defined list of ingredients (like "Does it have chocolate?" "Is it sweet?"). This is like the existing test suites used in software.

The problem is, what if the chef secretly swapped the vanilla extract for a weird chemical that tastes almost like vanilla to a human, but makes the cake slightly toxic? If your taste test only checks for "sweetness" and "chocolate," you might miss the poison. The cake passes the test, but it's not the same cake anymore.

The Solution: The "Fuzzing" Machine

The authors of this paper decided to stop relying on the short, pre-written taste test. Instead, they built a Fuzzing Machine (a tool called Differential Fuzzing).

Think of this machine as a robot that can throw thousands of random, weird, and extreme ingredients at the cake batter.

"What if we use 100 eggs?"
"What if we bake it at 500 degrees?"
"What if we add a pinch of salt?"

The machine bakes the Original Cake and the AI-Refactored Cake side-by-side with all these crazy inputs. If the two cakes come out identical in every single scenario, the AI passes. If there is even one tiny difference (a crumb here, a different texture there), the AI fails.

What They Found (The Shocking Results)

The researchers tested six different AI chefs (like GPT-4o, CodeLlama, etc.) on three different sets of recipes. Here is what happened:

The Chefs are Messy: Even the best AI chefs changed the "flavor" of the code 19% to 35% of the time. That means nearly 1 out of every 3 times you ask an AI to clean up your code, it accidentally changes how the code actually works.
The Harder the Job, The Messier: When the recipes were simple (like a single function), the AI did okay. But when the recipes were complex (like a whole program), the AI messed up the flavor even more often.
The Old Taste Test Failed: This is the biggest surprise. The researchers took the "bad" cakes (the ones that tasted different) and ran them through the old, pre-written taste tests.
- Result: The old tests said, "Yum! Perfect!" about 21% of the time.
- Translation: The existing tests were blind to the changes. They told developers the code was safe, when in reality, it was broken.

The Big Takeaway

The paper concludes that we cannot trust two things blindly:

We can't fully trust the AI to rewrite our code without breaking it, even if it looks perfect.
We can't fully trust the existing tests to catch the AI's mistakes.

The Analogy:
Imagine you hire a contractor to renovate your house. They promise to make the kitchen faster and more modern without changing the plumbing.

The Old Way: You check the lights and the sink. They work! You pay them.
The New Way (This Paper): You turn on the water, turn off the water, flood the floor, and shake the pipes. You find out that while the sink works, the pipes are now leaking into the neighbor's basement.

The Lesson: Just because the code "passes the test" doesn't mean it's the same code. We need much stricter, more creative testing (like the Fuzzing Machine) to make sure our AI helpers aren't secretly changing the rules of the game.

1. Problem Statement

The rapid integration of Large Language Models (LLMs) into automated code refactoring pipelines has introduced a critical reliability challenge: ensuring functional equivalence between the original code and the LLM-generated refactored code.

The Gap: Prior evaluations of LLM refactoring capabilities have largely relied on test-passing metrics (e.g., pass@k or test pass ratios) using predefined test suites.
The Flaw: These test suites often cover only a limited portion of the input space. Consequently, a refactored code snippet may pass all existing tests while still altering the program's underlying semantics (functional divergence).
The Need: There is a lack of robust evaluation methods that can detect semantic changes without relying on potentially incomplete, pre-existing test cases.

2. Methodology

The authors propose a novel evaluation framework called Eq@DFuzz, which utilizes differential fuzzing to verify functional equivalence.

A. Experimental Setup

Models Evaluated: Six widely used LLMs were tested:
- Open-source: CodeLlama (13B), Codestral (22B), StarChat2 (15B), Qwen-2.5 (32B), Olmo-3 (32B).
- Closed-source: GPT-4o.
Datasets: Three popular coding benchmarks were used to provide original code and reference test suites:
- HumanEval: Function-level code (164 problems).
- MBPP: Function-level code (100 sampled problems).
- APPS: Program-level code (100 sampled problems, noted for higher complexity).
Refactoring Types: Two specific tasks were prompted:
1. Performance Optimization ( $P_1$ ): Making code run faster.
2. Code Simplification ( $P_2$ ): Removing redundancy and making code concise.
Scale: The study generated and analyzed 4,368 total refactorings.

B. The Eq@DFuzz Approach

Instead of using the datasets' fixed test suites, the authors employed a differential fuzzing strategy:

Input Generation: A fuzzer (Atheris) generates a massive, diverse set of test inputs (2,000 for functions, 1,000 for programs) based on inferred constraints (types, dependencies, relational conditions).
Execution: Both the original code ( $C$ ) and the refactored code ( $C'$ ) are executed against every generated input ( $I$ ).
Equivalence Definition:
- $Eq@DFuzz(C, C') = 1$ if $\forall i \in I, C(i) = C'(i)$ .
- $Eq@DFuzz(C, C') = 0$ if any single output mismatch is detected.
- This binary definition is stricter than previous continuous similarity scores; a single semantic deviation classifies the refactoring as non-equivalent.

C. Baseline Comparison

The study compares the results of Eq@DFuzz against the traditional Test Passing metric ($Corr@Test$), which checks if the refactored code passes the dataset's original, limited test suite.

3. Key Results

RQ1: Extent of Functional Non-Equivalence

High Failure Rate: All six LLMs demonstrated a non-trivial tendency to alter program semantics. The rate of functionally non-equivalent refactorings ranged from 19% to 35%.
- Highest Failure: Codestral (~35%).
- Lowest Failure (but still significant): GPT-4o (~18.6%).
Impact of Complexity: Non-equivalence rates increased with code complexity. The APPS dataset (program-level) showed the highest failure rate (32.09%), significantly higher than MBPP (25.33%) and HumanEval (22.10%).
Refactoring Type: Contrary to the assumption that simplification is safer, both "Simplification" and "Optimization" tasks yielded nearly identical rates of non-equivalence (~26% overall).

RQ2: Reliability of Test Suites

The "Hidden" Defects: The study found that ~21.65% of the refactorings identified as non-equivalent by Eq@DFuzz still passed the original test suites.
Redundancy: Even the APPS dataset, which has significantly more test cases per problem than the others, failed to detect semantic divergence at a similar or slightly higher rate. This suggests existing test cases often validate redundant execution paths rather than critical semantic boundaries.
Conclusion: Relying solely on test passing leads to a significant overestimation of LLM refactoring correctness.

4. Key Contributions

Methodological Shift: Introduced Eq@DFuzz, a differential fuzzing-based equivalence checker that eliminates reliance on predefined test suites and explores a much larger input space.
Empirical Evidence of Risk: Provided large-scale evidence that even State-of-the-Art (SOTA) code LLMs frequently break functional equivalence (19–35% failure rate) during refactoring.
Critique of Current Benchmarks: Demonstrated that widely used coding benchmarks (HumanEval, MBPP, APPS) are insufficient for detecting semantic errors in refactored code, as ~21% of broken refactorings pass existing tests.
Complexity Correlation: Established a correlation between code complexity (program-level vs. function-level) and the likelihood of semantic divergence.

5. Significance and Implications

Trustworthiness: The findings challenge the assumption that LLMs can be safely integrated into CI/CD pipelines for automated refactoring without rigorous, fuzzing-based verification.
Evaluation Standards: The paper argues that the software engineering community must move beyond "test-passing" metrics for evaluating LLMs in tasks like code generation, translation, and refactoring.
Future Directions: The authors suggest that future work should focus on:
- Developing stronger equivalence-checking techniques (like differential fuzzing) as standard evaluation tools.
- Fine-tuning LLMs to optimize for both quality (speedup, simplicity) and functional equivalence simultaneously.
- Designing test suites that specifically target semantic boundaries rather than just coverage.

In summary, this paper serves as a critical warning: LLM-generated code refactorings are prone to subtle semantic errors that standard testing misses, necessitating a shift toward differential fuzzing for reliable validation.