Technical Summary: SWE-Refactor

Problem Statement

While Large Language Models (LLMs) have demonstrated significant capability in code generation, code refactoring presents distinct challenges. Refactoring requires precise, semantics-preserving edits that improve program structure without altering behavior. This task demands repository-level reasoning, an understanding of complex dependencies, and iterative verification—capabilities that are difficult to evaluate and often beyond the reach of current models.

Existing benchmarks for code refactoring suffer from four critical shortcomings:

1. Limited Scenario Coverage: They predominantly focus on atomic refactoring types (single transformations) rather than the compound refactorings (multiple coordinated transformations) common in real-world development.
2. Lack of Automated Construction: Many benchmarks rely on manual curation or LLM-generated ground truth, introducing bias and scalability issues.
3. Insufficient Repository-Level Context: Existing datasets often lack the structural information (e.g., class hierarchies, caller/callee relationships) necessary for realistic assessment.
4. Noisy Data: Many benchmarks include "impure" commits where refactoring is mixed with bug fixes or feature additions, making it difficult to isolate and evaluate the model's refactoring capabilities.

Furthermore, the field suffers from a lack of diversity in programming languages, with most recent benchmarks focusing exclusively on Python, despite Java being a dominant language in enterprise systems.

Methodology: SWE-Refactor

To address these gaps, the authors introduce SWE-Refactor, a repository-level benchmark constructed from real-world Java projects. The benchmark is built using a fully automated four-step pipeline:

1. Mining via Static Analysis: The authors leverage RefactoringMiner, an AST-based tool, to extract commits containing refactorings from 18 widely used Java projects.
2. Curation of Pure Refactorings: Using PurityChecker (an extension of RefactoringMiner), the pipeline filters out impure changes (e.g., bug fixes) to retain only "pure" refactorings. The authors manually verified a random sample of 200 instances to confirm the detectors' reliability.
3. Enrichment with Multi-Level Information: To support repository-level reasoning, the pipeline uses Eclipse JDT to extract comprehensive context, including project structure, full class bodies, method callers/callees, and build configurations.
4. Verification: The pipeline ensures functional correctness by compiling the project with the appropriate JDK version and running the full test suite (using JaCoCo) to verify that the refactoring preserves behavior.

Dataset Composition:

• Scale: 1,099 developer-written, behavior-preserving refactorings.
• Sources: 18 diverse Java projects.
• Types: Includes 922 atomic instances and 177 compound instances.
• Refactoring Types: Covers three atomic types (Extract Method, Move Method, Inline Method) and three compound types (Extract and Move, Move and Inline, Move and Rename).

Evaluation Metrics:
The benchmark evaluates LLM performance on two dimensions:

1. Functional Correctness: Measured by compilation success, test pass rates, and AST-Based Refactoring Verification (using RefactoringMiner to ensure the intended transformation occurred without unintended side effects).
2. Human-Likeness: Measured using CodeBLEU to assess similarity to the developer-written ground truth.

Key Contributions

1. SWE-Refactor Benchmark: A new, high-quality dataset of 1,099 pure refactorings from 18 Java projects, covering both atomic and compound transformations. It is the first to provide comprehensive repository-level context and automated verification for Java.
2. Automated Construction Pipeline: A reproducible, four-step pipeline that extracts real-world refactorings, filters for purity, enriches structural data, and verifies correctness without manual annotation or LLM-generated ground truth.
3. Comprehensive Evaluation: An extensive assessment of 9 widely used LLMs (including GPT-4o-mini, DeepSeek-V3, and various CodeLLaMa/Qwen variants) across different refactoring types and prompting strategies (Simple Prompting, RAG, and Multi-Agent workflows).

Experimental Results

The authors evaluated 9 LLMs on the 1,099 instances. Key findings include:

• Overall Performance: General-purpose models outperformed open-source code-specific models. DeepSeek-V3 achieved the highest success rate (41.58%), followed by GPT-4o-mini (39.85%). Open-source models like CodeLLaMa-7B achieved only 1.10% success.
• Atomic vs. Compound: Models performed significantly better on atomic refactorings (local edits) than on compound refactorings (cross-file, multi-step transformations).
  • DeepSeek-V3 excelled at Extract Method (301 successes).
  • GPT-4o-mini showed broader generalization in cross-file tasks like Move Method (92 successes).
  • Open-source models struggled almost entirely with compound types.
• Impact of Advanced Techniques:
  • Multi-Agent Workflows: A workflow using a Developer Agent and a Reviewer Agent (using GPT-4o-mini) achieved the highest overall success (579 instances), significantly outperforming simple prompting and Retrieval-Augmented Generation (RAG), particularly on complex compound refactorings.
  • Agentic Scaffolding (OpenAI Codex): Even a powerful agent (GPT-5.1-Codex) achieved only a 39.4% success rate on compound instances, compared to 82.6% on atomic instances, highlighting the persistent difficulty of multi-step, tightly constrained transformations.

Significance and Claims

The paper positions SWE-Refactor as a necessary evolution in software engineering benchmarks. Its significance lies in:

• Realism: By using developer-written commits from real projects, it avoids the biases and hallucinations associated with synthetic or LLM-generated benchmarks.
• Complexity: It explicitly targets the gap between atomic and compound refactorings, providing a rigorous test for repository-level reasoning capabilities.
• Language Diversity: It addresses the Python-centric bias in current literature by providing a robust Java benchmark, reflecting the languages used in large-scale enterprise systems.
• Reliability: The automated pipeline ensures that the ground truth is behavior-preserving and structurally correct, offering a trustworthy standard for future research.

The authors conclude that while LLMs show promise in code generation, they remain significantly challenged by complex, compound refactoring tasks, suggesting that current agentic frameworks are not yet reliable for multi-step software maintenance without human oversight.