Technical Summary: REFINE – A Multi-Agent LLM Approach for Evidence-Guided Code Refactoring

Problem Statement

While Large Language Models (LLMs) demonstrate strong capabilities in code generation and transformation, their application to software refactoring faces significant challenges. Effective refactoring requires not only modifying code to reduce quality issues (code smells) but also ensuring that changes do not introduce new defects, alter external behavior, or remove critical structural elements (e.g., public APIs, assertions).

Current LLM-assisted refactoring approaches often lack rigorous verification, leading to risks such as hallucinated suggestions, inconsistent transformations, and the removal of behavior-relevant code structures. There is a need for a systematic approach that:

1. Guides LLMs with evidence from static analysis.
2. Orchestrates a multi-agent workflow to plan, generate, and verify changes.
3. Provides traceable evidence of what was changed, what risks remain, and whether the output is a viable refactoring candidate rather than an automatically accepted solution.

Methodology: The REFINE Workflow

The authors introduce REFINE (Refactoring with Evidence-aware Flow for Integrated ageNtic Execution), a tool-agnostic, evidence-aware multi-agent framework designed for Java file-level refactoring. The system is implemented as a research prototype using a Next.js interface, a Java Spring Boot backend (for static analysis via PMD 7.x), and a Python agent service (using LangGraph v1.1) for orchestration.

The workflow operates through three primary stages:

1. Task Characterization

• Input: A single Java file from an open-source project.
• Evidence Gathering: Static analysis (PMD) identifies file-level code smells and rule-level evidence.
• Contextualization: The system extracts public API signatures, workspace context, and prioritizes detected smells to form a bounded refactoring task.

2. Refactoring Orchestration

The workflow coordinates eleven distinct roles (agents) to manage the process:

• Planning Agent: Combines deterministic rule-based guidance with optional LLM refinement to create a smell-oriented refactoring plan.
• Refactoring Agent: Invokes an LLM to generate a candidate transformation based on the plan, original source, and constraints (e.g., preserving public APIs).
• Verification Agent: Analyzes the generated candidate against configured evidence checks before it is retained.

3. Verification Trace & Analysis

REFINE does not treat LLM output as final. Instead, it re-analyzes the transformed file to compute:

• Smell Reduction: Absolute ($\Delta_{smell} = B - A$) and relative improvements in detected code smells.
• Static Preservation Proxies: Checks for public API preservation, exception handling, framework contracts, conditional logic, and critical assert/fail calls.
• Failure Diagnostics: Records specific reasons for rejection (e.g., public method removal).
• Traceability: Persists the original source, generated candidate, agent steps, metrics, and verification outcomes to link every decision to its evidence.

Experimental Design

• Dataset: 450 Java files from 15 open-source systems (e.g., JHotDraw, Apache Ant, Guava, JabRef), selected via stratified random sampling based on code-smell count and Lines of Code (LOC).
• LLM Configurations: Three frontier models were evaluated: OpenAI GPT-5.5, Google Gemini 3.1 Pro Preview, and Anthropic Claude Opus 4.8.
• Scale: This resulted in 1,350 model-pass outputs.
• Baseline: A matched direct-prompt baseline was conducted on a 150-file subset to compare the multi-agent workflow against simple prompting.
• Metrics: Code-smell reduction, quality indicators (Cyclomatic Complexity, Maintainability Index, etc.), structural changes, and preservation risks.

Key Results

1. Code Smell Reduction (RQ1)

REFINE achieved substantial reductions in detected code smells across all three LLM configurations:

• Total Reduction: 68.26% (GPT-5.5), 72.79% (Gemini 3.1), and 68.49% (Opus 4.8).
• Major Smells: The most significant improvements were in major code smells (86.51% to 91.60% reduction).
• Quality Indicators: Improvements in broader quality metrics were not uniform. While Gemini 3.1 showed significant reductions in Cyclomatic Complexity and LCOM, other metrics (Maintainability, Testability, Halstead effort) showed mixed or adverse changes depending on the model.

2. Preservation and Risks (RQ2)

• High Pass Rates: Most static preservation indicators (public method signatures, exception handling, framework contracts) passed at high rates (81.8% to 94.2%).
• Critical Risks:
  • Assert/Fail Calls: Only 57.1% of outputs preserved critical assert/fail calls across all models, indicating a systemic risk.
  • Public Method Removal: This was the most concrete diagnostic failure. Gemini 3.1 exhibited the highest rate of public method removal (71 cases), followed by GPT-5.5 (41) and Opus 4.8 (34).

3. Refactoring Behavior (RQ3)

Different models achieved smell reduction through distinct edit profiles:

• GPT-5.5: Produced the most compact edits.
• Gemini 3.1: Exhibited a "deletion-heavy" profile, removing the most lines and methods.
• Opus 4.8: Showed an "extraction-heavy" profile with the highest number of method extractions.
• Correlation: Larger edit volumes correlated with higher absolute smell reduction but not necessarily higher relative reduction.

4. Comparison with Direct Prompting

On the matched 150-file subset, REFINE outperformed direct prompting in:

• Smell Reduction: Median total reduction of 100.0% (REFINE) vs. 20.8% (Direct Prompt).
• Edit Footprint: Smaller median churn (14 LOC vs. 65 LOC).
• API Safety: Fewer public method removals (46 cases vs. 112).
• Trade-off: Direct prompting preserved critical assert/fail constructs more often (100% vs. 58%).

Significance and Claims

The paper positions REFINE not as a replacement for behavior-preserving refactoring tools, but as a traceable, evidence-aware mechanism for generating and evaluating refactoring candidates.

• Evidence-Awareness: The primary contribution is linking generated code to the specific static evidence (smells) that motivated the change and the verification checks it passed or failed.
• Controlled Generation: The multi-agent workflow provides more controlled candidate generation than direct prompting, resulting in smaller edits and fewer accidental API removals, though it does not eliminate all behavioral risks.
• Limitations: The authors explicitly state that the generated outputs are candidates, not production-ready refactorings. The static preservation checks are proxies and do not prove behavioral equivalence.
• Practical Implication: Generated candidates require compilation, testing, dependency analysis, and human review before adoption, particularly in dependency-rich or system-level settings.

The study concludes that while evidence-aware multi-agent workflows are promising for targeted file-level code-smell mitigation, neither direct prompting nor the multi-agent approach currently provides sufficient evidence of full behavior preservation to operate autonomously in complex software ecosystems.