RefAgent: A Multi-agent LLM-based Framework for Automatic Software Refactoring

This paper introduces RefAgent, a multi-agent LLM-based framework that autonomously plans, executes, and iteratively refines software refactoring through specialized agents, demonstrating significant improvements in code quality, test pass rates, and smell reduction compared to single-agent and traditional approaches.

The Gist

Imagine you have a massive, old library. Over the years, books have been added, pages torn out, and notes scribbled in margins. The stories are still there, but the organization is a mess. Some books are too thick, some chapters are written in a language no one understands, and finding a specific story takes forever. This is what happens to software code as it grows: it gets "messy," hard to maintain, and prone to breaking. This mess is called technical debt or code smells.

Traditionally, cleaning up this library (a process called refactoring) requires a team of human librarians to manually read every book, decide what to fix, rewrite the pages, and check if the story still makes sense. It's slow, expensive, and prone to human error.

Enter RefAgent, a new "robot librarian" system described in this paper. But instead of one robot trying to do everything, RefAgent is a team of specialized robot librarians working together, powered by advanced AI (Large Language Models).

Here is how RefAgent works, broken down into simple concepts:

1. The Team of Specialized Robots (Multi-Agent System)

Instead of one AI trying to be a genius at everything, RefAgent splits the job into four distinct roles, like a well-oiled machine:

• The Detective (Context-Aware Planner): This robot doesn't just look at one book; it looks at the whole library. It checks which books depend on each other (e.g., "Book A mentions Book B"). It measures how messy the books are and creates a blueprint for cleaning up. It decides what needs to be fixed and how.
• The Architect (Refactoring Generator): Once the Detective gives the blueprint, this robot actually does the rewriting. It takes the messy code and rewrites it according to the plan, trying to make it cleaner and more efficient.
• The Safety Inspector (Compiler Agent): Before anyone can read the new book, the Safety Inspector checks if the pages are glued together correctly. If the new code has a syntax error (like a typo that breaks the sentence), this robot catches it, tells the Architect, "Hey, this sentence doesn't make sense," and sends it back for a fix. They keep looping this until the code is perfect.
• The Quality Assurance Tester (Tester Agent): This robot ensures the story hasn't changed. Even if the book looks cleaner, does it still tell the same story? The Tester runs a battery of automated tests (like asking a thousand people to read a specific page and verify the plot). If the story changes, the Architect has to go back and fix it.

2. The "Self-Correction" Loop

One of the coolest things about RefAgent is that it doesn't just guess once. It uses a feedback loop.

• Imagine the Architect writes a new chapter.
• The Safety Inspector says, "This paragraph is broken."
• The Architect fixes it.
• The Tester says, "Now the ending is wrong."
• The Architect fixes that too.

They do this up to 20 times for each piece of code until everything is perfect. This is like a writer, an editor, and a fact-checker arguing and refining a manuscript until it's ready for print.

3. The Results: Did it Work?

The researchers tested RefAgent on 8 huge, real-world software projects (like JClouds and Apache projects). They compared their robot team against:

• Single AI: One robot trying to do everything alone.
• Search-based tools: Old-school tools that try to find fixes by brute force.
• Human Developers: The actual people who wrote the code.

The findings were impressive:

• Success Rate: RefAgent successfully fixed the code without breaking it 90% of the time (measured by passing tests). A single AI only managed about 45%.
• Cleanliness: It removed about 52% of the "code smells" (the messy parts).
• Human-like: It identified problems and fixed them in a way that was 80% similar to what human developers would have done.
• Better than the competition: It outperformed both the single AI and the old search-based tools significantly.

4. Why This Matters

Think of RefAgent as a self-driving car for software maintenance.

• Old way: You drive the car, but you have to manually check the oil, change the tires, and fix the engine every time something goes wrong.
• RefAgent way: You have a team of sensors and mechanics inside the car. They detect a flat tire, decide to change it, check if the new tire is secure, and verify the car drives smoothly, all while you just sit back and watch.

The Bottom Line

This paper proves that AI teams are much better at cleaning up messy code than single AI bots or old tools. By giving the AI specific roles (planner, coder, tester) and letting them talk to each other to fix mistakes, we can automate the boring, difficult work of software maintenance. This saves time, reduces errors, and keeps software healthy for longer, allowing human developers to focus on building new features instead of just cleaning up old messes.

Technical Summary

Here is a detailed technical summary of the paper "RefAgent: A Multi-agent LLM-based Framework for Automatic Software Refactoring."

1. Problem Statement

Software systems inevitably accumulate technical debt and "code smells" (design flaws that hinder maintainability) as they evolve. While software refactoring is the standard practice to address these issues without altering external behavior, manual refactoring is error-prone and time-consuming. Existing automated solutions face significant limitations:

• IDE Tools: Support only a limited set of refactorings and require manual intervention.
• Search-Based Refactoring: Formulate refactoring as an optimization problem but often radically alter program design, lack guarantees of behavior preservation, and are computationally prohibitive for large projects.
• Static LLM Approaches: Traditional Large Language Model (LLM) applications for refactoring rely on static prompts. They often struggle with scalability, maintaining behavioral correctness, and handling the complex, multi-step workflows required for real-world refactoring (e.g., planning, transforming, compiling, and testing).

There is a critical need for a dynamic, context-aware, and adaptive system that can autonomously orchestrate the entire refactoring lifecycle.

2. Methodology: The RefAgent Framework

The authors propose RefAgent, a fully automated, multi-agent framework designed to simulate the sequential, iterative nature of human refactoring workflows. Instead of a single LLM generating code in one shot, RefAgent distributes responsibilities among four specialized agents that collaborate via tool-calling and feedback loops.

Core Components

1. Context-Aware Planner Agent:

   • Role: Identifies refactoring opportunities and generates a structured plan.
   • Mechanism: Uses tools like jdeps to build a class dependency graph and DesigniteJava to extract quality metrics (e.g., cyclomatic complexity, lack of cohesion).
   • Output: A JSON-formatted plan detailing specific code regions (methods, fields) to refactor and explicit instructions, avoiding bias by using metrics for context rather than direct smell labels.

2. Refactoring Generator Agent:

   • Role: Executes the refactoring plan to produce the modified code.
   • Mechanism: Takes the original source code, dependencies, metrics, and the planner's instructions. It generates the refactored Java class.
   • Adaptability: If called by the Compiler or Tester agents due to errors, it dynamically adjusts its prompt to fix issues while adhering to the original refactoring plan.

3. Compiler Agent:

   • Role: Ensures syntactic correctness.
   • Mechanism: Interacts with the Maven compiler tool. If compilation fails, it analyzes the error logs, generates a summary, and feeds it back to the Generator Agent for correction.
   • Constraint: Operates in a feedback loop with a maximum of 20 iterations.

4. Tester Agent:

   • Role: Ensures behavioral preservation (functionality is not broken).
   • Mechanism:
     • Retrieves existing developer-written tests via jdeps.
     • Generates additional regression tests using EvoSuite (optimized for code coverage).
     • Executes tests via Maven.
   • Logic: If tests fail, it analyzes logs and requests fixes from the Generator Agent. If the patch fails after 20 iterations, the original code is retained to prevent regression.

Workflow

The framework processes a Java project by iterating through classes. For each target class, it retrieves context, plans, generates, compiles, and tests. The process relies on self-reflection (analyzing errors) and iterative refinement (up to 20 loops) to ensure high-quality output.

3. Experimental Setup

• Dataset: 8 open-source Apache Java projects (e.g., JClouds, Accumulo, SkyWalking) totaling over 2,000 classes and 600+ KLOC.
• Models Evaluated:
  • GPT-4o (Closed-source, 128K context).
  • StarCoder2-15B (Open-source).
  • DeepSeekCoder-33B (Open-source).
• Baselines:
  • Single-Agent Approaches: Comparing against a single LLM with static prompts (Pass@1 and Pass@3 metrics).
  • Search-Based Tool: RefGen, a tool using partial order reduction to schedule refactorings.
  • Human Developers: Historical refactoring commits from the projects.
• Evaluation Metrics:
  • Code Quality: Code Smell Reduction Rate (SRR) and QMOOD metrics (Reusability, Flexibility, Understandability, etc.).
  • Correctness: Unit Test Pass Rate and Compilation Success Rate.
  • Alignment: Precision, Recall, and F1-score comparing RefAgent's changes against developers and RefGen.

4. Key Results

The evaluation demonstrates that RefAgent significantly outperforms existing approaches across multiple dimensions:

• Quality Improvement:
  • Achieved a median Code Smell Reduction Rate of 52.5%.
  • Improved key quality attributes (e.g., Reusability) by a median of 8.6%.
• Correctness & Stability:
  • Unit Test Pass Rate: Median of 90% (GPT-4o/DeepSeek) and 85% (StarCoder).
  • Compilation Success Rate: Median of 87-88%.
  • Comparison: RefAgent improved the median unit test pass rate by 64.7% and compilation success by 40.1% compared to single-agent approaches.
• Alignment with Human/Tool Baselines:
  • vs. Developers: Median F1-score of 79.15%, indicating RefAgent identifies refactoring opportunities similar to human intuition.
  • vs. RefGen: Median F1-score of 72.7%, showing strong alignment with search-based optimization.
• Ablation Study:
  • Removing context retrieval caused a significant drop in pass rates, proving that dependency analysis is critical.
  • Iterative refinement (up to 20 loops) was essential; performance stabilized only after ~12 iterations.
  • The multi-agent architecture proved robust across different underlying LLMs (GPT-4o, StarCoder, DeepSeek), suggesting the framework's value lies in the orchestration rather than just the model.

5. Key Contributions

1. Novel Framework: Introduction of RefAgent, the first multi-agent LLM framework specifically designed for end-to-end automated software refactoring.
2. Dynamic Workflow: Moves beyond static prompting by implementing a closed-loop system where agents plan, execute, compile, and test iteratively, mimicking human developer workflows.
3. Comprehensive Evaluation: A rigorous study on 8 large-scale projects comparing against state-of-the-art search-based tools, single-agent LLMs, and historical human refactoring data.
4. Behavior Preservation: Demonstrates that complex, automated refactoring can be achieved without breaking existing functionality, evidenced by high unit test pass rates.

6. Significance

This work represents a paradigm shift in automated software engineering. It proves that multi-agent architectures can effectively manage the complexity of software refactoring, overcoming the brittleness and hallucination issues common in single-agent LLM applications. By successfully balancing the reduction of technical debt (code smells) with the preservation of functional correctness, RefAgent offers a viable path toward fully autonomous software maintenance, potentially reducing the high costs associated with technical debt and manual refactoring efforts. The study also highlights that the orchestration of specialized roles (planning, coding, testing) is more critical than the specific LLM model used, providing a blueprint for future agentic software tools.