Technical Summary: SmellBench

Problem Statement

Recent advances in Large Language Models (LLMs) have enabled the development of sophisticated code agents capable of handling complex software engineering tasks, including code generation, bug fixing, and repository-level reasoning. However, a critical gap exists in evaluating these agents: while they often produce functionally correct code, their misuse can lead to bloated, disorganized code that severely impairs readability, extensibility, and robustness.

Existing benchmarks primarily focus on functional correctness (e.g., passing test cases) but largely neglect long-term maintainability and code quality. Furthermore, current refactoring benchmarks suffer from significant limitations:

1. Manual Curation: High-quality, human-curated datasets are costly to produce and lack scalability.
2. Commit Mining: Automatically extracting refactoring cases from GitHub commit histories often results in "noisy" data where refactoring is entangled with feature implementation and bug fixes, making it difficult to isolate specific refactoring scenarios.
3. Lack of Purity: Existing datasets struggle to provide controlled, diverse, and high-quality refactoring cases with human-written ground truth.

Methodology: SmellBench Construction

To address these limitations, the authors propose SmellBench, an extensible benchmark that proactively injects code smells into clean code snippets from real-world repositories. The construction pipeline consists of four key stages executed within a Dockerized environment:

1. Repository Selection: Seven popular, large-scale Python repositories (e.g., click, pandas, numpy) were selected to ensure mature contributor guidelines and comprehensive test coverage.
2. Candidate Discovery: To avoid location bias and ensure validity, the pipeline employs a two-step process:
   • Coarse Filtering: Selects source files exceeding 500 lines of code to ensure structural richness.
   • Agent Localization: A code agent identifies specific methods or classes suitable for smell injection based on smell-specific constraints, rather than relying on rule-based matching.
3. Smell Injection: A powerful code agent (OpenHands + Claude Sonnet 4.5) modifies the clean code to introduce specific code smells while preserving compilability. The benchmark covers 7 distinct smell types:
   • Feature Envy
   • God Classes
   • Data Clumps
   • Shotgun Surgery
   • Dead Code
   • Interface Segregation
   • Deep Inlining
   • Configurations: Cases are stratified into 3 difficulty levels (Easy, Medium, Hard) and 2 instruction settings (Guided vs. Targeted) to test varying levels of task complexity and agent autonomy.
4. Quality Validation: The injected "smelly" code is validated against the repository's existing test suites. Cases are iteratively refined by the agent to ensure all unit tests pass, guaranteeing that the smell injection does not break functional correctness. This results in 294 valid, paired samples (clean vs. smelly) with human-written ground truth.

Evaluation Framework

SmellBench introduces a three-dimensional evaluation framework specifically designed for refactoring tasks, moving beyond simple pass/fail metrics:

1. Test Passing Rate (TPR): Measures functional correctness by verifying if the refactored code passes the original test suite.
2. Localization Accuracy (LA): Evaluates the agent's ability to correctly identify the specific code locations (classes or methods) requiring refactoring.
3. LLM-as-Judge: An LLM evaluates the quality of the refactoring across four complementary aspects:
   • Code Quality (CQ): Readability and maintainability improvements.
   • Structural Soundness (SS): Correctness and rationality of the new structure.
   • Cross-File Coordination (CC): Consistency and dependency handling across multiple files.
   • Smell Elimination (SE): Effectiveness in removing the targeted smell.

To ensure stability in the LLM-as-Judge evaluation, a preliminary Smell Analysis step is introduced to explicitly summarize the structural issues and refactoring objectives before the final judgment.

Experimental Results

The authors evaluated 2 open-source code agents (OpenHands, Qwen Code) paired with 6 representative LLMs (including Qwen3-Coder, DeepSeek-V3.2, GPT-5-Mini, Gemini-2.5-Flash, and Claude Sonnet 4.5).

• Overall Performance: Even the best-performing configuration (Qwen Code + Claude Sonnet 4.5) achieved a Smell Elimination (SE) score of only 50.34%. This indicates a significant gap between current code generation capabilities and the deep structural reasoning required for high-quality refactoring.
• Functional Correctness vs. Quality: Most models achieved Test Passing Rates (TPR) above 80%, demonstrating that agents can preserve functionality. However, high TPR did not correlate with high refactoring quality, reinforcing that functional correctness is insufficient for evaluating maintainability.
• Cross-File Challenges: Analysis revealed that agents struggle significantly with Shotgun Surgery (requiring coordinated changes across many files). While the average smell instance involved 2.5 files, Shotgun Surgery cases involved ~5.7 files, yet agents modified only ~2.7 files on average, failing to address the full scope of the smell.
• Localization Deficits: Agents exhibited weak autonomous localization capabilities. In "Guided" settings (where the agent must find the smell), Localization Accuracy stagnated around 65-67%, whereas "Targeted" settings (where the location is provided) boosted accuracy to over 90%.
• Open-Source Competitiveness: High-performance open-source models (e.g., Qwen3-Coder-480B) demonstrated competitiveness comparable to closed-source models like GPT-5-Mini in complex refactoring tasks.

Key Contributions and Significance

The paper claims the following contributions:

1. SmellBench Benchmark: A novel, extensible benchmark that proactively injects code smells, achieving a trade-off between data scalability and quality. It provides 294 high-quality cases across 7 smell types, 3 difficulty levels, and 2 instruction settings, grounded in human-written truth.
2. 3-Dimensional Evaluation Framework: A tailored evaluation system incorporating Test Passing Rate, Localization Accuracy, and LLM-based quality assessment (CQ, SS, CC, SE), offering nuanced insights beyond functional correctness.
3. Empirical Insights: Extensive experiments reveal that current state-of-the-art agents struggle with cross-file smell identification and coordinated modifications. The results highlight that relying solely on functional correctness masks significant deficiencies in software maintainability capabilities.
4. Decontamination and Purity: By generating new cases rather than mining existing commits, SmellBench avoids data leakage and ensures the isolation of refactoring tasks from other code changes.

The authors conclude that while code agents have made remarkable progress, there is substantial room for improvement in repository-level reasoning and the ability to handle complex, multi-file structural refactoring. SmellBench serves as a diagnostic tool to guide future developments in code agents toward better software maintainability.