💻 computer science

A Comprehensive Evaluation of Parameter-Efficient Fine-Tuning on Code Smell Detection

This paper addresses the challenges of automated code smell detection by introducing a rigorously validated benchmark and demonstrating that Parameter-Efficient Fine-Tuning (PEFT) methods significantly outperform existing baselines while substantially reducing computational costs compared to full fine-tuning.

Original authors: Beiqi Zhang, Peng Liang, Xin Zhou, Xiyu Zhou, David Lo, Qiong Feng, Zengyang Li, Lin Li

Published 2026-03-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Beiqi Zhang, Peng Liang, Xin Zhou, Xiyu Zhou, David Lo, Qiong Feng, Zengyang Li, Lin Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the manager of a massive, bustling library (the software code). Over time, books start getting messy: some have pages torn out, others are written in a confusing jargon, and some chapters are so long they make your head spin. In the world of software, these messes are called "Code Smells." They aren't necessarily broken, but they make the library hard to navigate, hard to fix, and prone to catching fire (bugs) later on.

For years, librarians (developers) tried to find these smells using two main tools:

The Rulebook (Heuristics): A strict checklist. "If a chapter is longer than 50 pages, it's a smell." The problem? This is too rigid. A 50-page chapter might be a masterpiece, while a 10-page chapter could be a confusing mess. The rulebook misses the nuance.
The Junior Librarians (Old AI): Machine learning models that tried to learn the rules. They were better but still often got it wrong because they didn't really "understand" the story, just the statistics.

Then came the Super Librarians (Large Language Models or LLMs). These are like geniuses who have read every book in the world. They can understand the story, the tone, and the context. They can tell you, "Hey, this short chapter is actually a nightmare because the plot makes no sense," or "This long chapter is fine because it's a clear instruction manual."

The Problem:
These Super Librarians are incredibly expensive to hire and train. To teach one specifically about your library's messy books, you usually have to "Full Fine-Tune" them. Imagine hiring a genius, locking them in a room with your entire library, and making them re-read every single book from scratch to learn your specific rules. It costs a fortune in electricity (computing power) and time.

The Solution: The "Sticky Note" Strategy (PEFT)
This paper asks: Do we really need to retrain the whole genius? Or can we just give them a few clever "Sticky Notes" (Parameter-Efficient Fine-Tuning or PEFT) to help them focus on the messy books?

The authors tested four different types of "Sticky Notes":

Prompt Tuning: Writing a special instruction at the very top of the page.
Prefix Tuning: Adding a secret code before every sentence.
LoRA: Giving the librarian a specialized pair of glasses that only highlights the messy parts.
(IA)³: A subtle filter that amplifies the librarian's attention to specific details.

What They Did (The Experiment)

Built a Better Library Catalog:
Existing catalogs of "messy books" were flawed. They were often just lists of page counts (metrics) or had labels made by the rigid Rulebook (which we know is imperfect).
- The Fix: The authors went into the wild, found real Java code from popular projects, and had human experts manually review thousands of examples. They created a Gold-Standard Catalog of four specific types of messes:
  - Complex Conditional: A sentence with too many "Ifs" and "Ands" that makes no sense.
  - Complex Method: A function that tries to do too many things at once.
  - Feature Envy: A character in a story who keeps borrowing things from a neighbor's house instead of using their own.
  - Data Class: A character who is just a box of stuff with no personality or actions.
The Great Test:
They took 9 different "Librarians" (ranging from small, efficient ones to massive, powerful ones) and tried to teach them using the four "Sticky Note" methods. They compared these against:
- The old Rulebooks (DesigniteJava, PMD).
- The Junior Librarians (Old AI).
- The Super Librarians just guessing without training (In-Context Learning).

The Results (The Plot Twist)

1. The "Sticky Notes" Worked Wonders:
The most surprising finding? You don't need to retrain the whole genius.
Using these lightweight "Sticky Note" methods (PEFT), the models performed just as well as, or even better than, the expensive "Full Retraining" method.

Analogy: It's like giving a genius a specialized highlighter pen. They can spot the messy books perfectly without needing to memorize the whole library again. Plus, it uses 90% less electricity (GPU memory).

2. Size Doesn't Always Matter:
Usually, bigger is better. But for some specific smells (like "Data Class"), a smaller, cheaper librarian (Small Language Model) actually did a better job than the massive Super Librarian.

Analogy: Sometimes you don't need a PhD to find a typo; a sharp-eyed high schooler with a good checklist is faster and just as accurate.

3. The "Sticky Note" Choice Matters:
Not all sticky notes are the same.

For the Super Librarians (LLMs), the (IA)³ method was the clear winner.
For the Smaller Librarians (SLMs), Prefix Tuning and (IA)³ were the best.
Lesson: You have to pick the right tool for the right job. One size does not fit all.

4. The Old Tools Are Outclassed:
The old Rulebooks and Junior Librarians were left in the dust. The new PEFT-trained models were significantly more accurate, catching smells that the old tools missed and ignoring false alarms the old tools raised.

Why This Matters to You

If you are a developer or a tech manager, this paper is great news. It means:

Cheaper: You can build powerful code-smell detectors without buying a supercomputer.
Faster: You can train these models in minutes instead of days.
Smarter: You can catch subtle design flaws that humans and old tools miss, keeping your software healthy and easy to maintain.

In a nutshell: The authors proved that you don't need to rebuild the engine to fix a car. Sometimes, you just need the right set of tools (PEFT) to tune it up, and it will run better than ever before.

1. Problem Statement

Code smells are indicators of suboptimal design that degrade software quality, maintainability, and readability. While automated detection is crucial, existing approaches face significant limitations:

Heuristics-based tools (e.g., DesigniteJava, PMD) rely on static rules and thresholds, leading to high sensitivity to parameter selection, lack of semantic understanding, and frequent false positives/negatives.
Traditional ML/DL models struggle to capture deep semantic and structural relationships within source code, often yielding unsatisfactory performance.
Large Language Models (LLMs) offer promise but face two major barriers:
1. Data Scarcity: Existing datasets often rely on software metrics rather than raw source code, focus on simple smells, or contain noisy labels generated by heuristics (the "Oracle Problem").
2. Adaptation Cost: Full fine-tuning of LLMs is computationally prohibitive (requiring massive GPU memory), while In-Context Learning (ICL) without fine-tuning often yields suboptimal results for specific tasks.

There is a lack of rigorous evaluation on whether Parameter-Efficient Fine-Tuning (PEFT) methods can effectively adapt LLMs for code smell detection while overcoming data and resource constraints.

2. Methodology

A. Dataset Construction (The "LM-Ready" Benchmark)

To address the data bottleneck, the authors constructed a high-quality, source-code-centric benchmark covering four non-trivial code smells:

Target Smells: Complex Conditional (CC), Complex Method (CM), Feature Envy (FE), and Data Class (DC).
Source: Mined from 401 popular, recent Java repositories on GitHub (2019–2024).
Pipeline:
1. Initial Detection: Used DesigniteJava and PMD to identify potential positive and negative samples.
2. Preprocessing: Extracted methods/classes using CodeSplitJava, removed duplicates, and filtered for a maximum token limit of 1,024.
3. Two-Stage Manual Review: A rigorous human annotation process was conducted to eliminate false positives/negatives and ensure label reliability. Annotators established specific criteria (e.g., nesting depth, data access patterns) and achieved high inter-rater agreement (Cohen's Kappa > 0.77).
Result: A balanced dataset with verified ground truth, split 8:1:1 for training, validation, and testing.

B. Experimental Setup

Models: Evaluated 4 Small Language Models (SLMs <1B params: CodeBERT, GraphCodeBERT, UnixCoder, CodeT5) and 5 Large Language Models (LLMs ≥1B params: StarCoderBase, DeepSeek-Coder, CodeLlama).
PEFT Techniques: Compared four state-of-the-art methods:
- Prompt Tuning: Learnable soft prompts appended to inputs.
- Prefix Tuning: Trainable prefix vectors prepended to Key/Value matrices in all layers.
- LoRA (Low-Rank Adaptation): Injecting low-rank decomposition matrices into attention projections.
- (IA)³ (Infused Adapter by Inhibiting and Amplifying Inner Activation): Rescaling vectors for Key, Value, and FFN activations.
Baselines:
- Heuristics: DesigniteJava, PMD.
- Deep Learning: DeepSmells, AE-Dense.
- General LLMs (ICL): GPT-4o-mini and DeepSeek-v3 (Zero-shot, Few-shot, and Criteria-informed).
Metrics: Precision, Recall, F1-score, and MCC (Matthews Correlation Coefficient) to account for class imbalance.

3. Key Contributions

High-Quality Benchmark: Creation of a reliable, source-code-centric dataset for four complex code smells, addressing the "Oracle Problem" and metric-dependency of previous datasets.
Comprehensive PEFT Evaluation: The first extensive study evaluating PEFT methods across both SLMs and LLMs for code smell detection, covering both method-level (CC, CM, FE) and class-level (DC, FE) understanding.
Systematic Baseline Comparison: A rigorous benchmarking against traditional tools, DL models, and SOTA LLMs using ICL.
Actionable Insights: Practical guidelines for selecting PEFT methods based on model size, code smell type, and resource availability.

4. Key Results

Effectiveness of PEFT vs. Full Fine-Tuning

Performance: PEFT methods generally match or exceed the performance of full fine-tuning across most models and smell types.
Efficiency: PEFT significantly reduces peak GPU memory usage (critical for LLMs) while updating only a tiny fraction of parameters (e.g., <1% of total weights).
Best Methods:
- For LLMs, (IA)³ consistently delivered the best results across all smell types.
- For SLMs, (IA)³ and Prefix Tuning were most effective.
- LoRA showed instability, sometimes performing significantly worse than other PEFT methods on specific models (e.g., StarCoderBase for CM detection).

Model Performance (SLMs vs. LLMs)

LLMs generally outperformed SLMs, particularly for Feature Envy (which requires complex class-level context).
SLMs (specifically GraphCodeBERT) significantly outperformed LLMs in detecting Data Class smells, suggesting smaller models can be highly effective for certain structural patterns.
DeepSeek-Coder-1.3B emerged as the top-performing LLM, often outperforming larger models like CodeLlama-7B.

Hyper-parameters and Low-Resource Scenarios

Hyper-parameters: Increasing the number of trainable parameters (e.g., LoRA rank $r$ ) does not guarantee better performance. Optimal settings vary by smell type.
Low-Resource: With very limited data (50 samples), performance drops significantly. However, with 250–500 samples, PEFT methods often outperform full fine-tuning, demonstrating robustness in data-scarce environments.

Comparison with SOTA Baselines

PEFT-tuned LMs outperformed all baselines (heuristics, DL, and ICL-based LLMs).
MCC Improvements: PEFT methods achieved MCC improvements ranging from 0.33% to 13.69% over the best baselines.
ICL Limitations: While ICL (especially with DeepSeek-v3) performed well for CC and DC, it lagged significantly behind PEFT for CM and FE, indicating that complex semantic reasoning requires model adaptation rather than just prompting.

5. Significance and Implications

Practical Adoption: The study proves that PEFT is a viable, cost-effective alternative to full fine-tuning for integrating LLMs into software engineering workflows, enabling real-time code smell detection without prohibitive hardware costs.
Data Quality: The constructed dataset sets a new standard for code smell research, moving away from noisy, metric-based labels toward human-verified, source-code-centric ground truth.
Strategic Guidance: The paper provides a decision framework for practitioners:
- Use (IA)³ for LLMs to maximize performance.
- Use Prefix Tuning or (IA)³ for SLMs.
- Use LoRA specifically in low-resource scenarios.
- Consider SLMs for specific smells (like Data Class) to save resources.
Future Directions: Highlights the potential of Retrieval-Augmented Generation (RAG) to further enhance detection without fine-tuning and calls for cross-language and broader smell-type evaluations.

In conclusion, this work demonstrates that Parameter-Efficient Fine-Tuning is not just a resource-saving technique but a performance-enhancing strategy that enables Large Language Models to effectively detect complex code smells, surpassing both traditional tools and unadapted LLMs.