💻 computer science

Cost-Effective Detection of Self-Admitted Technical Debt: A Collaborative Framework of Proxy Active Learning and Large Language Models

This paper proposes a cost-effective collaborative framework that combines a BERT-based proxy active learning strategy with full-parameter Flan-T5 training to accurately detect Self-Admitted Technical Debt using only 30% of the labeled data required by traditional full-supervision methods.

Original authors: Zhicheng Wang, Tianxin Lei, Tie Bao

Published 2026-11-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zhicheng Wang, Tianxin Lei, Tie Bao

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the life of a software project, speed often wins over perfection. When developers are racing to meet a deadline, they sometimes make a conscious choice to write code that is not quite right, knowing they will have to fix it later. They leave notes in the code itself, admitting that this section is a temporary workaround or a compromise. In the software world, this is known as self-admitted technical debt. It is a hidden tax on the future; while it helps the team ship a product today, that messy code makes the software harder to understand, more prone to breaking, and much more expensive to maintain tomorrow. Finding these hidden notes is crucial for keeping software healthy, but it is a difficult task. The notes are buried inside millions of lines of text, and they are rare, often appearing in only one or two out of every hundred comments. Furthermore, the language used is informal and full of human nuance, making it hard for standard computer programs to spot them without a massive amount of human help to teach the system what to look for.

A team of researchers from Jilin University in China has developed a new way to find this technical debt that saves time and money without sacrificing accuracy. They faced a dilemma: the most powerful computer models available today, known as large language models, are excellent at understanding human language and could spot these debt notes with high precision, but they are incredibly expensive to run and require vast amounts of labeled data to learn. On the other hand, smaller, faster models are cheap to run but often miss the subtle meaning behind the notes. The researchers solved this by creating a two-step partnership between a fast, simple model and a powerful, complex one. They first used the fast model to act as a scout, sifting through thousands of code comments to find the most interesting and difficult examples. Only these selected, high-value examples were then passed to the powerful model for deep learning. This approach allowed them to train a highly accurate system using only 30 percent of the data that would normally be required.

The study, which tested this method on ten popular open-source software projects written in the Java programming language, showed that this collaborative strategy works remarkably well. By using the fast model to filter the data, the researchers were able to train the powerful model to perform just as well as if it had been trained on the entire dataset, but with a fraction of the manual effort. In fact, for several of the projects tested, the new method actually performed better than the standard approach that uses all the data. The researchers found that when they stopped the training process after labeling just 30 percent of the comments, the system was already as good as, or sometimes even better than, systems trained on 100 percent of the data. This suggests that the key to efficiency is not just having more data, but having the right data. The fast model successfully identified the specific comments that were most useful for teaching the system, allowing the powerful model to learn quickly and effectively from a small, curated set of examples.

This work also looked at whether the method could work across different software projects, a scenario where a system trained on one project is asked to find debt in a completely different one. This is a common challenge because different teams use different words and styles. The researchers found that their method was very good at this too. The powerful model, having learned from the carefully selected examples, was able to generalize its knowledge and find debt in new projects with high accuracy, often outperforming other methods that tried to learn from the same projects without this filtering step. The results indicate that by combining the speed of a simple model with the deep understanding of a large one, software teams can identify and fix their technical debt much more efficiently. This means they can spend less time manually searching for problems and more time building reliable software, making the entire process of software maintenance more sustainable and cost-effective.

Technical Summary: Cost-Effective Detection of Self-Admitted Technical Debt

Problem Statement
Self-Admitted Technical Debt (SATD) refers to design compromises explicitly documented by developers in source code comments to facilitate future remediation. While identifying SATD is critical for reducing maintenance costs and enhancing software reliability, it presents significant challenges. The task is inherently a binary text classification problem characterized by extreme class imbalance (SATD instances often constitute less than 1.5% of comments) and data sparsity. Although deep learning models have improved detection accuracy, they rely heavily on large-scale, high-quality labeled datasets. Acquiring such data requires domain expertise and incurs prohibitive annotation costs. Furthermore, existing active learning approaches struggle to balance the need for deep semantic feature extraction with the computational overhead of Large Language Models (LLMs), which are often too expensive for high-frequency iterative training.

Methodology
The paper proposes a decoupled, collaborative framework that integrates Proxy Active Learning with Large Language Models (LLMs). The approach is structured into three core stages:

Data Preparation and Cold-Start Strategy:
To address the "cold-start" problem where random sampling fails to capture the sparse positive class, the authors design a hybrid initialization strategy. They utilize domain-specific keywords (e.g., "TODO," "FIXME," "HACK") to identify candidate positive samples. Within this keyword-rich pool, a K-Nearest Neighbors (KNN) algorithm calculates representativeness scores based on TF-IDF vectorization. The top 10% of samples with the highest representativeness scores form the initial training set, ensuring a sufficient supply of positive instances and typical exemplars without introducing bias from the deep learning model itself.
Proxy Screening Strategy (The "Proxy" Component):
To avoid the exorbitant inference costs of running LLMs during every active learning iteration, the framework employs a lightweight discriminative model, BERT, as a "proxy query engine."
- Hybrid Query Strategy: The BERT proxy executes a query strategy that fuses uncertainty (using the Least Confidence metric) and representativeness (using KNN density estimation in the semantic space).
- Fusion Mechanism: Instead of a weighted sum, the strategy employs a direct multiplicative fusion of normalized uncertainty and representativeness scores to rank unlabeled samples.
- Process: In each iteration, the BERT proxy screens the unlabeled pool to select the most informative samples (those near the decision boundary and representative of the feature space). These samples are then sent for manual annotation.
Collaborative Training with Target LLM:
The target learner is Flan-T5-Large, a generative model known for robust reasoning via instruction tuning. Unlike the proxy, Flan-T5 is not used for iterative screening. Instead, it undergoes one-time full-parameter fine-tuning only after the active learning process reaches a predefined stopping threshold (i.e., when the labeled dataset reaches a specific size). The classification head of Flan-T5 is modified (replacing the generation layer with a classification head) to perform the SATD detection task. This decoupling allows the system to leverage the semantic depth of the LLM for final performance while relying on the efficient BERT proxy for the costly data selection loop.

Key Contributions

Deep Active Learning with Hybrid Query: The authors propose a strategy that combines uncertainty and representativeness, enhanced by domain-specific keywords and TF-IDF for cold-start initialization. This resolves sampling redundancy and ensures the initial training set is representative of the sparse positive class.
Proxy-LLM Collaborative Framework: A novel architecture that decouples sample screening from model training. It utilizes BERT as a computationally efficient proxy to localize key samples, which are then used to train a larger-scale Flan-T5 model. This design harmonizes deep semantic comprehension with computational efficiency, bypassing the inference bottlenecks of LLMs in active learning loops.
Empirical Validation: The method was validated across ten popular Java open-source projects (e.g., Dubbo, Maven, SpringFramework) in both within-project and cross-project scenarios.

Results

Performance: The proposed Flan-T5-AL method achieves performance comparable to, and in several cases exceeding, fully supervised baselines. In within-project scenarios, it achieved a mean F1-score of 0.771, outperforming the fully supervised BERT baseline (0.739) and the fully supervised Flan-T5 baseline (0.754).
Cost Efficiency: The framework significantly reduces manual annotation requirements. Experimental results indicate that using only 30% of the labeled data (stopping threshold $\mu = 0.3$ ) allows the model to achieve performance ratios of 100% or higher compared to full-supervision baselines in projects like Groovy, Maven, and Storm.
Generalization: In cross-project scenarios (Leave-One-Project-Out), the collaborative framework demonstrated superior domain transfer capabilities, achieving a mean F1-score of 0.776, surpassing both fully supervised BERT (0.736) and Flan-T5 (0.761). The active learning component effectively filtered out project-specific noise, selecting universal semantic features of technical debt.

Significance
The paper claims that its primary significance lies in resolving the "high-precision requires high-cost" dilemma in SATD detection. By decoupling the screening process from the final model training, the framework enables the deployment of high-precision, LLM-driven detectors with a drastically reduced annotation budget (approximately 30% of the original requirement). This approach lowers the practical barrier for identifying SATD, making it feasible to integrate robust debt detection mechanisms into standard software maintenance workflows without the prohibitive computational and human costs associated with training LLMs on massive labeled datasets. The study suggests that this collaborative paradigm effectively transcends the performance ceilings of traditional discriminative models while maintaining computational feasibility.

Technical Summary: Cost-Effective Detection of Self-Admitted Technical Debt

More like this