A First Look at the Self-Admitted Technical Debt in Test Code: Taxonomy and Detection

Software is never truly finished. Even after a program is released, developers must constantly return to it to fix errors, add new features, and adapt to changing needs. This ongoing work is called maintenance, and it often requires more effort than the initial creation of the software itself. To keep this work manageable, programmers sometimes leave notes in their code, admitting that a particular section is messy, temporary, or not quite right. They might write a comment saying, "This is a hack," or "Fix this later." In the world of software engineering, these honest admissions are known as self-admitted technical debt. They are like a developer saying, "I know this isn't the best way to do it, but we needed to get it done now." While researchers have long studied these notes in the main code that runs a program, they have largely ignored the notes found in the code used to test that program. This is a significant oversight, because if the tests themselves are flawed or poorly written, the entire software system becomes unreliable.

A team of researchers at the University of Manitoba set out to understand this hidden layer of debt. They focused on a specific type of software written in Java, a language widely used for building complex applications. To get a clear picture, they gathered a massive collection of over one million comments from one thousand different open-source projects. From this vast pool, they randomly selected fifty thousand comments to examine by hand. This manual review was painstaking work, requiring the researchers to read each note and decide if it represented a genuine admission of a problem or just a standard explanation. After filtering out comments that were not relevant or came from a single project that skewed the data, they identified 615 comments that were true examples of technical debt in test code.

The researchers discovered that the nature of these debts in test code is quite different from what is found in the main application code. They sorted the 615 instances into eleven distinct categories. Some of these were familiar, such as notes about poor design or missing documentation. However, four categories were entirely new and specific to the world of testing. These included "limited tests," where a developer admits the test only checks a tiny, unrepresentative slice of the problem; "skip tests," where a test is explicitly turned off because it cannot run in the current environment; "on-hold," where a test waits for an external tool or service to become available; and "uncertainty," where the developer is unsure if the test is even correct. This taxonomy revealed that test code carries its own unique burdens, often related to the specific challenges of validating software behavior rather than building it.

Having mapped out what these debts look like, the team asked a second, more practical question: Can computers find them automatically? They tested seven existing tools designed to spot these notes in regular source code. They also tested a range of artificial intelligence models, including both open-source models and powerful, proprietary systems from major technology companies. The results were surprising. The existing tools, which rely on looking for specific keywords like "TODO" or "FIXME," performed the best among the traditional methods, but they still missed more than a third of the actual debts. They were good at being correct when they did find something, but they failed to find many of the real issues.

The artificial intelligence models fared even worse in different ways. The open-source models struggled to find the debts at all, often failing to recognize them unless the notes contained very obvious keywords. When they did find something, they were frequently wrong. The proprietary models, which are generally considered more advanced, showed the opposite problem. They found almost every single debt, but they also flagged hundreds of harmless comments as problems. They were so eager to find issues that they mistook routine explanations for admissions of failure. In the end, neither the traditional tools nor the most advanced AI systems could reliably detect these debts in test code.

The study concludes that the way developers write about problems in test code is fundamentally different from how they write about problems in the main code. The notes in test files often use language specific to the testing process, such as mentioning that a test is "disabled" or "skipped," which standard tools and AI models do not recognize as a sign of debt. The researchers found that current methods are not yet ready to handle this complexity. They have created a new dataset and a detailed map of these debt types to help future researchers build better detection tools. Until then, the task of finding and fixing these hidden flaws in test code remains a job that requires human attention.