💻 computer science

A Survey of Algorithm Debt in Machine and Deep Learning Systems: Definition, Smells, and Future Work

This paper presents a survey of 42 studies that expands the definition of Algorithm Debt in machine and deep learning systems, identifies its implicit presence and associated "smells," and outlines future research directions to improve system reliability.

Original authors: Emmanuel Iko-Ojo Simon, Chirath Hettiarachchi, Fatemeh Fard, Alex Potanin, Hanna Suominen

Published 2026-04-09

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Emmanuel Iko-Ojo Simon, Chirath Hettiarachchi, Fatemeh Fard, Alex Potanin, Hanna Suominen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are building a house. You have a tight deadline and a limited budget. To get the roof on quickly, you decide to use a cheap, flimsy ladder instead of a sturdy scaffolding, and you skip the step of reinforcing the foundation because "it looks fine for now."

You get the house finished on time! But six months later, the roof starts sagging, the ladder breaks when you try to paint the gutters, and the whole structure is harder to expand than it should be. You didn't just make a mistake; you took out a loan on your future comfort. You borrowed time today, but you are paying it back with interest tomorrow.

In the world of software, this is called Technical Debt.

This paper is about a specific, sneaky type of debt that happens when we build Machine Learning (ML) and Deep Learning (DL) systems (like the AI chatbots, self-driving cars, and recommendation engines we use every day). The authors call this Algorithm Debt.

Here is the breakdown of the paper in simple terms:

1. What is Algorithm Debt?

Think of an AI system as a very complex recipe.

Normal Software Debt is like writing the recipe in messy handwriting or using a confusing font. It's hard to read, but the cake still tastes okay.
Algorithm Debt is like using a bad cooking technique. Maybe you decided to chop the vegetables by hand instead of using a food processor because it was faster to start, or you used a recipe that works for a small family but will take 10 hours to cook for a banquet.

The recipe works right now, but it's inefficient. As soon as you try to feed a million people (scale up) or change the ingredients slightly (new data), the whole kitchen breaks down. The "debt" is the extra time, money, and computer power you have to spend later to fix that bad cooking technique.

2. The Big Discovery: It's Hiding Everywhere!

The authors went on a treasure hunt through 42 different research papers to find out how much of this debt exists.

The Surprise: They found that while only a few papers explicitly said, "Hey, this is Algorithm Debt," about 83% of the papers were actually talking about it without using the name!
The Metaphor: It's like a doctor noticing that patients are getting sick with "fevers" and "coughs" for years, but no one realized they all had the same underlying virus. Researchers were fixing the symptoms (slow computers, bad predictions) without realizing they were all caused by the same root problem: Algorithm Debt.

3. The Three "Smells" of Algorithm Debt

Just like a bad smell in a kitchen tells you something is rotting, the authors found nine specific "smells" (warning signs) that tell you Algorithm Debt is building up. They grouped them into three main categories:

A. The "Bad Ingredients" (Data Issues)

Plain Old Data: Using raw, messy data without cleaning it first. It's like trying to bake a cake with flour that still has the paper bag inside. The computer has to work extra hard to figure out what's what.
Unstable Dependencies: Relying on data that changes constantly (like weather reports) without a plan for when the data shifts. It's like building a house on a riverbank without checking the flood levels.

B. The "Bad Cooking Techniques" (Algorithm Design)

Bias with Batch Norm: A fancy math trick used to speed up training that, if done wrong, makes the AI learn the wrong patterns. It's like using a thermometer that is calibrated to the wrong temperature; your oven thinks it's 350°F when it's actually 200°F.
No Scaling: Forgetting to adjust the size of your ingredients. If you mix a cup of salt with a drop of water, the water dominates. In AI, if one number is huge and another is tiny, the AI gets confused.
Hyperparameters: These are the "knobs" you turn to tune the AI. Setting them to "default" without thinking is like driving a car with the gas pedal taped to the floor because you didn't want to adjust the seat.

C. The "Inexperienced Chef" (Lack of Expertise)

Inadequate ML Expertise: Sometimes, the debt happens because the person writing the code just doesn't know the best way to do it. They pick a sledgehammer to crack a nut because they don't know how to use a nutcracker. This leads to slow, clunky code that is hard to fix later.

4. Why Does This Matter?

If you ignore Algorithm Debt, your AI system will eventually become:

Too Slow: It will take forever to make decisions.
Too Expensive: It will burn through massive amounts of electricity and computer power.
Unreliable: It will start making mistakes when the real world changes (like a self-driving car failing in the rain because it was only trained on sunny days).

5. The Solution: A New Roadmap

The authors aren't just pointing out the problem; they are offering a plan to fix it:

Name it: Stop calling it "slow code" or "bad data." Call it Algorithm Debt so we know exactly what we are fighting.
Check the Smells: Use the nine "smells" as a checklist before you launch a project.
Build Tools: We need new software tools (like a "spell-checker" for AI algorithms) that can automatically spot these bad cooking techniques before they become a disaster.
Train the Chefs: We need to teach developers not just how to code, but how to understand the specific math and logic of AI so they don't make these mistakes in the first place.

The Bottom Line

Building AI is like building a skyscraper. You can't just throw up walls and hope it stands. If you cut corners on the algorithms (the blueprints and engineering logic), you might save a few days today, but you'll be paying for it with a collapsing building tomorrow. This paper is a wake-up call to stop borrowing against our future and start building AI systems that are efficient, scalable, and ready for the long haul.

1. Problem Statement

The rapid adoption of Machine Learning (ML) and Deep Learning (DL) has introduced unique maintenance challenges, leading to Technical Debt (TD). While TD is well-studied in traditional software, ML/DL systems introduce specific complexities related to data dependencies, algorithmic logic, and continuous model retraining.

A specific type of TD, Algorithm Debt (AD), has emerged but remains underexplored and often conflated with other debt types (e.g., Model Debt, Data Debt, or general performance issues).

The Gap: There is a lack of a consolidated definition for AD. Researchers and practitioners often fail to distinguish between algorithmic inefficiencies (AD) and other issues like poor data quality or suboptimal model architecture.
The Consequence: Without a clear definition and taxonomy, mitigation strategies are often misdirected (e.g., refactoring code structure when the root cause is an inefficient algorithm), leading to persistent performance degradation, scalability bottlenecks, and increased computational costs.

2. Methodology

The authors conducted a Systematic Literature Review (SLR) using a hybrid search protocol to investigate AD in ML/DL systems.

Search Strategy:
- Databases: IEEE Xplore, ACM Digital Library, Springer, and ScienceDirect.
- Supplementary: Google Scholar for snowballing (forward and backward citation tracking).
- Search String: Combined terms for TD (e.g., "Technical Debt," "Code Smells," "Refactor") and ML/DL (e.g., "Deep Learning," "Algorithm Debt"), specifically including "Hidden Technical Debt" to capture seminal works.
Selection Process:
- Initial Pool: 710 papers retrieved.
- Filtering: After removing duplicates and applying inclusion/exclusion criteria (e.g., English language, post-2010, focus on ML/DL systems), 35 primary studies were selected.
- Control Papers: 7 "control papers" (including the seminal work by Liu et al. [49] that first defined AD) were used to validate the search string.
- Final Corpus: 42 papers (35 primary + 7 control).
Data Sources:
- Literature: 42 academic papers.
- Dataset: A supplementary dataset of 7,159 Self-Admitted Technical Debt (SATD) comments explicitly labeled as "Algorithm Debt" from Liu et al. [49], used to ground the definition in real-world developer code.
Analysis:
- Thematic Analysis: Used to extract definitions, identify smells, and synthesize categories.
- Expert Validation: A TD expert reviewed all coding and categorization to ensure rigor, as the field is nascent and requires interpretive judgment.

3. Key Contributions

A. Consolidated Definition of Algorithm Debt (AD)

The paper moves beyond the initial definition (which focused solely on performance) to a holistic definition grounded in both literature and empirical code comments.

Definition: AD is a TD type arising from suboptimal implementation of algorithmic choices that leads to limited scalability and model degradation relative to system goals.
Three Key Manifestations:
1. Inefficiency: Redundant computations or non-optimized logic increasing time/memory usage.
2. Scalability Limitations: Algorithms that fail to handle large datasets, distributed execution, or complex models due to design constraints.
3. Model Degradation: Performance decline due to numerical instability, overfitting, concept drift, or data integrity loss caused by algorithmic flaws.

B. Distinction from Other TD Types

The authors clarify how AD differs from related concepts:

vs. Model Debt: Model Debt concerns architecture/selection (e.g., choosing the wrong model type). AD concerns the implementation logic of the chosen model (e.g., inefficient training loops).
vs. Data Debt: Data Debt concerns data quality/labeling. AD concerns how the algorithm processes that data (e.g., unstable dependencies).
vs. Performance Issues: AD is a conscious trade-off made by developers (e.g., "TODO: optimize this later") for short-term gains, whereas general performance issues may be unintentional bugs.

C. Taxonomy of Nine AD Smells

The study identifies nine specific "smells" (indicators) of AD, categorized into three root causes:

Data-Related Smells:
- Plain old data: Unformatted raw data requiring complex inference logic.
- Non-representative statistics: Biased metrics used for training.
- Unstable data dependencies: Inputs that shift over time (covariate shift).
- Out-of-domain data: Models failing on data distributions different from training.
- No scaling before sensitive operations: Failure to normalize inputs before critical steps.
- Unnormalized features: Features with extreme variance affecting convergence.
Algorithm Design Smells:
- Bias with batch norm: Incorrect application of normalization techniques (e.g., post-dropout).
- Hyperparameters: Suboptimal or default configuration settings.
Human/Expertise Smells:
- Inadequate ML expertise: Developers lacking domain knowledge, leading to poor algorithmic choices.

D. Conceptual Framework

The authors propose a framework linking Causes (Data, Design, Expertise) $\rightarrow$ Smells $\rightarrow$ Effects (Inefficiency, Scalability, Degradation) $\rightarrow$ Mitigation (Feature engineering, retraining, peer review, automated tooling, education).

4. Key Results

Prevalence of Implicit Discussion: A striking finding is that 83% (35/42) of the reviewed papers discussed AD-related issues implicitly without using the term "Algorithm Debt." These issues were often misclassified under "Model Debt," "Data Debt," or generic "Performance Issues."
Historical Context: AD-related problems existed prior to the formal definition of AD in 2020 (Liu et al.), suggesting a long-standing gap in recognizing algorithmic inefficiencies as a distinct debt category.
Developer Awareness: Analysis of SATD comments reveals that developers are often aware of these inefficiencies (e.g., comments like "This is inefficient but fine for now") but defer optimization due to time pressure or complexity, explicitly incurring debt.
Root Causes: The analysis confirms that AD is not just a technical issue but a socio-technical one, driven by a combination of poor data management, flawed algorithmic design, and gaps in developer expertise.

5. Significance and Future Work

Theoretical Impact: This survey establishes AD as a distinct, actionable research area within Software Engineering and ML. It provides the first consolidated definition and taxonomy, enabling researchers to differentiate AD from other TD types.
Practical Impact:
- For Practitioners: Offers a checklist of "smells" to audit ML/DL systems. It suggests that fixing AD requires more than just code refactoring; it requires algorithmic redesign, better feature engineering, and upskilling.
- For MLOps: Highlights the need to integrate AD detection into MLOps pipelines (e.g., automated checks for unstable dependencies or hyperparameter sanity).
Research Roadmap: The authors propose a 5-phase roadmap:
1. Formalize boundaries between AD and other TD types.
2. Empirically validate the 9 smells in real-world industry projects.
3. Develop LLM-based automated tools to detect AD smells in code and config files.
4. Conduct longitudinal studies to track AD accumulation over time.
5. Extend the framework to non-ML domains (e.g., game engines, HPC).

In conclusion, the paper argues that Algorithm Debt is a pervasive, often invisible, but critical barrier to the reliability and scalability of ML/DL systems. Recognizing it as a distinct entity is the first step toward developing targeted mitigation strategies and tools.