Technical Summary: Specification and Detection of LLM Code Smells

Problem Statement

While Large Language Models (LLMs) are increasingly integrated into software systems, their improper integration at the source code level can undermine system quality, reliability, and maintainability. Although prior research has established taxonomies for general code smells, machine learning-specific defects, and prompt-related issues, there is no formal catalog addressing code-specific poor practices unique to LLM inference integration. Existing works often focus on high-level symptoms, runtime failures, or prompt engineering rather than recurrent, low-level coding patterns that degrade software quality without necessarily causing immediate bugs.

Key Contributions

This paper introduces the concept of LLM code smells and provides three primary contributions:

1. A Catalog of Five LLM Code Smells: A formal specification of five recurrent problematic coding practices related to LLM inference, derived from academic literature, grey literature (official documentation, blogs), and empirical data.
2. SpecDetect4LLM: A dedicated static detection tool extending the existing SpecDetect4AI framework to identify these specific smells using Domain Specific Language (DSL) rules and semantic predicates.
3. Empirical Validation: A study assessing the prevalence and detection precision of these smells across a dataset of 200 open-source Python systems.

The Catalog of LLM Code Smells

The authors define five specific code smells, each characterized by a context, a problem, and a solution:

1. Unbounded Max Metrics (UMM): Occurs when developers fail to set explicit limits on token budgets, timeouts, or retry counts. This leads to system robustness issues, such as indefinite hanging requests, throttling, or inflated costs, and hinders maintainability due to reliance on changing defaults.
2. No Model Version Pinning (NMVP): Using moving aliases (e.g., gpt-4o) instead of immutable version snapshots (e.g., gpt-4o-2024-11-20). This erodes traceability and reproducibility, as model weights and behaviors can shift silently without notice.
3. No System Message (NSM): Omitting a system message in role-based chat APIs. This results in generic, less consistent outputs and reduces adherence to constraints, often requiring additional iterations to achieve desired results.
4. No Structured Output (NSO): Relying on free-form text generation when the system expects structured data (e.g., JSON). This introduces schema drift, type mismatches, and parsing errors, degrading reliability and downstream data integrity.
5. LLM Temperature Not Explicitly Set (TNES): Relying on implicit temperature defaults. Since defaults vary across providers and may change over time, this harms reproducibility and portability, and can silently alter model behavior.

Methodology

The research followed a three-step methodology:

1. Catalog Construction: The authors synthesized evidence from academic literature, grey literature (provider docs), and empirical artifacts (GitHub, Stack Overflow). They employed a systematic mapping process involving search, screening, synthesis, and triangulation to ensure each smell was observable at the code level, supported by multiple sources, and had actionable remediation.
2. Detection Approach: The team extended SpecDetect4AI, an open-source static analyzer, to create SpecDetect4LLM. This tool uses semantic predicates to detect coding patterns without direct Abstract Syntax Tree (AST) manipulation, allowing for high-level detection of LLM integration idioms.
3. Prevalence Assessment: The tool was executed on a curated dataset of 200 open-source Python systems (combining 100 newly crawled projects with 100 from prior work by Shao et al.). The study focused on measuring prevalence and detection precision via a stratified manual audit, rather than estimating recall, which would require exhaustive ground truthing.

Results

The empirical study yielded the following findings:

• Prevalence: LLM code smells were detected in 60.50% (121 out of 200) of the analyzed systems.
• Individual Smell Prevalence:
  • No Structured Output (NSO): 40.50% (81/200)
  • No Model Version Pinning (NMVP): 36.00% (72/200)
  • LLM Temperature Not Explicitly Set (TNES): 36.50% (73/200)
  • Unbounded Max Metrics (UMM): 38.00% (76/200)
  • No System Message (NSM): 34.50% (69/200)
• Detection Precision: The manual audit confirmed an average detection precision of 86.06% across all smells.
• Observations:
  • NMVP generated the highest volume of alerts (2,472) due to the repetition of moving aliases within single projects.
  • NSO and UMM, while generating fewer raw alerts, affected the largest share of distinct projects, suggesting these engineering-effort smells are pervasive but perhaps harder to implement.
  • TNES was both frequent and widespread, reflecting a common reliance on framework defaults during prototyping.

Significance and Claims

The paper claims that establishing a formal catalog of LLM code smells is essential for the reliability and maintainability of systems integrating LLMs. By moving beyond high-level defect taxonomies to source-code-level patterns, the authors provide a foundation for:

• Coding Guidelines: Offering concrete, actionable recommendations for developers to improve code quality.
• Tooling Support: Demonstrating that static analysis can effectively detect these specific integration issues with high precision.
• Governance: Highlighting that configuration and governance issues (like version pinning and temperature settings) often propagate as habits within projects, while engineering-effort smells (like structured output) require cross-cutting refactoring.

The authors position their work as a necessary step toward supporting practitioners in adopting best practices for LLM integration, laying the groundwork for future research into dynamic detection and the quantification of the impact of these smells on system performance and robustness.