🤖 AI

Bridging Generation and Training: A Systematic Review of Quality Issues in LLMs for Code

This paper presents a systematic review of 114 studies to establish a unified taxonomy and causal framework linking training data quality issues to defective code generation in LLMs, while synthesizing mitigation techniques and outlining a paradigm shift toward proactive, data-centric governance for reliable code models.

Original authors: Kaifeng He, Xiaojun Zhang, Peiliang Cai, Mingwei Liu, Yanlin Wang, Chong Wang, Kaifeng Huang, Bihuan Chen, Xin Peng, Zibin Zheng

Published 2026-05-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kaifeng He, Xiaojun Zhang, Peiliang Cai, Mingwei Liu, Yanlin Wang, Chong Wang, Kaifeng Huang, Bihuan Chen, Xin Peng, Zibin Zheng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a brilliant but very literal student how to write computer code. You give them a massive library of books (the training data) to study. The student reads everything, memorizes patterns, and then tries to write new code on their own.

This paper is a massive investigation into why that student sometimes writes code that is buggy, insecure, or just plain wrong. The authors, a team of researchers, looked at 114 different studies to answer a simple but crucial question: Is the student's bad writing their own fault, or is it because the books they were studying were full of mistakes?

Their conclusion is striking: It's mostly the books.

Here is the breakdown of their findings, explained through simple analogies:

1. The "Garbage In, Garbage Out" Reality

The paper argues that when an AI (Large Language Model) generates bad code, it's rarely because the AI is "thinking" poorly. Instead, it's because the AI is simply copying what it saw in its training data.

The Analogy: Imagine you are learning to cook by reading a cookbook. If the cookbook has a recipe that says, "Add a cup of salt to the soup," and you follow it, your soup will be inedible. You didn't make a mistake; the book did.
The Paper's Claim: The AI learns from a "cookbook" (training data) that contains millions of lines of code from the internet. If that internet code has security holes, outdated instructions, or typos, the AI will learn those mistakes as if they were the correct way to do things.

2. The Two Types of "Bad Books"

The researchers organized the problems in the training data into two main categories:

The "Bad Recipe" Problems (Code Attributes): These are specific mistakes inside the code snippets themselves.
- Example: A recipe that uses a tool that was banned 10 years ago. The AI learns to use that banned tool because it saw it in the book.
- Example: A recipe that forgets to wash the chicken, leading to a sick meal (a security vulnerability).
The "Bad Library" Problems (Non-Code Attributes): These are problems with the library as a whole, not just individual recipes.
- Example: The library has 1,000 copies of the same boring recipe but only one copy of a complex, healthy dish. The AI becomes an expert at the boring dish but fails completely at the complex one.
- Example: The library is filled with pages of random gibberish or ads (noise) that distract the student from learning real cooking.

3. How Mistakes Travel (The Pipeline)

The paper maps out exactly how a mistake in the "book" becomes a mistake in the "final dish." They found 18 different ways this happens, which they group into two paths:

Direct Copying (Direct Mappings): The AI sees a specific bad pattern and repeats it.
- Analogy: The student sees a typo in the book and types the same typo in their essay.
- Real-world result: The AI uses a deprecated (old) computer command that no longer works, causing the program to crash.
Distorted Learning (Indirect Mappings): The AI learns the wrong balance of things because the library was unbalanced.
- Analogy: Because the library had 99% of recipes for "burning toast" and only 1% for "making bread," the AI thinks burning toast is the only way to cook. It doesn't know how to make good bread because it never saw enough examples.
- Real-world result: The AI is great at writing simple Python code but terrible at writing code for other languages or complex tasks because the training data was unbalanced.

4. The Current Fix vs. The Better Fix

The paper points out that most companies are currently trying to fix this problem after the AI writes the code. They act like a food inspector who tastes the soup at the end and says, "This is too salty, throw it away."

The Problem: This is slow, expensive, and reactive. You keep throwing away bad soup.
The Paper's Recommendation: We need to become librarians and editors before the student starts studying.
- Instead of just filtering bad soup at the end, we need to go into the library, remove the bad recipes, fix the typos, and ensure there is a good mix of different types of dishes.
- The authors call this a shift from "reactive filtering" to "proactive data governance."

5. What's Still Hard?

Even with this new understanding, the paper admits there are big hurdles:

The "Leaky" Library: Sometimes the "books" the AI studies contain the answers to the tests it will take later. This makes the AI look smarter than it really is because it just memorized the answers.
The "Black Box" Library: We often don't know exactly which books were used to train the most popular AIs. Without knowing the source, it's hard to trace why a specific mistake happened.
The Moving Target: The world of software changes fast. A "good" recipe today might be a "bad" one tomorrow because technology updates. The library needs to be constantly updated, which is very difficult.

Summary

The paper concludes that to get reliable code from AI, we can't just blame the AI or try to fix its output after the fact. We have to fix the source material. If we want the AI to write perfect code, we must first ensure the "library" it learned from is clean, balanced, and up-to-date. The quality of the output is a direct reflection of the quality of the input.

Technical Summary: Bridging Generation and Training: A Systematic Review of Quality Issues in LLMs for Code

Problem Statement

Large Language Models (LLMs) have transformed software engineering by automating tasks such as code completion, repair, and test generation. However, their practical adoption is hindered by frequent generation of defective code, ranging from logical bugs and syntax errors to critical security vulnerabilities and compliance violations. While these failures are often attributed to inherent model limitations, empirical evidence increasingly suggests that their root causes lie within imperfections in the training corpora.

A significant gap exists in the current literature: the specific mechanisms linking training data quality issues to generated code quality issues remain largely unmapped. Existing research suffers from three critical deficiencies:

Fragmented Taxonomies: Studies focus on isolated quality dimensions (e.g., security or correctness) without a unified framework to categorize both generated code defects and training data flaws.
Unclear Causal Mappings: There is a lack of formalized mechanisms explaining how specific dataset flaws (e.g., deprecated APIs, data imbalance) propagate into code defects.
Disjoint Mitigation Strategies: Detection and mitigation techniques are scattered across data curation, model training, and post-generation phases, lacking an integrated lifecycle approach.

Methodology

This paper presents a Systematic Literature Review (SLR) of 114 primary studies published between late 2021 and early 2026. The review followed established software engineering guidelines, involving:

Search Strategy: A multi-stage process combining Boolean keyword searches across five major digital libraries (IEEE Xplore, ACM Digital Library, SpringerLink, ScienceDirect, arXiv) with backward and forward snowballing.
Selection Criteria: Studies were included if they investigated LLMs applied to code tasks, focused on specific quality attributes of code or data, and provided empirical grounding.
Quality Assessment: A structured evaluation using eight criteria (e.g., methodological rigor, causal analysis depth, transparency) with a scoring system (0–16). Studies scoring below 10 were excluded.
Synthesis: An iterative qualitative coding approach (Open, Axial, and Selective coding) was used to extract data and synthesize findings into unified taxonomies and causal frameworks.

Key Contributions

1. Unified Dual-Dimensional Taxonomy

The authors propose a comprehensive taxonomy that categorizes quality issues across two dimensions:

Generated Code Quality Issues (9 Dimensions):
1. Correctness: Syntax errors, logical flaws, and improper API/dependency usage.
2. Security: External-triggered vulnerabilities (e.g., SQL injection) and inherent vulnerabilities (e.g., hardcoded keys).
3. Compliance: Violations of legal/ethical constraints, including malicious code, privacy leaks, and copyright infringement.
4. Robustness: Inability to handle abnormal inputs or boundary conditions.
5. Maintainability: Structural complexity, poor modularity, and lack of reusability.
6. Understandability: Poor naming conventions, missing comments, and "magic numbers."
7. Efficiency: Suboptimal time/space complexity and resource wastage.
8. Parsimony of Output: Redundant, verbose, or repetitive code generation.
9. Miscellaneous: Issues like poor instruction-following.
Training Data Quality Issues (2 Major Groups):
- Code Attributes: Defects within individual samples (Correctness, Security, Compliance, Robustness, Maintainability, Understandability, Efficiency).
- Non-Code Attributes: Macro-level dataset properties (Compliance/Security risks in text, Distribution Imbalance, Redundancy, Inadequate Diversity, Data Contamination, Low-Value Data).

2. Causal Framework and Propagation Mechanisms

The paper formalizes a causal framework detailing 18 typical propagation mapping mechanisms that link data issues to code defects. These are divided into:

Direct Mappings (10 mechanisms): Explicit replication of code attribute defects. Examples include:
- Deprecated APIs $\rightarrow$ Compatibility Failures: Memorization of obsolete library calls.
- Syntax Defects $\rightarrow$ Compilation Errors: Propagation of structural invalidity from unverified snippets.
- Insecure Practices $\rightarrow$ Vulnerable Code: Replication of unsafe idioms (e.g., eval(), unvalidated input).
Indirect Mappings (8 mechanisms): Distortion of the learning distribution due to non-code attributes. Examples include:
- Redundancy $\rightarrow$ Reduced Diversity: Overfitting to repetitive patterns due to duplicate samples.
- Distribution Imbalance $\rightarrow$ Biased Performance: Skewed performance across languages or domains due to data scarcity.
- Instructional Misalignment $\rightarrow$ Semantic Drift: Generation of syntactically correct but semantically incorrect code due to mismatched docstrings.

3. Synthesis of Detection and Mitigation Techniques

The review synthesizes state-of-the-art techniques across the LLM lifecycle:

Detection: Moves beyond static rule-checking to hybrid frameworks combining Dynamic Analysis (execution-based testing, runtime monitoring), Static Analysis (rule-based tools, human review), and Model-Based Detection (LLM-as-a-judge, lightweight ML classifiers).
Mitigation: Identifies a shift from reactive post-generation filtering to proactive, data-centric governance. Strategies include:
- Data-Level: Execution-feedback filtering, rule-based sanitization, LLM-based cleaning, and data balancing.
- Model-Level: Fine-tuning, instruction alignment (RLHF/DPO), and reward-based optimization.
- Generation-Level: Prompt engineering, Retrieval-Augmented Generation (RAG), decoding-time interventions, and iterative self-reflection.

Results and Findings

Research Trends: Research on code quality has exploded since 2023, with a clear shift from a singular focus on Correctness (52 studies) to a multi-dimensional view including Security (24) and Maintainability (22).
Root Cause Analysis: The review confirms that generated code defects are rarely isolated reasoning deficits but are deeply rooted in training data quality. "Garbage in, garbage out" applies not just to explicit errors but also to systemic distributional biases (e.g., data imbalance causing poor generalization in low-resource languages).
Methodological Shift: The field is transitioning from reactive, heuristic-based post-generation filtering toward proactive, data-centric governance and closed-loop repair.
Gap Identification: Current research lacks formal causal attribution methods to trace specific generation failures back to specific data origins, and benchmark contamination remains a pervasive threat to evaluation integrity.

Significance and Claims

The paper claims to be the first systematic literature review to explicitly bridge the gap between training data quality and generated code quality. Its significance lies in:

Establishing a Unified Framework: Providing a consistent vocabulary and taxonomy that resolves the fragmentation in existing literature, allowing for systematic comparison of quality issues.
Formalizing Causality: Moving beyond correlation to map specific propagation mechanisms, enabling researchers and practitioners to trace generation failures back to their dataset origins.
Guiding Future Governance: Arguing that reliable LLMs for code cannot be achieved through model architecture improvements alone. Instead, the community must adopt integrated data curation and continuous evaluation strategies that treat data quality as the primary lever for software reliability.

The authors conclude that while "harness engineering" (e.g., RAG, tool use) offers deployment-time compensation, it does not replace the necessity of high-quality training data. Future reliability depends on shifting from disjointed, stage-specific interventions to end-to-end quality pipelines that ensure bidirectional traceability between training corpora and generated artifacts.