💻 computer science

Defective Task Descriptions in LLM-Based Code Generation: Detection and Analysis

This paper introduces SpecValidator, a lightweight, finetuned classifier that effectively detects defective task descriptions (such as lexical vagueness, under-specification, and syntax errors) to improve LLM-based code generation, outperforming larger models and revealing that defect robustness depends more on description quality and type than model capacity.

Original authors: Amal Akli, Mike Papadakis, Maxime Cordy, Yves Le Traon

Published 2026-04-28

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Amal Akli, Mike Papadakis, Maxime Cordy, Yves Le Traon

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a brilliant but literal-minded chef (the AI) to cook a specific dish based on a recipe you wrote (the task description). If your recipe says, "Add some spice," the chef might guess wrong. If it says, "Add 2 grams of salt," the chef knows exactly what to do.

This paper, "Defective Task Descriptions in LLM-Based Code Generation," investigates what happens when the "recipes" (prompts) given to AI coding assistants are vague, incomplete, or messy. The researchers found that even the smartest AI chefs can fail spectacularly if the instructions aren't perfect, and they built a tool to spot these bad instructions before the cooking starts.

Here is a breakdown of their findings using simple analogies:

1. The Problem: Garbage In, Garbage Out

The researchers discovered that AI coding tools are incredibly sensitive to how a task is described. They identified three main ways a "recipe" can go wrong:

Lexical Vagueness (The "Use Your Best Judgment" Problem): This is like telling the chef to "add a little bit of sugar" instead of "add 10 grams." The AI has to guess the amount. The study found this causes a moderate drop in performance. It's annoying, but the AI can often still figure it out, especially if the recipe is short and informal.
Under-Specification (The "Missing Ingredient" Problem): This is the worst offender. It's like telling the chef to "bake a cake" but forgetting to say what kind of cake, how long to bake it, or what temperature. The AI is left guessing critical details. The study found this causes massive failures (up to a 15% drop in success rate). Even the most advanced AI models couldn't handle this; they just guessed wrong.
Syntax and Formatting (The "Typo" Problem): This is like writing "bake at 350 degrees" but accidentally typing "bake at 350 degreess" or messing up the capitalization. Surprisingly, the AI is very good at ignoring these. It's like a human chef who can read a messy, handwritten note and still bake the cake perfectly. Typos barely affected the results.

2. The Surprise: Bigger Isn't Always Better

You might think a super-smart, massive AI chef (a large model) would be better at handling vague instructions than a smaller one. The researchers tested this and found it doesn't matter.

Whether the AI was a small, resource-friendly model or a massive, state-of-the-art reasoning model, they all failed equally when the instructions were incomplete. The size of the AI's brain didn't help; the clarity of the instructions was the only thing that mattered.

However, there was one exception: LiveCodeBench. This is a benchmark where the "recipes" are very detailed, including specific examples of inputs and outputs (like showing the chef a picture of the finished cake alongside the instructions). Because the context was so rich, these AI chefs were much more resilient to bad instructions. It proved that structure and examples save the day.

3. The Solution: The "Quality Inspector" (SpecValidator)

Since the AI chefs can't fix bad recipes themselves, the researchers built a tool called SpecValidator. Think of this as a Quality Inspector who looks at the recipe before the chef starts cooking.

How it works: It's a small, lightweight AI trained specifically to spot the three types of bad instructions (Vague, Missing Info, or Typos).
How good is it? It is surprisingly effective. It caught defects with an accuracy score (F1) of 0.804.
The Comparison: The researchers tested this small inspector against the "giants" (GPT-5-mini and Claude Sonnet 4). The giants, despite being huge and powerful, performed poorly at spotting these defects (scoring around 0.46–0.51). The small, specialized inspector beat them by a wide margin.

4. The "Real World" Test

The most interesting part of the study was testing SpecValidator on the original benchmarks (the "clean" recipes) that were supposed to be perfect.

The inspector found that 18% of the "perfect" recipes were actually defective.

When the researchers manually checked the ones flagged as "Under-Specified" (missing info), they confirmed that 73% of them were indeed missing critical details.
When they tried to use these "clean" but actually broken recipes with the AI chefs, the chefs failed almost every time.

This suggests that many of the standard tests we use to measure AI coding ability might be flawed because the instructions themselves are secretly broken.

Summary

Bad instructions kill performance: Missing details (Under-Specification) are the most dangerous, causing AI to fail even when it's a top-tier model.
Size doesn't save you: A bigger AI brain doesn't make up for a vague recipe.
Typos don't matter: AI is surprisingly good at ignoring small formatting errors.
Context is king: Benchmarks that provide clear examples (like LiveCodeBench) are much more robust.
We need a filter: A small, specialized tool (SpecValidator) is better at spotting bad instructions than the giant AI models themselves.
The benchmarks might be broken: Even the "gold standard" tests used to evaluate AI contain hidden defects that cause the AI to fail, which we didn't know until this tool found them.

The paper concludes that to get reliable code from AI, we need to treat the description of the task as a critical part of the process, not just an afterthought. We need to check the recipe before we let the chef cook.

1. Problem Statement

Large Language Models (LLMs) are increasingly used for code generation, but their performance relies heavily on the assumption that user-provided task descriptions are sufficiently detailed and well-formed. In practice, natural language prompts often contain defects such as:

Lexical Vagueness (LV): Imprecise terminology or synonyms that reduce specificity.
Under-Specification (US): Missing constraints, edge cases, or requirements (e.g., omitting numerical bounds).
Syntax and Formatting (SF): Surface-level noise like typos, formatting errors, or whitespace issues.

These defects can drastically degrade code correctness. Current evaluation benchmarks often use "clean" prompts, failing to reflect real-world scenarios where descriptions are imperfect. Furthermore, there is a lack of automated tools to detect these defects before code generation, and it is unclear how different model sizes or architectures handle such defects.

2. Methodology

A. Dataset Construction

The authors constructed a large-scale dataset of 6,573 defective task descriptions derived from three established benchmarks:

HumanEval: Structured docstrings with typed signatures.
MBPP: Short, informal natural language descriptions.
LiveCodeBench: Competitive programming problems with explicit I/O contracts and examples.

Defect Injection Strategy:
Using gpt-5-mini, the authors systematically mutated original descriptions to create three defect types per task:

LV: Replaced specific terms with broader synonyms (e.g., "sort" $\to$ "arrange") or renamed identifiers.
US: Removed exactly one explicit constraint (e.g., a numeric bound, sorting rule, or error handling requirement).
SF: Introduced typos, delimiter errors, or indentation issues without changing semantics.

Validation:

Automated: An independent LLM judge (Qwen2.5-Coder-32B) scored mutations for compliance and naturalness.
Manual: Three researchers manually verified a stratified random sample (100 instances), confirming high agreement with the automated judge (97% compliance, 86% naturalness).

B. Experimental Setup

Models Evaluated: 10 LLMs ranging from small open-source models (6–7B parameters) to large open-source models (15–34B) and state-of-the-art proprietary reasoning models (GPT-5-mini, Claude Sonnet 4).
Metrics:
- Code Correctness: Pass@1 (percentage of times the first generated solution passes unit tests).
- Defect Detection: Macro-F1, Matthews Correlation Coefficient (MCC), Precision, and Recall.

C. Proposed Solution: SpecValidator

The authors developed SpecValidator, a lightweight classifier designed to detect task description defects.

Architecture: Based on Qwen2.5-Coder-1.5B.
Training Regimes: Compared three approaches:
1. Linear Probing: Training only the classification head on frozen embeddings.
2. LoRA Fine-tuning: Low-rank adaptation (rank $r=16$ ) on attention layers.
3. Full Fine-tuning: Updating all parameters.
Baselines: Compared against zero-shot and few-shot prompting of GPT-5-mini and Claude Sonnet 4.

3. Key Contributions

Large-Scale Analysis: A comprehensive study of 10 models across 3 benchmarks, revealing that defect impact is driven by defect type and benchmark structure rather than model capacity.
SpecValidator: A highly efficient, LoRA-fine-tuned classifier (1.5B model) that outperforms frontier models in defect detection while updating only 0.14% of parameters.
Real-World Defect Discovery: Demonstrated that SpecValidator can generalize to unseen, real-world benchmark descriptions, successfully identifying latent "Under-Specification" defects in the ground truth of standard benchmarks (HumanEval, MBPP, LiveCodeBench).
Public Dataset: Release of a dataset containing 6,573 defective task description instances to support future research.

4. Key Results

A. Impact of Defects on Code Generation (RQ1)

Under-Specification (US) is the most severe: It caused Pass@1 drops of 7.9% to 15.3% across all models and benchmarks. Even state-of-the-art models (GPT-5-mini, Claude Sonnet 4) suffered ~9.7% drops. No model was immune.
Lexical Vagueness (LV) is moderate: Impact varied by benchmark (up to 12.2% drop on HumanEval, but only ~1.5–3.2% on MBPP). Larger models (e.g., Qwen2.5-32B) showed better resilience to LV.
Syntax/Formatting (SF) is negligible: Surface noise had minimal impact (<3.7% drop), suggesting code-specialized LLMs are robust to typos.
Benchmark Resilience: LiveCodeBench was nearly immune to all defects (deltas within ±2.0%). The authors attribute this to its rich contextual grounding (explicit I/O examples), which allows models to infer missing constraints.
Model Size Irrelevance: Larger model size did not correlate with better robustness. Sensitivity was determined by the description structure, not the model's parameter count.

B. Defect Detection Performance (RQ2)

SpecValidator (LoRA) achieved the best performance: F1 = 0.804, MCC = 0.745.
Baselines: GPT-5-mini (Zero-shot) and Claude Sonnet 4 performed poorly (F1 $\approx$ 0.47–0.52), indicating that strong code generation capabilities do not transfer to meta-reasoning about prompt quality in a zero-shot setting.
Efficiency: The LoRA approach (0.14% trainable parameters) outperformed Full Fine-tuning (100% parameters), likely due to overfitting in the latter.
Error Mode: The primary confusion was between Clean and Under-Specified classes, as US defects leave no positive textual trace (they are grammatically correct but missing information).

C. Generalization to Real Prompts (RQ3)

When applied to the original "clean" descriptions of the benchmarks, SpecValidator flagged 83 out of 456 (18.2%) as defective (specifically Under-Specified).
Validation:
- These flagged descriptions resulted in near-zero Pass@1 across all 10 models (90.9% failure rate on MBPP, 98.0% on LiveCodeBench).
- Manual inspection of 33 flagged MBPP cases confirmed 72.7% were genuinely under-specified.
Conclusion: Standard benchmarks contain latent defects that significantly hinder code generation, and SpecValidator can detect them.

5. Significance and Implications

Redefining Robustness: The study challenges the notion that larger models are inherently more robust. Instead, robustness is a function of the task description structure (e.g., presence of I/O examples).
Benchmark Quality: The discovery of latent defects in major benchmarks (like MBPP and LiveCodeBench) suggests that current evaluation metrics may be inflated or misleading if they do not account for prompt quality.
Practical Tooling: SpecValidator offers a computationally cheap, highly accurate method to audit prompts before code generation, preventing wasted compute and failed deployments.
Future Directions: The paper argues for moving beyond simple perturbation studies to structured analysis of specification defects and emphasizes the need for benchmarks with richer contextual grounding to ensure reliable LLM-based code generation.