When Prompt Under-Specification Improves Code Correctness: An Exploratory Study of Prompt Wording and Structure Effects on LLM-Based Code Generation

1. Problem Statement

While Large Language Models (LLMs) are increasingly used for code generation, the correctness of their output is highly sensitive to prompt wording. Prior research has established that under-specification (missing or weakened requirements) and minor prompt perturbations generally degrade performance. However, these findings are largely derived from minimal-specification benchmarks (e.g., HumanEval, MBPP) which feature short, single-docstring descriptions with limited structural redundancy.

The authors hypothesize that these benchmarks exaggerate the fragility of LLMs. In real-world scenarios, tasks often include rich, multi-layered specifications (descriptions, constraints, I/O formats, examples). The paper investigates whether the negative effects of under-specification persist in structurally rich environments and whether, counter-intuitively, removing specific prompt elements could actually improve code correctness by disrupting misleading retrieval cues.

2. Methodology

Benchmarks

The study contrasts two benchmarks with distinct structural properties:

• HumanEval (HE): 164 simple Python problems with short docstrings. Lacks separate constraints, structured I/O, or examples. Represents "minimal specification."
• LiveCodeBench (LCB): 1,055 competitive programming problems (LeetCode, Codeforces, AtCoder) featuring full prose descriptions, explicit constraints, multiple sample I/O pairs, and format specifications. Represents "rich specification."

Models

The study evaluates 10 LLMs across three categories:

• Small Open-Source (6–7B): CodeLlama-7B, DeepSeek-Coder-6.7B, Qwen2.5-Coder-7B.
• Large Open-Source (15–34B): StarCoder2-15B, Codestral-22B, CodeLlama-34B, DeepSeek-Coder-33B, Qwen2.5-Coder-32B.
• Proprietary/Reasoning: GPT-5-mini, Claude Sonnet 4.

Prompt Mutations

The authors systematically generated 3,651 mutated prompts using three transformation types:

1. Lexical Vagueness (LV): Replacing precise terms/identifiers with broader synonyms (e.g., "sort" $\to$ "arrange", delimiter $\to$ filler).
2. Under-Specification (US): Removing a single explicit constraint (e.g., value bounds, ordering rules, edge-case descriptions) while keeping the prompt grammatically valid.
3. Syntax and Formatting (SF): Introducing surface-level noise (typos, indentation errors, delimiter issues) without altering semantics.

Evaluation Metrics

• Pass@1: The primary metric for correctness.
• Transition Analysis: Tracking Pass$\to$Fail (degradation) and Fail$\to$Pass (improvement) transitions to distinguish between uniform degradation and opposing forces.
• Attention Analysis: Examining model attention patterns to understand which prompt regions drive decisions.
• Root-Cause Analysis: Manual inspection of cases where mutations improved correctness to identify linguistic mechanisms.

3. Key Contributions & Findings

Finding 1: Robustness is Structural, Not Architectural

• HumanEval: All mutation types caused significant performance drops (e.g., US mutations caused an average -11.8% drop).
• LiveCodeBench: The same mutations resulted in near-zero net impact (e.g., US mutations caused only -0.9% drop).
• Conclusion: Robustness is not an intrinsic property of model size or architecture. It is governed by prompt structure. Rich specifications provide redundancy (constraints, examples, I/O) that act as fallback signals when parts of the prompt are degraded. Attention maps confirm models on LCB rely heavily on these auxiliary components, whereas HE models rely almost exclusively on the description.

Finding 2: Under-Specification Can Improve Correctness

Contrary to the assumption that "more specification is always better," the study found that under-specification often improves code correctness on LCB.

• On LCB, the ratio of Fail$\to$Pass to Pass$\to$Fail transitions was nearly 1:1 (0.99) for LV mutations, meaning improvements canceled out degradations.
• On HumanEval, degradations dominated heavily (Ratio 0.40).
• Consistent Improvements: The study identified 47 specific LCB tasks where removing constraints or vagueness consistently improved pass rates across multiple models.

Finding 3: Mechanisms of Improvement (Disruption of Retrieval Cues)

Manual analysis revealed that original prompts often contain "retrieval cues" that anchor models to memorized but incorrect solutions. Under-specification disrupts these cues, forcing the model to reason from the problem structure.

• Lexical Vagueness (LV):
  • Variable Renaming: Breaking over-fitted associations (e.g., renaming s to seq prevents retrieval of a specific wrong algorithm).
  • Terminology Generalization: Replacing domain-specific nouns (e.g., "currency") with generic terms (e.g., "resource") prevents the model from applying domain-specific heuristics (like backward financial sweeps) and encourages general algorithmic reasoning (like forward DP).
• Under-Specification (US):
  • Constraint Removal: Specific constraints (e.g., $C_i \le 10^9$) often prime models to use optimized shortcuts (e.g., star-graph diameter formulas) that fail on general cases. Removing the constraint forces the model to use a general, correct algorithm (e.g., two-BFS).
  • I/O Priming: Constraints sometimes trigger incorrect input parsing patterns (e.g., LeetCode-style input().split() vs. competitive programming stdin formats). Removing them forces correct parsing.

4. Significance and Implications

• Re-evaluating Benchmarks: Minimal benchmarks like HumanEval may overestimate the fragility of LLMs because they lack the redundancy found in real-world tasks. They fail to capture the "cancellation effect" where structural richness mitigates prompt errors.
• Prompt Engineering: The study challenges the dogma that prompts must be maximally precise. In some cases, over-specification (specifically including misleading constraints or domain-specific identifiers) silently misleads models. Practitioners should be aware that removing certain cues can sometimes yield better results.
• Model Development: The sensitivity to retrieval cues indicates a generalization gap that model scale alone does not resolve. Even state-of-the-art reasoning models (GPT-5, Claude) fall prey to these cues, suggesting a need for training strategies that prioritize structural reasoning over pattern matching.
• Future Research: The paper calls for benchmarks that reflect the structural richness of real-world development and encourages investigating the specific linguistic triggers that cause retrieval-based failures.

Summary

This exploratory study demonstrates that the impact of prompt under-specification is not uniform. While it degrades performance on minimal benchmarks, rich specifications in real-world tasks provide redundancy that neutralizes these effects. Surprisingly, under-specification can improve code correctness by disrupting misleading retrieval cues that cause models to rely on memorized, incorrect patterns. The authors conclude that robustness is a function of prompt structure, and that "less is more" in specific contexts where precision triggers overfitting.