💬 NLP

Code Is More Than Text: Uncertainty Estimation for Code Generation

This paper proposes a novel three-axis uncertainty estimation framework for code generation that leverages code-specific properties like token fragility, intent-code gaps, and executability to significantly outperform natural language-derived baselines in detecting unreliable outputs.

Original authors: Yuling Shi, Caiqi Zhang, Yuexian Li, Haopeng Wang, Yeheng Chen, Nigel Collier, Xiaodong Gu

Published 2026-06-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yuling Shi, Caiqi Zhang, Yuexian Li, Haopeng Wang, Yeheng Chen, Nigel Collier, Xiaodong Gu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very talented, but sometimes overconfident, robot assistant who writes computer code for you. Sometimes, it writes perfect code. Other times, it writes code that looks right but has a tiny, invisible mistake that will cause the whole program to crash later.

The big problem is: The robot doesn't always know when it's making a mistake. It might say, "I'm 100% sure this is correct!" when it's actually wrong. This is dangerous because if you trust a wrong program, it could break your software or cause safety issues.

This paper introduces a new way to ask the robot: "How sure are you, really?"

The authors argue that asking a robot about code is different from asking it about writing a story. You can't just use the same "confidence meter" you use for text. They found three special reasons why code is unique, and they built a three-part "uncertainty detector" based on these reasons.

Here is how their three-part detector works, using simple analogies:

1. The "One Wrong Brick" Problem (Lexical Uncertainty)

The Concept: In a story, if you use the wrong word, the sentence might still make sense. But in code, if you get one single symbol wrong (like a missing comma or a wrong math sign), the whole program breaks.
The Analogy: Imagine building a house of cards. If you place one card slightly crooked, the whole tower might fall. The robot's "confidence" isn't spread evenly across the whole house; it's usually fine everywhere except that one shaky card.
The Solution: Instead of checking the whole story, the authors look for the "shaky cards." They check the specific parts of the code where the robot seems most confused (high "entropy"). If the robot is hesitating on even one tiny piece of code, they flag the whole thing as risky.

Result: This method is incredibly fast and cheap, catching many errors that other methods miss.

2. The "Plan vs. Execution" Gap (Algorithmic Uncertainty)

The Concept: A robot can have a great idea for how to solve a problem but mess up the actual steps. Sometimes, two different code solutions look totally different on the surface but do the same thing. Other times, they look similar but do different things.
The Analogy: Imagine asking the robot to explain how to bake a cake.

Method A: Ask it to write the recipe (the code).
Method B (The Paper's Idea): Ask it to explain the plan in plain English first ("First, mix the eggs, then add flour...").
If the robot gives you five different plans for the same cake, it's confused about the strategy. If all five plans are the same, it's confident in its logic.
The Solution: The authors ask the robot to generate several different "plain English plans" for the code. If the plans disagree, the robot is uncertain about the logic, even if the code looks okay.

3. The "Test Drive" (Functional Uncertainty)

The Concept: Code is special because you can actually run it. You can see if it works or not.
The Analogy: Imagine the robot builds a toy car. Instead of just looking at the blueprints, you give it a track to drive on.

The robot builds the car (the code).
The robot also invents a few test tracks (test cases) to see if the car works.
The robot drives the car on those tracks.
The Solution: If the car crashes on 4 out of 5 test tracks the robot invented for itself, the robot should be very uncertain that the car is good. This is a direct "behavioral" check that you can't do with regular text (you can't "run" a paragraph of a story to see if it's true).

The "Three-Legged Stool" (The Ensemble)

The authors combined these three methods into one system.

Leg 1: Checks for shaky cards (Lexical).
Leg 2: Checks if the plans match (Algorithmic).
Leg 3: Checks if the car drives (Functional).

They found that using all three together is much better than using just one. It's like having a safety net made of three different materials; if one fails, the others catch the error.

Key Takeaways from the Paper

Code is different: You can't just copy-paste methods used for writing stories to check code. Code needs its own special rules.
Speed vs. Accuracy: The "shaky card" check (Lexical) is super fast and almost as good as the slow, complex methods. This is great for things like auto-complete in your editor where you need an answer instantly.
The Best Result: When they combined all three methods, they got the best results, correctly identifying uncertain code much better than previous methods.
Comments vs. Code: They found something funny: The robot's confidence in the comments (the English explanations inside the code) is actually a bad sign. If the robot is unsure about the English comments, it often means the code is wrong. But if it's unsure about the code itself, that's the real danger.

In short, the paper says: To know if a robot is confident about code, don't just listen to what it says. Check its shaky spots, compare its plans, and run a test drive.

Technical Summary: Code Is More Than Text: Uncertainty Estimation for Code Generation

Problem Statement

Large Language Models (LLMs) are increasingly deployed as code generators for IDE assistants and autonomous agents. However, silently incorrect programs pose significant safety and reliability risks, as errors can slip past review and compound in downstream steps. While Reliable Uncertainty Estimation (UE) is essential for selective prediction and human-in-the-loop review, current methods are largely direct ports from Natural Language (NL) generation. These NL-derived methods treat code merely as a token sequence, failing to exploit properties that make code distinct:

Token Fragility: Unlike NL, where meaning is often recoverable despite errors, a single wrong token (e.g., an operator or index) can break an entire program.
Intent–Code Gap: Algorithmic intent can be abstracted from concrete implementation; surface-level code variations may hide identical logic, while identical syntax may hide different algorithms.
Executability: Code can be executed against test cases, providing a behavioral signal of correctness unique to code generation.

The paper argues that existing UE methods, which rely on uniform sequence uncertainty or surface clustering, miss these critical signals.

Methodology

The authors propose a three-axis framework for code uncertainty estimation, where each axis corresponds to one of the distinct properties of code. The signals are designed to be orthogonal and complementary.

1. Lexical Uncertainty (Token Fragility)

Concept: Addresses the sparsity of uncertainty in code. Instead of averaging entropy over the entire sequence (which dilutes critical errors), the method focuses on the most uncertain positions.
Implementation: Top-K Token Entropy. The model generates a single greedy output. The uncertainty score is the average of the $K$ highest token-level entropies ( $H(y_t)$ ) in the sequence.
Rationale: A single high-entropy token often indicates a critical decision point where the model is unsure, which is more predictive of failure than the average entropy of the whole sequence.

2. Algorithmic Uncertainty (Intent–Code Gap)

Concept: Measures consistency at the level of algorithmic intent rather than surface syntax.
Implementation: Pseudo-Code Consistency. Instead of sampling multiple code implementations (which introduces implementation noise), the model is prompted to generate $N$ natural-language solution plans (pseudo-code) describing the algorithmic steps.
Metric: Inter-sample agreement is measured using step-aware ROUGE-L, which splits plans into semantic steps and computes bidirectional matching. High disagreement among plans indicates high uncertainty.

3. Functional Uncertainty (Executability)

Concept: Leverages the ability to execute code to measure behavioral correctness.
Implementation: Behavioral Consistency. The model generates a candidate program and $M$ self-generated test cases. The candidate is executed against these tests in a sandbox.
Metric: The uncertainty score is the failure rate ( $1 - \text{pass rate}$ ) of the candidate against its own self-generated tests. This serves as a calibration signal rather than a selection mechanism (unlike CodeT, which reranks candidates).

Ensemble

The three signals ( $U_{lex}$ , $U_{algo}$ , $U_{func}$ ) operate on different scales. The authors rank-normalize each score to $[0, 1]$ and combine them via a weighted sum. The default weights are $(0.2, 0.4, 0.4)$ for Lexical, Functional, and Algorithmic axes, respectively.

Key Contributions

Three-Axis Framework: The paper introduces a code-specific UE framework that maps directly to token fragility, algorithmic intent, and executability, moving beyond NL-derived abstractions.
Empirical Validation: The method is evaluated across four benchmarks (APPS-Intro, APPS-Interview, HumanEval, MBPP) and five code-capable LLMs (including Qwen3-14B, DeepSeek-Coder-V2, and Devstral-Small-2505).
Cost-Performance Analysis: The study demonstrates that Top-K token entropy alone, despite being a single-pass method, matches the performance of the strongest multi-pass NL baselines at over $3\times$ lower computational cost.
Signal Orthogonality: Analysis confirms that the three signals capture distinct information (low pairwise correlations) and target different failure modes (e.g., lexical for single-token errors, algorithmic for strategy mismatches, functional for semantic errors).

Results

Performance Gains: The three-axis ensemble improves the average AUROC from 0.696 (strongest NL-derived baseline) to 0.776 across all models. On Qwen3-14B, the ensemble reaches 0.800 AUROC.
Lexical Efficiency: Top-K token entropy (single-pass) achieves an average AUROC of 0.728, matching the multi-pass "Consistency (VR)" baseline (0.728) while being significantly faster (2.29s vs 7.10s per problem on HumanEval).
Token Scope Asymmetry: An analysis of token scope reveals that entropy calculated on code tokens only (AUROC 0.716) is a strong signal, whereas entropy on comment tokens only performs worse than random (AUROC 0.375). This suggests that uncertainty in natural-language explanations within code anti-correlates with functional correctness.
Cross-Language Generalization: The lexical signal (Top-5 Entropy) generalizes to non-Python languages (C++, Java, Go, JS), consistently outperforming single-pass baselines and multi-pass consistency methods.

Significance and Claims

The paper claims that code uncertainty estimation deserves code-specific design rather than direct ports from natural language. The authors argue that as code-LLM deployment matures, the bottleneck shifts from capability scaling to calibration.

Key takeaways include:

Token fragility implies that uncertainty in code is sparse; focusing on high-entropy tokens is more effective than sequence averaging.
Pseudo-code abstraction provides a cleaner basis for consistency estimation than raw code, separating intent from implementation noise.
Executability offers a unique, low-cost behavioral signal unavailable in NL generation.
Practical Guidance: For latency-sensitive settings (e.g., IDE autocomplete), Top-K token entropy is recommended as a competitive, low-cost default. When calibration is paramount and cost is less constrained, the full three-axis ensemble provides the strongest performance.

The authors position this work as a "first instantiation" of these axes, suggesting that future frameworks should expand to include other code-specific signals (e.g., type checks, static analysis) rather than forcing them into NL-style estimators.