When to Answer and When to Defer: A Decision Framework for Reliable Code Predictions

Technical Summary: When to Answer and When to Defer

Problem Statement
Code language models are increasingly deployed for classification (e.g., vulnerability detection) and generative tasks (e.g., code synthesis). However, these models frequently exhibit poor alignment between predictive confidence and actual correctness, often assigning high confidence to incorrect predictions and low confidence to correct ones. Existing calibration and uncertainty estimation methods, largely developed for natural language, fail to transfer effectively to code. Specifically, post-hoc calibration techniques often reduce average miscalibration but fail to improve the ranking of predictions by correctness likelihood, which is a prerequisite for effective selective prediction (abstention). Furthermore, current approaches treat uncertainty as a passive indicator rather than an actionable signal, lacking integrated mechanisms to resolve or validate uncertain cases.

Methodology: A Unified Decision Framework
The authors propose a deployment-oriented framework that integrates uncertainty estimation, model calibration, and tool-based abstention handling. The framework is designed to operate across both classification and generative tasks through three core components:

1. Uncertainty Estimation: The system extracts internal signals from models to inform downstream decisions. This includes metrics derived from predictive distributions and methods quantifying model variability under controlled perturbations. These signals serve as the basis for computing correctness likelihood rather than acting as direct filters.
2. Calibration Strategies:
   • Pre-deployment Calibration: For classification models, the framework introduces an explicit abstention head or a "reject" class in the label space, trained with selective risk-aware losses. For generative models, it employs reward-aware training to align generation with correctness proxies (e.g., validation pass rates), encouraging conservative behavior under uncertainty.
   • Post-deployment Calibration: To enable abstention without retraining, the framework applies weighted calibration schemes (emphasizing high-error or underrepresented classes) and variance-aware calibration (mapping prediction dispersion to correctness probabilities). For generative models, calibration occurs at both the token level (for incremental trust) and sequence level (for holistic output assessment).
3. Abstention and Recovery (MCP Layer): When a calibrated score falls below a tunable threshold, the system abstains and activates a Model-Context-Protocol (MCP) recovery layer.
   • For Generative Tasks: The MCP layer addresses failure modes by augmenting inputs with missing task information (e.g., generating intermediate plans), injecting external knowledge (e.g., documentation snippets), or enforcing simplifications (e.g., constraining generation length).
   • For Classification Tasks: Abstention triggers domain-specific recovery using static analyzers, program slicing tools, or rule-based validators to assess control flow and data dependencies, ensuring decisions are supported by symbolic evidence.

Key Contributions

• Unified Framework: A generalizable architecture for selective abstention and structured recovery across both classification and generative code modeling tasks.
• Task-Specific Calibration: Novel calibration strategies tailored to code, including logit-based behavioral calibration for classification and weighted scaling for generative models.
• Dual-Phase Abstention Mechanisms: Methods for explicit abstention via model retraining (pre-deployment) and via calibrated confidence thresholds without retraining (post-deployment).
• Actionable Abstention: A re-conceptualization of abstention as a decision primitive that triggers tool-guided interventions (MCP) rather than a terminal failure.
• Open Science Commitment: A pledge to release code, configurations, and datasets under a permissive license.

Preliminary Results
The authors present empirical findings evaluating the transferability of uncertainty metrics and calibration techniques to pretrained code models:

• Generative Models: A weighted logistic scaling method applied to DeepSeek-Coder-7b and CodeLlama-7b on the MBPP+ benchmark consistently reduced calibration error (ECE) and Brier score compared to Platt scaling and isotonic regression. At 80% coverage, selective prediction accuracy exceeded 70%.
• Classification Models: A behavioral calibration method operating directly on model logits (bypassing softmax) was applied to defect prediction tasks. This approach yielded the lowest Brier and ECE scores among tested strategies, achieving over 90% selective accuracy at 80% coverage for DeepSeek-Coder-7b and Qwen-Coder-7b.
• Uncertainty Metrics: Evaluation of 16 uncertainty metrics across defect prediction, vulnerability detection, and code generation revealed that while many correlate with correctness, none serve as a task-agnostic solution for reliable abstention without behavioral calibration. Uncalibrated metrics often overestimate reliability on erroneous predictions due to superficial pattern matching in code.

Significance and Claims
The paper argues that reliable deployment of code models requires moving beyond passive confidence reporting to operational systems that act cautiously under uncertainty. The authors claim that their framework provides a foundation for risk-aware, coverage-controlled use of code models. By treating abstention as a principled decision-making mechanism supported by lightweight analysis tools, the work aims to enable robust deployment in high-stakes software engineering workflows without sacrificing coverage for safety. The research emphasizes that effective abstention in code tasks requires behavioral calibration—aligning internal confidence with correctness at the per-instance level—rather than relying solely on distributional metrics.