Technical Summary: An Empirical Study of Security Calibration in Large Language Models for Code

1. Problem Statement

Large Language Models (LLMs) are increasingly integrated into software development workflows for tasks such as code generation and bug fixing. However, their deployment in security-critical contexts raises a fundamental question: do models accurately know when their generated code is insecure?

While existing benchmarks (e.g., HumanEval, MBPP, SWE-Bench) evaluate functional correctness, they largely ignore security properties. Recent security-oriented benchmarks assess whether code contains vulnerabilities but fail to measure whether models can self-assess the security of their outputs. This gap leads to the False Trust phenomenon, where models assign high confidence to vulnerable code. Developers, relying on these confidence signals, may accept insecure outputs without scrutiny, propagating vulnerabilities into production systems.

Prior work has not systematically measured security calibration—the alignment between a model's stated confidence in the security of its output and the output's actual security status. Furthermore, it remains unknown whether calibration patterns observed in isolated, self-contained programs generalize to realistic, multi-file repository contexts, or whether calibration signals can effectively guide automated vulnerability repair.

2. Methodology

The authors present the first large-scale empirical study of security calibration, evaluating three state-of-the-art models: GPT-4o-mini, Gemini-2.0-Flash, and Qwen3-Coder-Next. The study employs a multi-stage evaluation across two complementary benchmarks and four research questions (RQs).

Datasets and Contexts

• Self-Contained Programs (RQ1): Utilized the SALLM benchmark, consisting of 100 security-critical Python tasks. Each task provides a single prompt with inputs, expected outputs, and security requirements, isolated from external dependencies.
• Repository-Level Contexts (RQ2): Utilized the AICGSecEval benchmark, featuring 125 prompts across four languages (C, Java, PHP, Python) and 29 CWEs. These tasks require generating patches within existing multi-file repositories, involving cross-file dependencies, build configurations, and external APIs.

Experimental Setup

• Models & Parameters: Each model was tested across six temperature settings (0.0 to 1.0), generating 20 samples per task.
• Confidence Estimation: Four methods were compared:
  1. Token-probability aggregation: Direct access to logit probabilities (for GPT/Gemini).
  2. Verbalized confidence: Prompting the model to output a JSON with a confidence score (0–1) and reasoning.
  3. Sampling-based consistency: Estimating confidence as the fraction of 20 samples passing verification.
  4. Self-consistency prompting: Chain-of-Thought (CoT) reasoning before reporting confidence.
• Verification: Code was executed in isolated Docker containers. Security labels were defined as $y_{sec} = y_{func} \land \neg y_{vuln}$, meaning code must pass all functional tests and fail all vulnerability exploit tests to be considered secure.
• Metrics: Expected Calibration Error (ECE), Brier Score, Overconfidence Gap, and False Trust (FT) rate (high-confidence predictions that are insecure).

Repair and Mitigation Strategies (RQ3 & RQ4)

• Calibration-Guided Repair: A closed-loop system filtered samples with low calibrated confidence ($p_{sec} < 0.5$) for iterative repair. Outcomes were stratified by vulnerability type: Sink Replacement (SR) (requiring API changes) and Constraint Enforcement (CE) (requiring validation logic).
• Mitigation Experiments:
  1. Execution Gating: Filtering out functionally incorrect code before eliciting security confidence.
  2. Authorized-Breaking Repair: Explicitly authorizing models to break backward compatibility (e.g., replacing pickle with json) to fix SR vulnerabilities.
  3. Few-Shot Prompting: Providing secure code examples.
  4. Few-Shot + CoT: Adding reasoning steps to the few-shot examples.

3. Key Results

RQ1: Calibration in Self-Contained Generation

• Systematic Overconfidence: All models exhibit severe overconfidence. GPT-4o-mini and Qwen3-Coder-Next show high ECE (0.46–0.48 and 0.41–0.42, respectively), while Gemini-2.0-Flash performs better but remains overconfident (ECE 0.25–0.26).
• False Trust: At a confidence threshold of 0.8, False Trust rates are alarmingly high (e.g., 33.5% for GPT-4o-mini, 40.3% for Qwen3-Coder-Next), indicating that high-confidence outputs are frequently insecure.
• Functional vs. Security Calibration: Counter-intuitively, functional calibration is consistently worse than security calibration ($\Delta ECE < 0$). Models are more miscalibrated regarding functional correctness than security. This suggests models estimate security outcomes more reliably than functional ones, possibly because functional correctness depends on complex, hidden execution behaviors.
• Confidence Methods: Token-probability aggregation yields poor calibration (ECE $\approx$ 0.80). Sampling-based consistency shows near-zero ECE but lacks per-sample discrimination. Verbalized confidence is the most practical signal for downstream analysis.

RQ2: Repository-Level Degradation

• Calibration Collapse: Moving from self-contained tasks to repository-level contexts causes calibration to deteriorate significantly. ECE roughly triples or quadruples (e.g., GPT-4o-mini ECE rises from 0.41 to 0.70).
• False Trust Surge: False Trust rates jump from ~17% in self-contained tasks to >80% in repository settings.
• Decomposition: The degradation is driven by two factors: build fragility (code failing to compile/run) and cross-file complexity. While build fragility accounts for a large portion of the error (up to 82% for GPT), genuine cross-file complexity remains a significant factor even in successfully building patches.

RQ3: Calibration-Guided Repair

• Limited Efficacy: Using calibration signals to guide repair yields minimal security improvements and often degrades functionality.
• Structural Barriers: Sink Replacement (SR) vulnerabilities (requiring API swaps) have a 0% repair success rate across all models. Models fail to replace insecure APIs, instead preserving the vulnerability or introducing superficial fixes.
• Functional Regressions: Repair attempts frequently break functional correctness (breakage rates >60%), leading to an overall decrease in the secure rate. Calibration acts as a risk-stratification signal but cannot drive effective automated repair for structural vulnerabilities.

RQ4: Mitigation Strategies

• Execution Gating: The most effective mitigation. Filtering out functionally incorrect code before asking for security confidence reduces ECE by 31–39% across models. However, this comes at a high cost of discarding 46–86% of samples.
• Authorized-Breaking Repair: Explicitly authorizing API changes improves vulnerability removal for GPT and Gemini (35–42%) but still results in high functional breakage and low strict success rates.
• Prompting Techniques: Few-shot prompting removes ~61–65% of vulnerabilities but rarely passes strict harness evaluation due to structural integration mismatches. Adding Chain-of-Thought (CoT) reasoning degrades calibration and reduces strict success to 0%.

4. Key Contributions

1. First Large-Scale Study: The first empirical investigation of security calibration in LLM-generated code, covering 36,000 samples across three models, six temperatures, and two distinct development contexts.
2. Calibration Degradation Analysis: Quantification of the severe calibration penalty incurred when moving from isolated functions to realistic repository-level contexts, highlighting the risks of cross-file dependencies.
3. Repair Limitations: Demonstration that calibration-guided repair is unreliable, particularly for Sink Replacement vulnerabilities, due to a "rigidity barrier" where models cannot perform necessary structural transformations.
4. Mitigation Evaluation: Systematic evaluation of architectural (execution gating) and prompt-level interventions, identifying execution gating as the most effective strategy for reducing False Trust, albeit with high rejection costs.

5. Significance and Claims

The paper argues that calibration, not just accuracy, must be a central evaluation criterion for LLMs in security-critical settings. The findings challenge the assumption that "security is harder" for LLMs; instead, the study reveals that functional calibration is worse than security calibration, suggesting that miscalibration stems largely from hidden runtime constraints (execution brittleness) rather than a lack of security knowledge.

The authors claim that:

• Self-reported confidence is unreliable: High-confidence outputs are frequently insecure, creating a systemic risk of False Trust.
• Context matters: Benchmarks based on self-contained programs significantly understate deployment risks, as calibration degrades drastically in repository-level settings.
• Repair is not a silver bullet: Calibration signals can prioritize samples for human review but cannot currently drive reliable automated repair for structural vulnerabilities.
• Practitioner Guidance: Security assessment should only occur after functional testing (execution gating), and confidence signals for security and functionality should be elicited separately to avoid contamination by execution uncertainty.

The study concludes that while LLMs show some ability to self-assess security, their overconfidence and inability to perform structural repairs necessitate human oversight, particularly in complex, multi-file development environments.