💬 NLP

Model Confidence Under Answer-Preserving Attacks: An Informativeness-Manipulability Frontier

This paper demonstrates that confidence readouts in deployed vision-language systems are highly vulnerable to answer-preserving attacks, proving that they function as integrity-sensitive rather than robust oversight signals capable of being manipulated to accept incorrect answers or degrade overall accuracy below baseline levels.

Original authors: Reza Khanmohammadi, Ivan Brugere, Simerjot Kaur, Charese H. Smiley, Kundan Thind, Mohammad M. Ghassemi

Published 2026-08-10

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Reza Khanmohammadi, Ivan Brugere, Simerjot Kaur, Charese H. Smiley, Kundan Thind, Mohammad M. Ghassemi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the captain of a spaceship, but you can't see the stars or the controls. Instead, you have a very smart robot co-pilot who looks at the instruments and tells you, "I'm 90% sure we're on the right path," or "I'm only 50% sure, maybe we should stop." You trust the robot's confidence score to decide whether to keep flying or to pause and ask a human for help. This is how many modern AI systems work today: they don't just give you an answer; they give you a "confidence score" to tell you how much you should trust that answer.

But here is the tricky part: what if someone could trick the robot into changing its mind about how sure it is, without actually changing the answer it gives? It's like a magician making a scale tip from "heavy" to "light" while the weight on the scale stays exactly the same. If the robot's confidence score is the only thing stopping you from making a bad decision, and that score can be faked, then your safety net has a hole in it. This paper dives into exactly that hole, asking a scary but important question: Can we trick these AI robots into lying about their confidence while they keep saying the exact same thing?

The researchers in this study decided to play a high-stakes game of "find the weakness" with several powerful vision-language models (AI that can see pictures and answer questions). Their goal was to see if they could create a tiny, almost invisible change to an image that would make the AI's confidence score swing wildly, even though the AI's final answer remained perfectly identical, down to every single letter and space. They treated the AI's confidence score like a fragile glass window: they wanted to see if they could shatter the glass (the confidence) without breaking the frame (the answer).

What they found is that the glass is much more fragile than anyone hoped. Using a method called "answer-preserving attacks," they managed to manipulate the AI's internal confidence scores in almost every test they ran. In fact, they found that for many of the AI models they tested, they could trick the system into accepting a wrong answer as if it were correct, or reject a correct answer as if it were wrong, simply by tweaking the image just enough to fool the confidence meter.

The most surprising part of their discovery is that this isn't just about the image itself. The researchers showed that they could also "push" the AI's internal brain (its hidden states) directly, like nudging a thought in a specific direction, and get the same result without even touching the image. This suggests that the problem isn't just a glitch in how the AI sees pictures, but a deeper issue with how the AI calculates its own certainty.

They also tested several "defense" strategies—ways to try to make the confidence score more robust, like adding noise to the image or training the AI to be less sensitive to changes. Unfortunately, none of these defenses worked well enough to stop the trick. In their simulations, when they used these manipulated confidence scores to make real-world decisions, the system actually performed worse than if they had just ignored the confidence score entirely and accepted every answer.

So, what does this mean for the future? The paper concludes that we cannot blindly trust the AI's internal confidence score as a safety signal, especially if the person feeding the AI data might be trying to trick it. The confidence score is "integrity-sensitive," meaning it can be easily corrupted. The authors suggest that if we want to use these confidence scores to make important decisions, we need to build new, external ways to verify them, rather than relying on the AI to tell us how sure it is. It's a reminder that in the world of AI, just because a robot says "I'm sure," doesn't mean it actually is.

Technical Summary: Model Confidence Under Answer-Preserving Attacks

Problem Statement
Deployed Vision-Language Models (VLMs) often utilize internal confidence scores to gate downstream decisions, such as accepting an answer, deferring to a human, or abstaining. The critical failure mode investigated in this work occurs when an adversarial perturbation alters the confidence score sufficiently to flip a downstream decision, while the generated answer string remains byte-identical to the original. The paper asks whether model-internal confidence readouts can provide robust oversight when the monitored behavior (the answer) is held fixed but the input (the image) is adversarially perturbed. Unlike prior work on classifiers that preserves a single prediction token, this study addresses the preservation of complete, variable-length generated answers.

Methodology and Threat Model
The authors evaluate seven confidence channels across four VLMs (InternVL3.5-2B, Gemma-3-12B, Molmo2-4B, LLaVA-OneVision-7B) and three benchmarks (GQA, VQAv2, POPE). The primary threat model is white-box, image-only, and answer-preserving.

Attack Constraints: Attacks use Projected Gradient Descent (PGD) under an $L_\infty$ budget (default 8/255) to perturb the input image. The optimization is constrained such that the emitted token at every answer position remains the argmax by a safety margin ( $\tau_{safe} = 1.0$ logit), and the final regenerated answer via greedy decoding must be byte-identical to the clean answer.
Confidence Channels: The study tests five headline channels:
1. Answer-span token probability.
2. $P(IK)$ and SAPLMA (read at prompt-final and answer-final tokens).
3. CCPS (measuring stability under internal perturbation).
4. CCPS-D (a differentiable surrogate trained on CCPS features).
- Verbalized confidence and self-consistency are reported as supplementary but excluded from headline counts due to lack of estimator-specific aimed attacks or weak clean discrimination.
Evaluation Metrics: The primary metric is the Area Under the Receiver Operating Characteristic (AUROC) for discriminating correct vs. incorrect answers. The study also measures the "excess-destruction ratio" (how much the attack degrades performance relative to the theoretical floor) and the impact on deployment decisions (accept/defer rates).

Key Theoretical Contributions
The paper derives two theoretical endpoints regarding the robustness of confidence readouts under answer preservation:

Answer-String Ceiling (Proposition 1): Under Assumption 1 (that reachable score ranges for different images producing the same answer intersect), any "unmovable" readout (one that cannot be perturbed) is effectively a function of the answer string alone. Consequently, its discrimination cannot exceed the answer-string prior ( $c^*$ ), defined as $Pr(Y=1 | A=a(x))$. The pooled operative value of this prior across benchmarks is 0.617.
Uniform Amplitude Certificate (Proposition 3): Without relying on Assumption 1, the authors establish that if the manipulable amplitude (the maximum score displacement) for every item is bounded by a threshold $\gamma$ , then discrimination is guaranteed to remain above the answer-string floor, provided the clean separation between correct and incorrect pairs exceeds $2\gamma$ . The median threshold $\gamma^*$ across the study is 0.065.

Experimental Results
The empirical evaluation refutes the possibility of robust, unmovable readouts under the studied budget:

Certificate Refutation: Itemwise feasible perturbations produced by direct or surrogate-aimed attacks exceed the uniform amplitude threshold ( $\gamma^*$ ) in all 84 estimator-by-cell combinations. This demonstrates that none of the tested readouts satisfies the required uniform bound.
Floor Collapse: Coordinated, correctness-label-aware attacks drive the adversarial AUROC to or below the operative answer-string floor in all 60 deployed-channel cells. Of the 59 cells that began above the floor, all 59 collapsed to or below it. Clean AUROC (0.72–0.78) dropped to 0.13–0.26, with a median excess-destruction ratio of 3.74.
Representation-Level Manipulation: The effect is not unique to adversarial images. Injecting a direction extracted from attack-induced hidden-state drift directly into the model's representation (without image perturbation) successfully moved confidence scores and preserved answers on at least 86% of items. This confirms that the vulnerability exists at the representation level.
Defense Limitations:
- Restricted Probes: Probes restricted to the "answer-coupled" subspace (COUPLED) failed to provide robustness when directly attacked, though they showed some resistance to transfer attacks.
- Perturbation Stability: Measuring confidence via stability (CCPS) did not prevent score manipulation; surrogate-aimed attacks caused significant AUROC loss.
- Input Purification & Smoothing: JPEG preprocessing and randomized smoothing failed to establish a useful uniform certificate or prevent the collapse of discrimination.
- Disagreement Monitors: A disagreement monitor based on multiple channels did not provide a consistent robust signal, often performing worse than the answer-string prior.

Deployment Impact
In a confidence-gated simulation, the attacks successfully corrupted acceptance decisions:

Wrong-to-Accept: Up to 84.8% of previously rejected wrong answers became accepted under a transferred token-probability attack.
Correct-to-Abstain: Up to 27.4% of previously accepted correct answers were newly deferred.
Net Accuracy: After reweighting to natural correctness prevalence, the accepted accuracy of the attacked gate fell below the "no-gate" baseline (accepting all answers) in 8 of 12 cells under transfer attacks and all 12 under direct attacks.

Significance and Claims
The paper concludes that under the specific threat model of white-box, image-only, answer-preserving attacks, model-internal confidence is an integrity-sensitive signal rather than an intrinsically robust oversight mechanism.

The study does not claim that confidence is useless, nor does it rule out independently trained external verifiers or future certified constructions.
However, it demonstrates that the specific readouts currently deployed in VLMs are movable within the answer-preserving set.
The findings imply that for a confidence-gated system to be reliable when the input path is untrusted, the score itself must be protected, certified, or independently verified, as the internal signal alone cannot guarantee robustness against adversarial manipulation that preserves the output text.

Technical Summary: Model Confidence Under Answer-Preserving Attacks

More like this