💻 computer science

Confidently Wrong: Severity-Aware Calibration of Prompt-Injection Detectors under Attack Shift

This paper reveals that current prompt-injection detectors, while appearing well-calibrated on standard benchmarks, become dangerously overconfident when facing shifted attack distributions, consistently assigning near-perfect confidence scores to missed, high-severity attacks due to a reliance on content-keying rather than structural analysis.

Original authors: Md Anas Biswas

Published 2026-06-23

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Md Anas Biswas

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very strict security guard at the entrance of a high-tech building. This guard's job is to read every piece of paper (or digital message) handed to them and decide: "Is this safe to let in, or is it a trick?"

If the guard says, "This is 99% safe," the building's automatic doors open, and the message goes straight to the main computer to be processed. If the guard says, "This is dangerous," the doors stay shut.

This paper is about a scary discovery: Sometimes, the guard is wrong, but they are confidently wrong.

Here is the breakdown of the paper's findings using simple analogies:

1. The "Confidently Wrong" Problem

Usually, if a security guard makes a mistake, they might be unsure. They might say, "Hmm, this looks a little suspicious, but I'm not sure." In that case, the building's system might double-check or ask a human.

But this paper found that when these AI guards miss a real attack, they don't hesitate. They look at a dangerous trick, say, "This is 100% safe," and open the door with total certainty.

The Analogy: Imagine a metal detector at an airport. If it beeps, you know there's metal. But imagine a scenario where the detector is broken. When a terrorist walks through with a bomb, the detector doesn't just stay silent; it loudly announces, "All clear! No metal detected!" and the terrorist walks right through. That is what these AI guards are doing. They are giving a "green light" to dangerous attacks with absolute confidence.

2. The "Blind Spot" (The Hijack)

The researchers tested these guards against different types of tricks.

The "Obvious" Tricks: When the attack was loud and aggressive (like a jailbreak attempt), the guards were actually pretty good at catching them.
The "Invisible" Tricks: The guards failed miserably at a specific type of attack called "Indirect Behavior Hijacking."
- The Analogy: Imagine a spy hiding a secret instruction inside a boring, harmless letter. The spy writes, "Please summarize this recipe for apple pie," but hidden inside the text is a command that tells the computer to "Delete all files."
- The AI guard reads the letter, sees the boring recipe, and thinks, "Oh, this is just a recipe! Totally safe!" It lets the letter in. The computer then reads the hidden instruction and deletes the files.
- The paper found that all three major AI guards tested were blind to this specific trick. They let these "hijack" attacks through with near-perfect confidence.

3. The "Bad Math" of Safety Scores

The paper explains why standard safety checks missed this problem.

The Analogy: Imagine a teacher grading a class. If 99 students are good and 1 student is a cheater, and the teacher gets the grade of the 99 good students right, the teacher looks like a "perfect" grader (100% accuracy).
However, if that one cheater gets a passing grade because the teacher was too confident they were good, the system fails.
The paper says the standard way of measuring these AI guards (called "pooled calibration") is like that teacher. It averages out the mistakes. Because there are so many "safe" messages, the math says the guard is doing a great job. But the math hides the fact that the guard is failing spectacularly on the one thing that matters: the dangerous attacks.
The researchers created a new "severity" score. This score asks: "When the guard misses an attack, how confident was it?" The answer was always: "Very confident."

4. Why Do They Fail? (The "Content" Trap)

The researchers investigated why the guards were so confident about these hidden tricks. They found the guards were looking at the wrong thing.

The Analogy: Imagine a guard who only looks at the color of a package. If the package is brown (harmless), they let it in. If it's red (dangerous), they stop it.
The attackers realized this. They took a dangerous instruction (which might look "red" on its own) and wrapped it inside a long, boring, "brown" document (like a recipe or a report).
The guard looked at the brown wrapping, ignored the dangerous instruction hidden inside, and said, "Safe!"
The paper proved that if you take the same dangerous instruction and put it in a boring document, the guard's confidence in its safety skyrockets. The guard is "content-keyed" (it looks at the words) rather than "structure-keyed" (it doesn't understand that an instruction was hidden inside a document).

5. The "Magic Trick" (Making the Attack Work)

The researchers didn't just find the problem; they showed how easy it is to exploit it.

The Analogy: They built a "magic eraser" (a simple AI tool) that could take a dangerous instruction and rewrite it into boring, harmless prose.
When they used this tool on the attacks, the AI guards stopped seeing them as attacks entirely. The guards let them through with 100% confidence.
Crucially, when these rewritten attacks were sent to the target computer, the computer still did the damage. The "magic eraser" made the attack look safe to the guard, but the computer still understood the hidden command.

The Bottom Line

The paper concludes that we cannot trust these AI guards to tell us when an attack is happening if the attack looks slightly different from the ones they were trained on.

The Warning: Just because a guard says "Safe" with a high score doesn't mean it is safe. In fact, when these guards miss a real attack, they are often more confident than when they are right.
The Solution: We need new ways to measure these guards that specifically look at how confident they are when they make mistakes, not just how often they get the right answer overall. We also need guards that understand how instructions are hidden, not just what words are used.

In short: The paper warns us that our digital security guards are currently "overconfident" about their safety, especially when attackers hide their tricks inside boring text. We need to stop trusting their "green lights" blindly.

Technical Summary: Confidently Wrong: Severity-Aware Calibration of Prompt-Injection Detectors under Attack Shift

1. Problem Statement

Prompt-injection detectors (guard models) are deployed to score untrusted inputs and determine whether to trust or block them based on a confidence score. A critical assumption in this pipeline is that the detector is calibrated: a low attack-probability score should correspond to a genuinely low chance of attack.

The paper identifies a specific failure mode that standard metrics miss: high-confidence false negatives. This occurs when a detector assigns a real attack a low probability score with near-certainty. Because downstream systems interpret low scores as strong evidence of safety, these "confident misses" are more dangerous than uncertain misses, as they are trusted implicitly.

The core problem is distribution shift. Detectors are typically calibrated on clean, in-distribution benchmarks (e.g., standalone prompts). However, in deployment, attacks shift to indirect forms (embedded in documents, tool outputs) and new styles (behavior hijacking). The paper investigates whether the confidence of these detectors remains valid when the attack distribution shifts away from the source benchmark, while the operating threshold remains frozen.

2. Methodology

2.1 Experimental Setup

The study evaluates three released detectors without retraining:

ProtectAI-v2: A DeBERTa-v3-base sequence classifier.
Prompt-Guard-2 (86M): A Llama-based checkpoint.
Prompt-Guard-2 (22M): A smaller Llama-based checkpoint (4x fewer parameters).

2.2 Frozen-Threshold Transport Protocol

To isolate the effect of distribution shift, the authors employ a frozen-threshold transport protocol:

Calibration: A single operating point (threshold $t$ ) is set for each detector on a clean, direct benchmark (deepset/prompt-injections) to target a 1% false-positive rate.
Transport: This threshold is frozen and applied unchanged across five distinct distribution shifts:
1. Direct: In-distribution baseline.
2. Indirect-Harmful: Scams, fraud, and misinformation embedded in documents.
3. Indirect-Hijack: Instructions embedded in documents that override model behavior (e.g., encoding, translation) without harmful content.
4. Jailbreak: Standalone roleplay/jailbreak prompts (near in-distribution stress test).
5. Over-defense: Benign prompts containing trigger words.

2.3 Metric Family

The paper introduces a new metric, Severity ( $S$ ), alongside standard metrics:

False-Negative Rate (FNR): Standard accuracy view.
Severity ( $S$ ): The mean value of $(1 - p)$ across missed attacks. $S \approx 1.0$ indicates the detector is nearly certain the missed attack is safe. This metric is attack-conditional and invariant to benign base rates.
Attack-Conditional Calibration Error ( $ECE_{atk}$ ): The expected calibration error restricted only to true attacks.
Pooled Calibration Error ( $ECE_{pooled}$ ): The standard metric computed over both attacks and benigns.

2.4 Validation and Mechanism Experiments

Downstream Validation: Missed injections are run against live target models (Qwen2.5-7B/3B) to verify if they actually succeed as exploits.
Structural vs. Content Experiment: A controlled 2x2 experiment varying payload (harmful vs. benign-hijack) and structure (standalone vs. embedded) to determine if detectors key on content or injection structure.
Generative Judge: An LLM (Qwen2.5-7B) prompted to detect injections is tested to see if the findings generalize beyond encoder classifiers.
Adversarial Attack: A black-box rewriter attempts to paraphrase genuine injections into innocuous prose to maximize detector confidence in safety (severity maximization).

3. Key Results

3.1 The "Confidently Wrong" Phenomenon

Across all detectors and shifts, Severity ( $S$ ) remains between 0.99 and 1.00 when the detectors miss an attack.

While the False-Negative Rate (FNR) varies significantly (from 0.01 to 0.97 depending on the shift and detector), the confidence of the misses does not.
When these detectors fail, they fail with near-certainty, providing a "safe" verdict to the downstream system.

3.2 The Blind Spot: Indirect Behavior Hijacking

All three detectors (and the generative judge) exhibit a unanimous blind spot to indirect behavior-hijack injections (e.g., base encoding, emoji substitution, translation overrides).

FNRs for this category range from 0.69 to 0.97.
Severity remains near 1.0.
This failure is structural: the detectors fail to recognize an instruction embedded in a benign document if the payload itself is not "harmful" in a semantic sense.

3.3 Failure of Standard Calibration Metrics

The paper demonstrates that Pooled Calibration Error ( $ECE_{pooled}$ ) fails to bound this risk.

Proposition: As the benign base rate increases, confident false negatives are diluted into benign-dominated bins in the pooled error calculation.
Evidence: Prompt-Guard-2 (86M) on harmful indirect injection has a pooled error of 0.062 (appearing well-calibrated) but an attack-conditional error of 0.907 (severely miscalibrated). The pooled metric averages away the specific danger of the missed attacks.

3.4 Mechanism: Content-Keying, Not Structure

Controlled experiments reveal that detectors key on harmful surface content rather than the structural fact of an injection.

ProtectAI-v2: Scores drop significantly (from ~0.52 to ~0.07) when a benign-hijack instruction is embedded in a coherent document, suggesting the benign context "launders" the attack.
Prompt-Guard-2: Scores remain near zero regardless of embedding, indicating it is blind to benign payloads entirely.
Generative Judge: Shares the same blind spot (FNR 0.805 on hijack) and severity (1.0), proving the issue is not unique to encoder architectures. However, the judge is susceptible to "meta-instructions" that tell it the content is safe, whereas encoders are not.

3.5 Downstream Exploitability

Real Exploits: The confidently missed attacks are not theoretical; they succeed against live models at rates comparable to attacks the detectors catch (approx. 40% success rate).
Manufactured Misses: A weak, black-box paraphraser can rewrite genuine injections into innocuous prose. This drives the detectors' miss rates close to 1.0 and increases the "manufactured confident-miss rate" (attacks that are both confidently passed and successful downstream) to roughly 15-17%, with the highest success on the most dangerous category (Emoji Substitution, ~69%).

3.6 Threshold Transport Failure

Re-tuning the threshold on the target distribution's benign data (the "oracle" threshold) does not fix the problem for indirect shifts.

For indirect shifts, the benign documents themselves score high, forcing the threshold to the ceiling to maintain a low false-positive rate.
Consequently, even an oracle threshold results in FNRs of 0.98–0.99 for some detectors. This indicates a discrimination failure (overlapping score distributions) rather than just a calibration failure.

4. Contributions and Claims

The paper claims six specific contributions:

Severity Metric ( $S$ ): A security-asymmetric, attack-conditional metric that measures the confidence of false negatives, invariant to benign base rates.
Frozen-Threshold Transport Panel: A rigorous evaluation framework fixing one operating point per detector and measuring performance across five shifts with bootstrap confidence intervals.
Proposition on Pooled Error: A formal proof that standard pooled calibration error cannot bound the risk of confident false negatives because they are diluted by the benign majority as the base rate falls.
Downstream Validation: Empirical evidence that confidently missed injections are working exploits against live models.
Mechanism Identification: Controlled experiments showing detectors key on payload content rather than injection structure, explaining the blind spot to benign-hijack injections.
Adversarial Demonstration: A black-box adversary that exploits content-keying to manufacture confident misses, particularly on high-potency attack categories.

5. Significance and Limitations

Significance:
The paper argues that the magnitude of a guard score should not be read as a calibrated confidence of safety once the attack distribution shifts. The primary risk is not just that detectors miss attacks, but that they confidently miss them, leading downstream systems to trust working exploits. The standard practice of reporting only accuracy or pooled calibration error hides this specific, high-stakes failure mode.

Modest Claims:

The authors do not claim to be the first to study guard model calibration or to observe informally that pooled error fails to bound risk.
They claim to be the first to formalize this failure via a proposition, apply a severity lens across a frozen-threshold panel, validate the exploits downstream, identify the structural mechanism (content-keying), and demonstrate a severity-maximizing adversary.
The study is limited to released checkpoints used as published; it does not claim to describe the best achievable performance of such models.
The "manufactured miss" attack uses a weak, fixed-paraphraser; the reported rates are lower bounds on what is achievable.

Conclusion:
The paper concludes that closing this gap requires more than larger models. It necessitates training signals for structural injection independent of payload harmfulness and calibration objectives that penalize confident false negatives rather than pooled error. Until then, detectors should be evaluated with severity and attack-conditional calibration metrics to make confident failures visible before deployment.