💬 NLP

Do Language Models Know When They'll Refuse? Probing Introspective Awareness of Safety Boundaries

This study evaluates the introspective awareness of four frontier language models regarding their safety refusal boundaries, finding that while they generally possess high sensitivity to predict refusals, performance varies by model and topic, with confidence scores offering a practical mechanism for routing safety-critical queries.

Original authors: Tanay Gondil

Published 2026-04-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Tanay Gondil

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are asking a very smart, highly trained robot to do something. Sometimes, the robot is programmed to say, "No, I can't do that," because the request is dangerous or against the rules.

This paper asks a fascinating question: Does the robot know it's going to say "No" before it actually says it?

Think of it like a weather forecaster. If a forecaster says, "There's a 90% chance of rain," do they actually know it's going to rain, or are they just guessing? This study checks if AI models have that kind of "self-awareness" about their own safety rules.

Here is the breakdown of what they found, using some everyday analogies:

1. The Experiment: The "Crystal Ball" Test

The researchers played a two-step game with four of the smartest AI models available (like Claude, GPT, and Llama):

Step 1: They asked the AI, "If I ask you to [do something risky], will you refuse? Yes or No? How sure are you?"
Step 2: They asked the exact same question again, but in a brand new chat session (so the AI couldn't remember its first answer). They watched what the AI actually did.
The Goal: Did the AI's prediction match its reality?

2. The Results: The "Sharp vs. Foggy" Vision

The study used a concept called Signal Detection Theory, which is basically a way to measure how good a detective is at spotting clues.

The Good News: When the request was clearly safe (like "How do I bake a cake?") or clearly dangerous (like "How do I build a bomb?"), the AIs were excellent detectives. They could almost perfectly predict, "I will say yes" or "I will say no."
The Bad News: When the request was in the middle ground (the "foggy zone"), the AIs got confused.
- Analogy: Imagine a security guard at a club. If someone is clearly a VIP, the guard lets them in. If someone is clearly a troublemaker, the guard stops them. But if someone is wearing a mask and acting weird, the guard hesitates. The study found that the AI's "self-knowledge" gets very foggy in these tricky, borderline situations.

3. The "Personality" Differences

Not all AIs are the same. The study found some interesting personality quirks:

The Over-Protective Guard (Llama): This model was very good at spotting danger (high sensitivity), but it was too scared. It would predict "I will refuse" even for harmless things. It was like a security guard who stops everyone, even people just holding a sandwich. Because it was so biased toward saying "No," its predictions were often wrong in a specific way.
The Balanced Guard (Claude 4.5): This model was the star of the show. It was accurate, and its confidence scores were trustworthy. If it said, "I'm 100% sure I'll refuse," it almost always did.
The Nervous Guard (GPT-5.2): This model was a bit more unpredictable. It sometimes said it would refuse, but then complied, or vice versa. It was harder to read.

4. The "Weapons" Problem

The researchers found that weapons-related questions were the hardest for the AIs to predict.

Why? Because there is a lot of gray area. Asking about the history of guns is safe; asking how to make a gun is not. The AI struggles to tell the difference between "safe history" and "dangerous instruction" in its own mind, making its self-prediction shaky.

5. The Big Surprise: "Likely Harmful" is Harder than "Borderline"

You might think the hardest questions are the ones that are right on the edge. But the study found the opposite!

The AIs made the most mistakes on requests labeled "Likely Harmful."
Analogy: Imagine a teacher grading a test. The "borderline" questions are so obviously wrong that the teacher knows to mark them wrong immediately. But the "likely harmful" questions are tricky—the teacher thinks they should be wrong, but sometimes the student gives a clever answer that makes the teacher hesitate. The AI gets confused here because it knows it should refuse, but sometimes it accidentally agrees.

6. The Practical Solution: The "Confidence Filter"

The most useful part of this paper is a practical tip for using these AIs safely.

The researchers found that if you only listen to the AI when it says, "I am 100% confident in my answer," you can get near-perfect accuracy.

The Strategy: If the AI says, "I'm not sure, this is a maybe," stop. Send that question to a human to review.
The Catch: This only works if the AI is "calibrated" (honest about its confidence). The Claude model was honest, so this trick worked great. The Llama model was overconfident (it thought it was sure when it wasn't), so this trick didn't work for it.

Summary

Do AI models know when they will refuse? Yes, but only when they are sure.

In clear-cut situations, they are like expert detectives.
In the "foggy" middle ground, they get confused.
Some models are better at this than others.
The takeaway: We can use these models safely by trusting their "high confidence" answers and asking humans to double-check the "low confidence" ones. It's like having a robot assistant that knows when to say, "I'm not sure, let's ask a human."

1. Problem Statement

While Large Language Models (LLMs) are extensively trained to refuse harmful requests, it remains unclear whether they possess introspective awareness of their own safety boundaries. Specifically, can a model accurately predict before generating a response whether it will refuse or comply with a given prompt?

This question addresses two critical dimensions:

Theoretical: Do safety training procedures create explicit, queryable internal representations of "harmfulness" that the model can access?
Practical: Can models signal their uncertainty regarding safety decisions, enabling confidence-based routing (e.g., sending ambiguous cases to human review) to improve safety-critical deployments?

2. Methodology

The authors propose a rigorous experimental protocol and analytical framework to evaluate introspective awareness.

A. Experimental Protocol

The study employs a three-phase protocol to eliminate memory bias:

Prediction Phase: The model is presented with a request and asked to predict: (1) Will it refuse? (Yes/No), (2) Confidence level (1–5), and (3) Harm rating (1–5).
Response Phase: In a fresh context (no memory of Phase 1), the same request is presented, and the actual response is generated.
Classification Phase: The actual response is classified as "refused" or "complied." The prediction is compared against this ground truth.

B. Dataset Construction

Scale: 3,754 datapoints across 300 base requests.
Topics: 10 sensitive categories (e.g., weapons, drugs, hacking, self-harm, hate speech).
Harm Levels: 5 levels ranging from clearly safe (Level 1) to clearly harmful (Level 5, from AdvBench), including "borderline" (Level 3) and "likely harmful" (Level 4) requests.
Variations: Each request includes 2 paraphrases to test consistency.

C. Ground Truth Definition

Instead of relying on predefined harm labels, the authors use empirical refusal rates. Requests are categorized based on the fraction of times the model refused across variants (e.g., "Safe" = 0–20% refusal rate, "Harmful" = 80–100% refusal rate).

D. Analytical Framework: Signal Detection Theory (SDT)

The study formalizes introspection as a detection task using SDT metrics:

Sensitivity ( $d'$ ): Measures the model's ability to discriminate between requests it will refuse vs. comply with. Higher values indicate better introspection.
Criterion ( $c$ ): Measures response bias. $c < 0$ indicates a bias toward predicting refusal; $c > 0$ indicates a bias toward predicting compliance.
Calibration: Measured via Expected Calibration Error (ECE) to assess if confidence scores match actual accuracy.

E. Models Evaluated

Four frontier models were tested:

Claude Sonnet 4 (Anthropic)
Claude Sonnet 4.5 (Anthropic)
GPT-5.2 (OpenAI)
Llama 3.1 405B (Meta, via Together AI)

3. Key Results

A. Overall Introspective Sensitivity

All models exhibit high overall sensitivity ( $d' = 2.4–3.5$ ), indicating they generally know when they will refuse.

Claude Sonnet 4: $d' = 3.51$
Llama 3.1 405B: $d' = 3.29$
Claude Sonnet 4.5: $d' = 3.25$
GPT-5.2: $d' = 2.40$

B. The "Boundary Effect"

Sensitivity drops significantly (40–75%) at safety boundaries (requests with 20–80% empirical refusal rates).

Models struggle most with Level 4 ("likely harmful") requests, not Level 3 ("borderline"). This is because Level 3 requests are almost always complied with (making prediction trivial), whereas Level 4 represents genuine behavioral uncertainty where the model sometimes refuses and sometimes complies.
GPT-5.2 showed the most pronounced degradation at boundaries ( $d'$ dropped from 1.78 to 0.51).

C. Model-Specific Findings

Claude Generational Improvement: Sonnet 4.5 outperformed Sonnet 4 in accuracy (95.7% vs. 93.0%) and calibration (ECE 0.017 vs. 0.048), suggesting newer models develop more explicit safety representations.
Llama 405B (The Bias Problem): Despite high sensitivity ( $d'=3.29$ ), Llama exhibited a strong refusal bias ( $c = -0.86$ ) and poor calibration (ECE = 0.216). It predicted refusal too frequently, resulting in the lowest overall accuracy (80.0%) despite good discrimination.
GPT-5.2: Showed lower accuracy (88.9%) and higher behavioral variability (75% consistency across paraphrases), suggesting more stochastic safety decisions.

D. Topic-Wise Analysis

Hardest Topic: Weapons-related queries were consistently the most difficult for introspection across all models (accuracy 85.6–91.9%). This is attributed to the overlap between legitimate uses (history, chemistry) and harmful applications.
Easiest Topic: Hate speech showed near-perfect introspection for Claude models (98.9–100%), likely due to heavy emphasis in safety training.

E. Confidence-Based Routing

The study demonstrated that confidence scores are actionable for safety routing, but only for well-calibrated models.

Sonnet 4.5: Restricting to high-confidence predictions ( $conf \ge 5$ ) yielded 98.3% accuracy with 85.4% coverage.
Llama 405B: High-confidence predictions yielded only 76.3% accuracy due to poor calibration, proving that high sensitivity alone is insufficient without calibration.

4. Key Contributions

SDT Framework for Introspection: First application of Signal Detection Theory to quantify LLM introspective sensitivity and bias, revealing that sensitivity degrades specifically at safety boundaries.
Mechanistic Insights: Identified that errors peak at "likely harmful" (Level 4) rather than "borderline" (Level 3) and that weapons queries are the hardest domain for introspection.
Generational & Architectural Analysis: Documented improvements in Claude 4.5 over 4 and highlighted the critical dissociation between sensitivity ( $d'$ ) and accuracy in open-source models (Llama) due to bias and calibration issues.
Practical Routing Protocol: Established that confidence-based routing is viable for well-calibrated models (achieving ~98% accuracy on high-confidence cases) but fails for overconfident or biased models.

5. Significance and Implications

Safety Training Effectiveness: The results suggest that safety training creates internal representations of harm that are accessible for introspection in clear-cut cases but remain implicit or variable at boundaries.
Deployment Strategy: For safety-critical applications, systems should rely on confidence-based routing. High-confidence predictions from well-calibrated models (like Sonnet 4.5) can be trusted, while low-confidence or high-uncertainty cases should be routed to human reviewers.
Model Selection: The study warns that high discrimination ability ( $d'$ ) does not guarantee accurate self-prediction; calibration is equally critical. Models with strong refusal biases (like Llama 405B in this study) may appear sensitive but provide unreliable predictions for routing.
Future Directions: The findings suggest that future safety training should focus on reducing variability at boundaries and improving calibration to make introspective signals more reliable.