🤖 machine learning

Do Thinking Tokens Help with Safety?

This paper challenges the assumption that thinking tokens in reasoning models enable genuine safety deliberation, revealing instead that refusal or compliance outcomes are largely determined before visible thinking begins, with the thinking process often serving as mere prefix completion rather than substantive revision.

Original authors: Narutatsu Ri, Abhishek Panigrahi, Sanjeev Arora

Published 2026-06-25

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Narutatsu Ri, Abhishek Panigrahi, Sanjeev Arora

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, well-trained robot assistant. You ask it a question, and before it answers, it has a special "thinking phase" where it writes down a long internal monologue (a "thinking trace") to figure out the best response.

The big hope in the AI world was that this thinking phase acts like a safety committee meeting. The idea is: "Let's pause, think hard, and ask ourselves, 'Is this request dangerous? Should I say no?'" If the robot thinks long enough, it should be safer and smarter about refusing bad requests while still helping with good ones.

This paper, however, pulls back the curtain and says: That's not really what's happening.

Here is the breakdown of their findings using simple analogies:

1. The Decision is Made Before the Meeting Starts

The researchers found that the robot's final decision (to say "Yes" or "No") is actually locked in before it even writes the first word of its thinking process.

The Analogy: Imagine a judge who has already decided the verdict before the trial begins. The lawyers (the thinking tokens) come in and argue their case, but the judge's mind is already made up.
The Evidence: They looked at the robot's "brain state" (hidden representation) at the very first moment it starts thinking. They could predict with 84% to 95% accuracy whether the robot would eventually refuse or comply, just by looking at that single first moment. The actual words written during the "thinking" phase didn't change the outcome; they were just the robot talking itself through a decision it had already reached.

2. The "Thinking" is Just a Script, Not a Debate

The paper argues that the thinking trace isn't a place where the robot re-evaluates its safety. Instead, it's more like reading a script or filling in the blanks.

The Analogy: Think of a movie actor who has already memorized the ending of the scene. Even if the script says the character is "struggling with a moral dilemma," the actor is just acting out the struggle because the director (the model's training) told them to. The struggle doesn't actually change the ending; the ending was written in the first draft.
The Evidence: When the researchers stopped the robot's thinking process early (after just 20% of the text) and forced it to finish, the result was almost always the same as if it had thought for the full time. The "thinking" rarely changed the robot's mind. In fact, about 74% of the time the robot seemed to be debating a safety issue in its text, its internal decision was already locked in one way or the other.

3. Current Safety Fixes Are Just Making the Robot "Over-Refuse"

The paper tested many different methods people use to try to make these robots safer (like adding safety reminders or retraining them).

The Analogy: Imagine trying to fix a car that drives too fast by putting a giant "STOP" sign on the dashboard. It stops the car from speeding, but now it also refuses to drive at all, even when you just want to go to the grocery store.
The Evidence: Most of these safety methods didn't actually make the robot think better about safety. Instead, they just made the robot refuse more often, even when the request was harmless. They didn't fix the underlying issue (that the robot doesn't actually deliberate); they just pushed the robot toward a "better safe than sorry" attitude, which annoys users with harmless requests.

The Bottom Line

The common belief is that "thinking" gives AI a safe space to reconsider its actions. This paper suggests that for current models, thinking is mostly an illusion.

The robot decides whether to say "Yes" or "No" almost instantly, and the long, thoughtful text it generates afterward is just a post-hoc rationalization—a fancy way of explaining a decision it made before it started talking. To make AI truly safe, we can't just tell it to "think harder"; we need to teach it to actually change its mind based on that thinking, rather than just acting out a script it already knows.

Technical Summary: Do Thinking Tokens Help with Safety?

Problem Statement

Large Reasoning Models (LRMs) generate extended "thinking traces" before producing a final response, a mechanism credited with improving performance on complex tasks like math and coding. A prevailing hypothesis suggests that this deliberative mode should similarly enhance safety by providing the model a "safe space" to evaluate whether a request violates safety principles before committing to an answer. This intuition underpins many rule-based and specification-based safety defenses. However, recent work indicates that safety alignment in standard instruction-tuned models is often shallow, with behavior determined by early tokens and easily circumvented. The central question this paper addresses is whether the extended thinking process in LRMs genuinely improves safety decision-making or if it merely serves as post-hoc rationalization for decisions already made early in the generation process.

Methodology

The authors evaluate four frontier open-weight reasoning models: Qwen3-8B, Olmo-3-7B-Think, Phi-4-Reasoning, and GPT-OSS-20B. The study employs a multi-faceted approach:

Decodability Analysis: The authors probe the hidden representations of the first thinking token ( $h_0$ ) to determine if the final refusal/compliance outcome is already encoded before any visible thinking occurs. They use a linear probe (logistic regression) on the first token's representation and measure performance via AUROC and Balanced Accuracy (BAcc). They also compute the Fisher discriminant ratio ( $J(t)$ ) across the entire thinking trace to track the separability of refusal and compliance groups over time.
Continuation Variance: To test if thinking alters the decision trajectory, the authors generate multiple independent thinking traces for a single prompt, truncating them at various percentages (20% to 100%) of their natural length. They then sample final responses conditioned on these prefixes to measure the variance in outcomes. Low variance implies the prefix locks the decision.
Oscillation Analysis: The authors analyze sentence-level segments of the thinking trace using frontier LLMs (GPT-5.4, Gemini 3 Pro, Sonnet 4.6) to assign stance labels (refusal-leaning, neutral, compliance-leaning). They identify "oscillations" (changes in stance) and determine if these changes are "meaningful" (statistically shifting the final response distribution) or "performative" (textual changes with no impact on the final decision).
Defense Evaluation: The study evaluates existing inference-time defenses (e.g., SafePath ZS, PSR, SafeRemind) and training-based defenses (e.g., STAR-1, R1-ACT, ThinkSafe, STAIR, RAPO) to see if they improve the Attack Success Rate (ASR) and Over-Refusal Rate (ORR) tradeoff or increase the deliberative nature of the thinking trace.

Key Contributions and Results

1. Safety Decisions are Decodable Before Visible Reasoning

The most striking finding is that the refusal/compliance outcome is strongly predictable from the hidden representation of the first thinking token ( $h_0$ ), before any substantial reasoning text is generated.

Metrics: Linear probes on $h_0$ achieve AUROC scores between 0.84 and 0.95 and Balanced Accuracy between 76% and 88% across all models.
Contrast: Probes based solely on the surface text of the first token perform near chance (AUROC ~0.5), indicating the signal is encoded in the latent representation, not the token identity.
Trajectory: The Fisher discriminant ratio shows a "U-shape" or "valley": high separability at the start ( $t=0$ ), dropping through the middle of the trace, and recovering at the end. This suggests the decision is consolidated early and inherited by the rest of the trace.

2. Thinking Acts as Prefix Completion, Not Deliberative Revision

The thinking process rarely steers the model away from the trajectory established by the initial prefix.

Continuation Variance: Even when truncating the thinking trace at just 20% of its length, the variance in final outcomes is near zero (<0.2). Later thinking tokens almost never change a refusal to a compliance or vice versa.
ASR-ORR Tradeoff: Enabling thinking does not consistently improve both ASR and ORR. Instead, it shifts models along the tradeoff curve (e.g., reducing ASR often increases ORR, or vice versa), rather than moving the Pareto frontier toward the ideal low-ASR/low-ORR region.

3. Most "Deliberation" is Performative

While thinking traces often contain text that appears to deliberate (oscillating between refusal and compliance stances), these segments have minimal causal impact on the final decision.

Performative vs. Meaningful: Of the oscillations found, 71–92% are "performative," meaning the text changes stance but the final response distribution remains locked. Only a small fraction of segments are "meaningful" (statistically significant shifts).
Locked State: Approximately 73–77% of all oscillations occur when the model's decision is already "locked" (probability >0.95 or <0.05), rendering the textual deliberation irrelevant to the outcome.

4. Existing Defenses Fail to Induce True Deliberation

The study evaluates various safety interventions and finds they largely fail to induce genuine safety deliberation.

Inference-Time Defenses: Methods like SafePath ZS and PSR generally reduce ASR at the cost of higher ORR (over-refusal). Crucially, they suppress the already weak deliberation signals, reducing the number of meaningful oscillations by up to 95% in some models.
Training-Based Defenses: Methods like STAR-1 and ThinkSafe induce larger behavioral shifts but similarly push models toward over-refusal. They do not consistently increase the deliberative quality of the thinking trace; some even reduce meaningful oscillations while preserving the early decodability of the refusal signal.

Significance and Claims

The paper argues that the current paradigm of reasoning models does not utilize thinking tokens for the safety deliberation commonly assumed. Instead, safety behavior in these models is:

Pre-determined: Encoded in the hidden state of the first thinking token.
Non-deliberative: The subsequent thinking trace serves largely as a rationalization of an early decision rather than a revision process.
Vulnerable to Misinterpretation: Current defenses often mistake the appearance of deliberation (textual oscillation) for actual safety reasoning, and many interventions inadvertently suppress the few existing deliberation signals.

The authors conclude that safety behavior in current LRMs is "much less deliberative than commonly assumed." They highlight the need for new training objectives that reward real safety deliberation rather than merely encouraging the generation of safety-flavored reasoning text. The work suggests that simply extending thinking time or inserting safety reminders is insufficient; the mechanism of decision-making itself must be altered to allow the thinking process to genuinely influence the safety outcome.