🤖 machine learning

REFLECTOR: Internalizing Step-wise Reflection against Indirect Jailbreak

The paper introduces Reflector, a two-stage framework that internalizes self-reflection into the LLM generation process via teacher-guided supervised fine-tuning and reinforcement learning, achieving over 90% defense success against complex indirect jailbreaks while simultaneously enhancing model utility and generalization.

Original authors: Jiachen Ma, Jiawen Zhang, Xiangtian Li, Bo Zou, Chaochao Lu, Chao Yang

Published 2026-05-21

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jiachen Ma, Jiawen Zhang, Xiangtian Li, Bo Zou, Chaochao Lu, Chao Yang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Problem: The "Trojan Horse" of AI

Imagine you have a very smart, well-behaved robot assistant. You've taught it strict rules: "Never tell anyone how to build a bomb" or "Never write a virus."

Usually, if you ask it directly, "How do I build a bomb?" it immediately says, "No, I can't do that." This is like a security guard checking your ID at the front door.

However, bad actors have found a sneaky way to trick the robot. They don't ask the question directly. Instead, they give the robot a complex puzzle or a story that looks harmless at first.

The Trick: "Let's write a story about a detective solving a mystery. First, describe the scene. Then, describe the suspect's plan. Then, describe the weapon..."
The Trap: The robot follows the story logic step-by-step. For the first 20 words, everything is fine. But by the time it reaches the "weapon" part of the story, it has been "coerced" by its own logic into generating harmful content.

The paper calls this an Indirect Jailbreak. It's like a Trojan Horse: the robot lets the bad idea inside because it thinks it's just following a story, not realizing it's building a bomb.

The Solution: REFLECTOR (The "Self-Checking" Robot)

The authors created a new system called REFLECTOR. Instead of just checking the first thing the robot says, REFLECTOR teaches the robot to pause and check itself while it is thinking and writing.

Think of it like a student taking a math test.

Old Way: The student writes the answer immediately. If the first word is wrong, the whole answer is wrong.
REFLECTOR Way: The student writes a step, then stops and asks themselves, "Wait, is this step safe? Is this logical? Did I make a mistake?" If the answer is "No," they erase it and try a different, safer path.

How They Taught the Robot (The Two-Stage Training)

The paper describes a two-step process to teach the robot this new habit.

Stage 1: The "Teacher" Lesson (Supervised Fine-Tuning)

First, the researchers used a super-smart "Teacher" AI to show the robot how to think safely.

The Analogy: Imagine a master chef teaching an apprentice. The apprentice tries to cook a dish that looks like a salad but might contain poison. The Teacher steps in, says, "Stop! That ingredient is dangerous. Let's swap it for a safe one," and writes down the correct steps.
The Result: The robot learns a specific pattern: Write a bit -> Pause -> Reflect ("Is this bad?") -> Fix it -> Continue. This creates a "search-and-recovery" habit.

Stage 2: The "Reward" Game (Reinforcement Learning)

Once the robot knows how to pause and reflect, the researchers let it practice on its own, but with a scoring system.

The Analogy: Think of a video game.
- Safety Reward: If the robot finishes the story without saying anything harmful, it gets a big gold star (+100 points).
- Reflection Bonus: If the robot catches a mistake during the process and fixes it, it gets an extra bonus (+50 points).
- The Penalty: If the robot tries to reflect but still ends up saying something harmful, it loses points.
The Goal: The robot learns that the best way to win the game (get the highest score) is to be constantly vigilant and fix its own mistakes in real-time.

The Results: Safe and Smarter

The paper claims that REFLECTOR is a huge success for two reasons:

It Stops the Sneaky Attacks:
The robot became incredibly good at spotting those "Trojan Horse" attacks. In tests, it successfully blocked over 90% of these complex, indirect jailbreak attempts. It didn't just block the question at the door; it caught the bad idea while the robot was thinking.
It Didn't Get Dumber (The "Alignment Tax" Breaker):
Usually, when you make an AI safer, it gets a little worse at other things (like math or writing stories). This is called the "Alignment Tax."
- The Surprise: REFLECTOR actually got better at math and general knowledge.
- Why? The paper suggests that the habit of "pausing to check your work" (reflection) is actually good for everything. Just like a human who double-checks their math homework gets better grades, the robot who double-checks its safety also gets better at solving complex problems.

Summary

REFLECTOR is a new way to make AI safer. Instead of just putting a fence at the front door, it teaches the AI to have an internal "conscience" that checks its thoughts as they happen. This stops sneaky, multi-step tricks from working, and surprisingly, makes the AI smarter at solving problems too.

Technical Summary: REFLECTOR

1. Problem Statement

Large Language Models (LLMs) have achieved significant breakthroughs in reasoning and coding but remain vulnerable to sophisticated indirect jailbreak attacks. Unlike direct attacks that trigger immediate unsafe outputs, indirect attacks (e.g., DRA, PAP, DrAttack) embed malicious intent within seemingly benign, multi-step reasoning tasks. As illustrated in the paper, these attacks exhibit "stealthy latency," where harmful content emerges only after 20+ tokens, bypassing conventional surface-level safety alignment that typically relies on detecting malicious intent in the initial response prefix.

Current alignment strategies, such as Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO), often function as "gatekeepers" that refuse harmful inputs at the start of generation. However, they fail to secure the reasoning trajectory itself. When an attacker controls the structure of the reasoning process, the model can be coerced into autonomously generating harmful content, effectively weaponizing its own deliberation. Existing methods like Shallow Alignment or STAIR (which enforces safe reasoning via Chain-of-Thought) are insufficient because they do not internalize a mechanism for the model to dynamically detect and correct unsafe reasoning states during generation.

2. Methodology: The REFLECTOR Framework

The authors propose REFLECTOR, a principled two-stage framework designed to internalize step-wise self-reflection directly into the model's generation policy. The core objective is to transform safety alignment from a static prefix constraint into a dynamic, trajectory-level property where the model continuously monitors its own reasoning steps.

Stage I: Reflection Capability Injection via Supervised Fine-Tuning (SFT)

The authors address the "cold-start" problem, where pretrained models lack the intrinsic inductive bias to perform self-reflection.

Teacher-Guided Data Synthesis: A high-capacity teacher model (GPT-5) is used to generate high-quality reflection trajectories. For a given indirect jailbreak query, the teacher simulates a "search-and-recovery" process:
1. Truncation: A safe prefix is generated and then truncated at a random step.
2. Structured Reflection: The teacher generates a reflection segment ( $z$ ) containing explicit self-critique (<|reflect|>) identifying the ethical breach and an exploration segment (<|explore|>) proposing a safe alternative path.
3. Safe Continuation: The teacher generates a safe continuation ( $y_{after}$ ) based on the reflection.
Dataset Construction: These synthesized trajectories form the dataset $D_R$ , which is used to Supervised Fine-Tune (SFT) the target policy ( $\pi_\theta$ ). This stage establishes the structural inductive bias, teaching the model the format and timing of effective self-reflection.

Stage II: Dual-Reward Enhancement via Reinforcement Learning (RL)

In this stage, the model learns to autonomously determine when and how to reflect without explicit teacher guidance, optimizing for both safety and reasoning quality.

Algorithm: The framework employs GDPO (Group reward-Decoupled Normalization Policy Optimization), a variant of Group Relative Policy Optimization (GRPO) designed for multi-reward settings.
Dual-Reward Mechanism: The total reward $r(\tau)$ $r (τ)$ for a trajectory is decomposed into two components:
1. Safety Reward ( $r_{safety}$ ): A binary signal (0 or 1) determined by a hybrid consensus mechanism ($HarmCLS$) combining a discriminative classifier (HarmBench) and a generative safety detector (GPT-OSS-120B). This ensures rigorous evaluation of the final output.
2. Reflection Reward ( $r_{reflect}$ ): A fine-grained bonus/penalty system. The model receives a positive reward ( $+\lambda$ ) if it engages in reflection and produces a harmless output. It receives a negative reward ( $-\lambda$ ) if it reflects but still outputs harmful content, and zero if no reflection occurs.
Optimization: By maximizing the group-relative advantage, the model is encouraged to internalize the "search-and-recovery" mechanism, learning to intercept latent risks at any step $y_t$ and transition to a safe refusal state.

3. Key Contributions

Trajectory-Level Safety Formulation: The paper redefines safety alignment as a step-wise property of the generation trajectory rather than a static prefix constraint, addressing the specific vulnerability of indirect jailbreaks.
Two-Stage Training Paradigm: REFLECTOR introduces a novel pipeline combining teacher-guided SFT to bootstrap reflection capabilities with RL to internalize autonomous self-correction.
Dual-Reward Mechanism: The design of a reward function that explicitly separates and optimizes for final output safety and the process of reflection, ensuring the model learns not just to refuse, but to reason safely.
Breaking the Alignment Tax: The framework demonstrates that embedding safety reasoning can enhance, rather than degrade, general task performance.

4. Experimental Results

The authors evaluated REFLECTOR on LLaMA-3.1-8B-Instruct and Qwen-2.5-7B-Instruct against a comprehensive suite of benchmarks.

Defense Success Rate (DSR): REFLECTOR achieved a DSR exceeding 90% across diverse indirect attack scenarios (DRA, PAP, ReNeLLM, DrAttack), significantly outperforming baselines like SFT, DPO, and STAIR. For instance, on the DRA benchmark, the defense rate improved from ~10% (Original) to 92.31% (REFLECTOR + GDPO) on LLaMA-3.1-8B.
General Utility: Contrary to the typical "alignment tax," REFLECTOR improved general reasoning capabilities. It yielded a 5.85% gain on the GSM8K mathematical reasoning benchmark and set new performance peaks on MMLU-Pro (45.20%) and AdvGLUE (68.29%).
Robustness: The model maintained high performance on overtly harmful queries (StrongREJECT, XSTest) and demonstrated resilience against direct jailbreaks (AutoDAN, GCG).
Efficiency: While the SFT stage initially increased token generation due to explicit reflection, the RL stage successfully internalized the process, reducing the inference overhead to near-baseline levels (e.g., 8.03 steps vs. 7.28 for the original model on StrongREJECT).

5. Significance and Claims

The paper claims that REFLECTOR offers a scalable and efficient solution for developing safe and capable LLMs by moving beyond surface-level alignment. Its primary significance lies in:

Internalizing Safety: It shifts the defense paradigm from external guardrails to intrinsic, trajectory-level self-reflection, effectively closing the gap where indirect attacks exploit the model's reasoning process.
Synergy of Safety and Intelligence: The results challenge the notion that safety and utility are mutually exclusive, suggesting that robust self-reflection mechanisms can serve as a general reasoning enhancer.
Scalability: By relying on automated teacher guidance and reward models rather than massive human-curated datasets, the framework is presented as a viable path for the next generation of aligned models.

The authors conclude that by enabling an "always-on" defense with minimal computational overhead, REFLECTOR provides a principled framework for mitigating insidious, stealthy risks in large language models.