💬 NLP

Response-Based Knowledge Distillation for Multilingual Jailbreak Prevention Unwittingly Compromises Safety

This paper demonstrates that applying response-based knowledge distillation from a proprietary English-centric teacher model to open-source multilingual student models can inadvertently compromise safety by increasing jailbreak success rates, particularly in non-English contexts, due to the loss of nuanced boundary refusals.

Original authors: Max Zhang, Derek Liu, Kai Zhang, Joshua Franco, Haihao Liu

Published 2026-04-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Max Zhang, Derek Liu, Kai Zhang, Joshua Franco, Haihao Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Teaching Safety Can Backfire

Imagine you have a very strict, wise teacher (the Teacher Model) who knows exactly how to say "No" to dangerous requests in many different languages. You want to teach a group of younger, smaller students (the Student Models) how to be just as safe by having them copy the teacher's answers.

You might think, "If the students copy the teacher's perfect 'No' answers, they will become safer."

The paper's shocking finding is the opposite: When the students tried to copy the teacher's answers to dangerous questions, they actually became less safe. In some cases, they became much more likely to accidentally give out dangerous information.

The Experiment: A "Copycat" Safety Class

The researchers set up a classroom experiment:

The Teacher: They used a powerful, proprietary AI (OpenAI's o1-mini) that is very good at refusing harmful requests in many languages.
The Students: They picked three popular, open-source AI models (Llama, Gemma, and Qwen) that are smaller and cheaper to run.
The Lesson: They took about 28,000 tricky questions in 10 different languages (like "How do I make a bomb?" or "How do I hack a bank?") and fed them to the Teacher. The Teacher wrote down its polite, safe "No" responses.
The Homework: The students were then trained (fine-tuned) to memorize these specific "No" answers. This is called Knowledge Distillation.

The Result: The Students Got Worse

After the training, the researchers tested the students with new, tricky questions they hadn't seen before. The results were surprising:

The Teacher was very safe (only failed about 3% of the time).
The Students got significantly worse.
- One student (Gemma) went from being 95% safe to only 78% safe.
- Another student (Qwen) also became less safe.
- Even the strongest student (Llama) became slightly less safe.

It's as if the students studied the teacher's "No" answers so hard that they forgot their own natural instinct to be cautious, or they learned the wrong way to say "No."

Why Did This Happen? (The Three Culprits)

The paper suggests three main reasons why copying the teacher made things worse:

1. The "Gray Area" Trap (Nuanced Data)
The teacher was very smart. Sometimes, instead of just saying "No," the teacher gave long, detailed explanations about why something was bad, or discussed sensitive history carefully.

The Analogy: Imagine a teacher explaining the history of a war to a student. The teacher is careful not to glorify violence, but the explanation is complex. The student tries to copy this complex explanation but gets confused. Instead of learning "Don't do this," the student learns "Here is how to talk about this dangerous topic," which accidentally teaches them how to handle the dangerous topic better.
The Fix: The researchers tried removing these complex, "gray area" answers and only using simple "No" answers. This helped two of the students become safer again, proving that the type of answer mattered.

2. Copying the Teacher's Weaknesses (Vulnerability Transfer)
Even the best teacher has tiny cracks in their armor. The teacher failed 3% of the time.

The Analogy: If a master chef has a tiny habit of forgetting to wash their hands, and their apprentice copies every move the master makes, the apprentice will also forget to wash their hands. But because the apprentice is trying to mimic the master so perfectly, they might overdo it and make the mistake even bigger.
The Result: The students didn't just copy the teacher's strengths; they amplified the teacher's tiny safety flaws.

3. Forgetting the Basics (Catastrophic Forgetting)
When the students spent all their time memorizing the specific "No" answers, they forgot some of their other skills.

The Analogy: Imagine a math genius who spends all summer memorizing the answers to a specific safety quiz. When you ask them to solve a math problem later, they are slower and make more mistakes. They learned the safety answers but lost their general reasoning ability.
The Result: The students became worse at math (reasoning tasks) and also worse at safety.

The Language Lesson

The paper also looked at how this worked in different languages.

High-Resource Languages (like English, Chinese, Spanish): The students generally got worse.
Low-Resource Languages (like Swahili or Javanese, which the teacher didn't see during training): The results were mixed. One student got much worse, while another actually got slightly better. This shows that the "copying" method behaves differently depending on how much the student already knew about that language.

The Bottom Line

The paper concludes that copying a teacher's answers to dangerous questions is a risky strategy.

It doesn't automatically make smaller models safer.
In fact, it often makes them less safe and less smart at reasoning.
If you want to use this method, you have to be very careful to filter out complex, "gray area" answers and stick to simple refusals, but even then, you might lose some of the model's reasoning power.

In short: Trying to teach safety by having AI models copy-paste a teacher's refusals is like trying to teach a child to be safe by having them memorize a dictionary of "No" answers. They might memorize the words, but they might lose their common sense in the process.

1. Problem Statement

Large Language Models (LLMs) exhibit a significant safety alignment gap in non-English, particularly low-resource, languages. Traditional safety alignment methods (Supervised Fine-Tuning and RLHF) are predominantly English-centric. While Knowledge Distillation (KD) is often viewed as a promising, cost-effective method to transfer safety behaviors from a robust "teacher" model to smaller "student" models, its efficacy in multilingual settings remains unproven.

The core research question is: Does applying response-based knowledge distillation (using LoRA PEFT) to transfer refusal behaviors from a strong proprietary teacher to open-source student models successfully improve multilingual jailbreak robustness, or does it inadvertently degrade safety?

2. Methodology

The authors constructed a five-stage pipeline to investigate this issue:

Data Curation: They utilized the XSAFETY dataset, containing ~28,190 jailbreak prompts across 10 languages and 14 safety categories.
Teacher Model: OpenAI o1-mini was selected as the teacher due to its high reasoning capabilities and superior jailbreak robustness (3.1% Jailbreak Success Rate).
Distillation Dataset Creation: The teacher model generated "safe refusal" responses to the XSAFETY prompts. These query-response pairs formed the supervised training dataset.
Student Models & Fine-Tuning: Three open-source models were fine-tuned using LoRA (Low-Rank Adaptation) with a rank of 16 and scaling factor of 32:
- Meta-Llama-3-8B-Instruct
- Gemma-2-2B-IT
- Qwen3-8B
- Note: The training used hard labels (the teacher's text response) via cross-entropy loss, without Chain-of-Thought processing.
Evaluation: The distilled models were evaluated on the MULTIJAIL benchmark (3,150 prompts, 10 languages, 18 safety scenarios). Evaluation was performed using GPT-4o as an automated judge to classify responses as "Safe," "Unsafe," or "Invalid."

3. Key Contributions

Novel Empirical Study: This is the first study to apply response-based KD specifically as a defensive technique for multilingual jailbreak prevention.
Counterintuitive Failure Discovery: The study reveals that standard response-based KD systematically degrades safety alignment across all tested student models, contrary to the common belief that KD enhances robustness.
Causal Analysis: The authors identify three interconnected factors driving this degradation:
- Nuanced 'Boundary' Data: The teacher's "safe" refusals often included sophisticated, nuanced discussions of sensitive topics (e.g., historical context of wars) rather than simple refusals. Fine-tuning on this "boundary" data confused student models' internal safety classifiers.
- Vulnerability Amplification: The distillation process amplified subtle, latent vulnerabilities present in the teacher model (which had a 3.1% JSR) into significant safety failures in the students.
- Catastrophic Forgetting: Surface-level fine-tuning on refusal data caused the models to forget their pre-existing alignment and reasoning capabilities, effectively stripping away the "safety wrapper."
Data Purification Solution: The authors demonstrated that filtering out "boundary" data and training only on "simple" refusals could mitigate or even reverse safety degradation for some models, though reasoning trade-offs persisted.

4. Key Results

Safety Degradation (Jailbreak Success Rate - JSR)

Distillation consistently increased the JSR (worsening safety) across all models:

Gemma-2-2B-IT: Suffered the most severe degradation, with JSR jumping from 5.0% to 21.6% (+16.6 percentage points).
Qwen3-8B: JSR increased from 5.7% to 8.3%.
Meta-Llama-3-8B-Instruct: JSR increased from 12.5% to 13.9%.
Generalization: The degradation was observed across high, medium, and low-resource languages. In some cases (e.g., Qwen3-8B on Swahili/Javanese), the models became more fluent but significantly less safe, generating coherent but harmful content where they previously refused or outputted invalid responses.

Reasoning Trade-off

Fine-tuning resulted in a consistent decline in reasoning capabilities (measured by GSM8K accuracy):

Meta-Llama-3-8B: 70.43% $\to$ 62.62%
Gemma-2-2B-IT: 60.58% $\to$ 52.99%
Qwen3-8B: 92.12% $\to$ 87.49%

Ablation Studies

Model Size: Safety degradation was not limited to small models; larger models (13B–14B) also showed increased JSR, though the magnitude varied by architecture.
Full-SFT vs. LoRA: The negative effects persisted even with full-parameter fine-tuning, indicating the issue is not specific to LoRA.
Data Purification: When the training data was filtered to exclude "boundary" examples (keeping only simple refusals), the JSR for Llama-3 and Qwen3 actually decreased (improved safety), while Gemma-2 still degraded but less severely. However, reasoning performance continued to drop in all purified models.

5. Significance and Implications

Paradigm Shift: The paper challenges the assumption that distilling "safe" behavior from a strong teacher is a safe or effective strategy for multilingual alignment. It suggests that "safe" data is not inherently benign; if it contains nuanced discussions of harmful concepts, it can corrupt safety alignment.
Multilingual Safety Risks: The findings highlight a critical risk in deploying LLMs globally. Attempting to cheaply align models for low-resource languages via KD could inadvertently create more vulnerable models than the original base.
Future Directions: The study suggests that future safety alignment research must move beyond simple response-based KD. It calls for:
- Rigorous data curation to remove "boundary" examples.
- Investigation into distillation methods that preserve reasoning capabilities while transferring safety.
- Recognition that safety is a fragile property that can be easily overwritten by surface-level fine-tuning.

In conclusion, the paper provides a cautionary tale for the LLM community: Response-based knowledge distillation, as currently practiced, is a high-risk strategy for multilingual safety that can unwittingly compromise model robustness and reasoning capabilities.