💬 NLP

Safety-Aligned Weights Are Not Enough: Refusal-Teacher-Guided Finetuning Enhances Safety and Downstream Performance under Harmful Finetuning Attacks

This paper proposes a Refusal-Teacher-guided finetuning framework that improves both safety and downstream task performance under harmful finetuning attacks by directly training base models with safety guidance, rather than relying on the weak initialization provided by pre-aligned weights.

Original authors: Seokil Ham, Yubin Choi, Yujin Yang, Seungju Cho, Younghun Kim, Changick Kim

Published 2026-08-25

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Seokil Ham, Yubin Choi, Yujin Yang, Seungju Cho, Younghun Kim, Changick Kim

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the rapidly evolving world of artificial intelligence, a powerful new capability has emerged: the ability to take a massive, general-purpose language model and tailor it to specific needs using a user's own data. This service, often called "finetuning," allows companies and individuals to customize these digital minds for specialized tasks, from writing code to analyzing medical records. However, this flexibility comes with a hidden danger. If the data used for customization contains harmful instructions—such as requests to build weapons or generate hate speech—the model can learn to ignore its safety rules. This vulnerability, known as a "harmful finetuning attack," threatens to turn helpful tools into dangerous ones. The core challenge for developers is to find a way to let users customize their models without accidentally teaching them to be unsafe.

For some time, the standard solution to this problem was a two-step process. First, developers would take a base model and train it extensively on safety data, teaching it to refuse harmful requests. This created a "safety-aligned" model. Then, they would take this already-safe model and further train it on the user's specific data. Researchers from the Korea Advanced Institute of Science and Technology (KAIST) have now discovered that this traditional approach is flawed. They found that starting with a model that has already been heavily trained on safety rules actually makes it harder for the model to learn new tasks effectively. The safety training shifts the model's internal settings in a way that is not ideal for learning new skills, leading to a compromise where the model either performs poorly on the user's task or, worse, forgets its safety rules when exposed to bad data.

To solve this, the researchers proposed a different strategy that flips the script. Instead of starting with a model that has already been forced to be safe, they began with a raw, adaptable base model and trained it directly on both the user's data and safety data at the same time. However, simply mixing these two types of data together creates a new problem: the model gets confused because the goals of being helpful and being safe can sometimes pull in opposite directions. To resolve this conflict, the team introduced a "Refusal-Teacher." This is a separate, safety-trained model that acts as a guide. It does not do the learning itself; instead, it watches the learning process and provides two crucial services. First, it filters out the harmful instructions from the user's data before the main model sees them, preventing the bad data from causing damage. Second, it gently guides the model on how to handle safety without being too harsh, using soft examples rather than rigid rules to help the model learn smoothly.

The results of this new method are striking. In tests where the researchers introduced varying amounts of harmful data—ranging from none to half of the training set—their approach consistently outperformed existing methods. When the training data contained 50% harmful prompts, the traditional methods failed to keep the model safe, with harmful responses spiking to nearly 80% in some cases. In contrast, the new method kept harmful responses below 1% while still allowing the model to learn the user's task with high accuracy. This success held true across different types of tasks, such as solving math problems or analyzing news articles, and worked with several different model architectures. The researchers also found that this method reduced the internal "tension" or conflict that occurs when a model tries to learn two opposing things at once, leading to a more stable and efficient training process.

The study suggests that the key to safe customization lies not in pre-hardening the model with safety rules, but in keeping the model flexible and guiding it carefully during the learning process. By using a teacher to filter out poison and distill safety lessons, the system achieves a balance that was previously out of reach. The researchers demonstrated that this approach works even when the harmful data is disguised or comes from different sources, showing a robustness that current defenses lack. While the work is a significant step forward, it highlights that safety in artificial intelligence is not a one-time setting but a dynamic process that requires constant, intelligent guidance. The findings offer a clear path for service providers to offer customization without sacrificing the safety of the tools they deploy, ensuring that as these models become more specialized, they remain trustworthy.

Technical Summary: Refusal-Teacher-Guided Finetuning for Safe FaaS

Problem Statement

The paper addresses the vulnerability of Finetuning-as-a-Service (FaaS) to harmful finetuning attacks. In this scenario, users submit customization data containing a mixture of harmless prompts and harmful prompts (e.g., instructions for generating unsafe content). If a provider finetunes a Large Language Model (LLM) on this data, the model's safety alignment can degrade, leading to harmful outputs.

The authors identify a critical flaw in the prevailing two-stage pipeline used to mitigate this risk:

Alignment Stage: Pre-trained models are aligned with safety data.
Finetuning Stage: The resulting safety-aligned model is finetuned on user data.

The paper argues that this approach is suboptimal because safety-aligned weights provide a weak initialization for downstream task learning. This leads to a trade-off where the model either suffers from poor task performance or loses safety alignment after user finetuning. Furthermore, directly finetuning a base model on both user data and safety data simultaneously (without a specific mechanism) introduces gradient conflicts between safety and utility objectives, destabilizing training.

Methodology: Refusal-Teacher-Guided Finetuning

To overcome these limitations, the authors propose a paradigm shift: instead of finetuning safety-aligned weights, they propose finetuning base weights under explicit safety-teacher guidance. This is implemented via the Refusal-Teacher (Ref-Teacher)-guided finetuning framework, which consists of two stages:

1. Teacher Preparation Stage

The goal is to train a safety-aligned teacher model capable of distinguishing harmful from harmless prompts and generating soft refusal labels.

Refusal Feature: The framework leverages the "refusal feature," a vector encoding the rejection behavior of safety-aligned LLMs. It is defined as the mean difference between feature representations of harmful and harmless prompts at a specific layer $l$ .
Training Objective: The Ref-Teacher is trained using a combined loss:
- Safety-Alignment Loss ( $L_{align}$ ): Standard supervised fine-tuning loss pairing harmful prompts with refusal responses and harmless prompts with helpful responses.
- Regularization Loss ( $L_{regul}$ ): A loss term that enforces clearer separation between harmful and harmless prompt features relative to the refusal feature. It pushes the cosine similarity of harmful features toward 1 and harmless features toward -1.
Dynamic Update: Since a pre-aligned model may not be available to extract the refusal feature initially, the framework dynamically updates the refusal feature during training by accumulating prompt features in buffers.

2. Finetuning Stage

The base LLM is finetuned on user data and safety-alignment data under the frozen guidance of the Ref-Teacher. This stage employs two complementary mechanisms:

Alignment Distillation: Instead of hard labels, the Ref-Teacher generates soft refusal labels. The base model is trained to match the Ref-Teacher's safe behavior on safety-alignment data using a KL-divergence loss. This provides richer supervision and smoother loss surfaces, mitigating gradient conflicts between safety and utility.
Data Filtering: The Ref-Teacher uses its refusal feature to classify prompts in the user data. Prompts with a high cosine similarity to the refusal feature (exceeding a threshold $\tau$ ) are identified as harmful and excluded from the supervised finetuning loss (weighted by a binary indicator $\omega_i = 0$ ). This prevents harmful samples from destabilizing the training process.

The overall objective function combines the filtered supervised loss for user tasks and the alignment distillation loss for safety:
$L_{ft} = \frac{1}{N_{user}} \sum \omega_i \cdot \ell(x_i, y_i) + \alpha T^2 \cdot \frac{1}{N_{safety}} \sum KL(p^T_{t,i} || p^T_{s,i})$

Key Contributions

Paradigm Shift: The paper challenges the standard two-stage pipeline (Safety-Align $\to$ Finetune), demonstrating that it yields suboptimal safety and utility. It proposes shifting to base-weight finetuning under teacher guidance.
Ref-Teacher Framework: The introduction of a framework that dynamically trains a teacher model to perform Alignment Distillation (via soft labels) and Data Filtering (via refusal features) simultaneously.
Gradient Conflict Mitigation: By combining soft label distillation and data filtering, the method effectively reduces gradient conflicts between safety and utility objectives, leading to more stable training.
No Pre-aligned Requirement: The teacher preparation stage allows for the computation of refusal features and discriminative representations within a single training process, removing the dependency on a pre-existing safety-aligned model.

Experimental Results

The framework was evaluated across diverse settings, including varying harmful prompt ratios ( $p$ ), user data sizes, downstream tasks (GSM8K, SST2, AGNEWS, AlpacaEval), and model architectures (Llama3-8B, Gemma2-9B, Qwen2-7B).

Safety and Utility Trade-off: The proposed method consistently achieved the lowest Harmful Score (HS) and the highest Finetuning Accuracy (FA) across all harmful ratios compared to baselines (e.g., RepNoise, Vaccine, Booster, LDIFS, SafeInstruct).
Robustness: Even with high ratios of harmful data ( $p=0.5$ ), the method maintained low HS (e.g., 0.9% for Ref-Teacher variant vs. 71.3% for standard SFT) while preserving high task accuracy.
Generalization: The approach generalized well across different harmless task datasets and model backbones.
Ablation Studies:
- Alignment Distillation (AD) alone: Reduced gradient conflicts but failed to improve safety if harmful prompts remained.
- Data Filtering alone: Effectively removed harmful prompts and reduced conflicts but risked overfitting and lower accuracy due to reduced data volume.
- Combination: The synergy of both components was necessary to achieve optimal safety and utility.
Loss Smoothness: Alignment distillation reduced the gradient Lipschitz estimate, indicating a smoother loss surface and improved training stability.

Significance and Claims

The paper claims that safety-aligned weights are not sufficient as an initialization for downstream adaptation when user data may contain harmful prompts. The authors argue that the standard two-stage pipeline creates a bottleneck where safety and utility objectives interfere.

The significance of this work lies in demonstrating that finetuning base weights under explicit safety-teacher guidance is a superior paradigm. By decoupling the safety enforcement (via the teacher) from the base model's initialization, the framework preserves the base model's adaptability while dynamically enforcing safety. This approach effectively resolves the tension between maintaining safety alignment and achieving high performance on user-specific tasks, even in the presence of harmful finetuning attacks. The authors emphasize that their method does not require a pre-aligned model to start, making it a practical solution for FaaS providers.