💬 NLP

Safety Measurements for Fine-tuned LLMs Should be Grounded in Capability

This paper argues that evaluating the safety of fine-tuned large language models requires grounding assessments in specific capability goals rather than arbitrary settings, as this approach reveals critical issues such as incoherent model outputs, the unreliability of automated safety judgments for such cases, and the significant variability of safety conclusions based on benchmark and evaluator choices.

Original authors: Krishnapriya Vishnubhotla, Hillary Dawkins, Isar Nejadgholi, Svetlana Kiritchenko

Published 2026-06-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Krishnapriya Vishnubhotla, Hillary Dawkins, Isar Nejadgholi, Svetlana Kiritchenko

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, well-behaved robot assistant (a Large Language Model) that has been trained to be helpful but also to refuse dangerous requests, like "How do I build a bomb?" or "How do I hack a bank?" This robot is your "base model."

Now, imagine you want to teach this robot a specific new job, like solving math problems or answering science quizzes. You do this by "fine-tuning" it—giving it a massive stack of practice worksheets. The paper argues that while you are trying to make the robot better at math, you might accidentally break its safety guardrails, or worse, break its ability to speak clearly.

Here is the breakdown of the paper's findings using simple analogies:

1. The "Arbitrary Settings" Problem

The Analogy: Imagine two chefs trying to improve a recipe. Chef A changes the oven temperature by 5 degrees and cooks for 10 minutes. Chef B changes the temperature by 50 degrees and cooks for 2 hours. They both claim their method is "safe" because the food didn't burn in their specific test. But because they used such different settings, you can't really compare them.

The Paper's Point: Previous research on making fine-tuned models safe used random, inconsistent settings (like different "oven temperatures" or training times). This made it hard to know what was actually working. The authors say we need to anchor these tests to a specific goal (like "solve math problems correctly") rather than just guessing random settings.

2. The "Robot Becomes a Nonsense Machine"

The Analogy: Imagine you train a parrot to only answer "Yes" or "No" to questions. After weeks of this, you ask the parrot, "Is the sky blue?" and it says "Yes." But then you ask, "How do I make a bomb?" and the parrot, stuck in its new habit, just squawks "Yes" or "No" randomly, or just repeats "Yes" over and over. It's not that the parrot wants to give a dangerous answer; it's just that it forgot how to speak in full sentences.

The Paper's Point: When models are fine-tuned on tasks with strict formats (like multiple-choice questions or Yes/No answers), they sometimes lose the ability to generate coherent sentences. When asked safety questions, they might produce gibberish.

The Danger: Automated safety checkers (like a robot judge) look at this gibberish and get confused. They might think the nonsense is "unsafe" when it's actually just "broken," or they might miss real dangers because the output is so weird.

3. The "Two Different Judges" Problem

The Analogy: Imagine you are grading a student's essay. Judge A says, "If the student didn't write a full paragraph, they failed." Judge B says, "If the student didn't say 'No' to a bad question, they failed." You could have a student who is actually safe but gets a failing grade from Judge A, and a different student who is unsafe but gets a passing grade from Judge B.

The Paper's Point: The paper tested safety using two different "judges":

Judge 1 (Refusal): Did the model say "No" to the bad question?
Judge 2 (Harmfulness): Did the model say something dangerous, even if it didn't say "No"?
They found these judges often disagreed. A model might stop saying "No" (which Judge 1 hates) but actually start giving helpful, safe explanations (which Judge 2 likes). Depending on which judge you use, you might conclude the model is "safe" or "unsafe" just by chance.

4. The "Safety vs. Skill" Trade-off

The Analogy: Think of a safety-preserving method (like SafeLoRA) as a "seatbelt" for the robot. You put the seatbelt on to make sure the robot doesn't crash while learning a new skill.

The Result: The seatbelt works! The robot is safer. But, the seatbelt is a bit heavy and stiff. The robot can still solve the math problems, but it takes a little longer or gets slightly fewer of them right.
The Catch: For some robots (specific models), the seatbelt works great. For others, the seatbelt is so heavy it actually makes the robot trip and fall (safety gets worse, or skills drop significantly).

The Paper's Point: The authors tested a method called SafeLoRA (which tries to keep the robot safe while learning). They found:

It usually makes the model safer.
But it almost always makes the model slightly worse at the actual task (lower accuracy).
The results varied wildly depending on which robot model and which dataset you used.

The Big Takeaway

The paper concludes that measuring safety in AI is currently like trying to measure the weather with a broken thermometer.

Don't trust random tests: You need to tie safety tests to specific, real-world goals (like "can it solve math?").
Watch out for gibberish: If a model starts talking nonsense after training, safety checkers can't be trusted.
One size doesn't fit all: A safety method that works for one model might fail for another, and different "safety judges" will give you different answers.

The authors aren't saying we should stop trying to make AI safe. They are saying we need better, more consistent, and more grounded ways to measure if we are actually succeeding.

Technical Summary: Safety Measurements for Fine-tuned LLMs Should be Grounded in Capability

Problem Statement

Adapting foundation Large Language Models (LLMs) to specific user tasks or styles via supervised fine-tuning (SFT) or domain adaptation often compromises the safety features of the original models. Recent research indicates that even fine-tuning on benign datasets can unintentionally degrade safety guardrails. While various mitigation methods (e.g., SafeLoRA, SaLoRA) have been proposed to preserve safety during fine-tuning, previous evaluations of these methods suffer from significant limitations:

Arbitrary Experimental Settings: Prior works often define empirical settings (hyperparameters, layers targeted) without anchoring them to specific capability goals, leading to inconsistent comparisons.
Evaluation Brittleness: Automated safety judgments are increasingly shown to be unreliable, particularly when models generate incoherent outputs.
Inconsistent Conclusions: The assessment of whether fine-tuning improves or degrades safety varies drastically depending on the choice of safety benchmark and the specific safety evaluator model used.

The authors argue that without grounding fine-tuning experiments in verifiable capability goals, it is impossible to draw meaningful conclusions about safety impacts or to fairly compare mitigation strategies.

Methodology

The authors conduct a multi-dimensional evaluation focusing on the interplay between capability (task performance and generation coherence) and safety.

Experimental Setup

Models: Four open-source instruction-tuned LLMs were used: Llama-3.2-1B, Llama-3.1-8B, Qwen-3-4B, and Qwen-3-8B.
Fine-tuning Tasks: To ensure verifiable capability measurements, the authors selected four datasets with objective correctness metrics:
- GSM8k: Grade school math word problems.
- ARC: Grade school science multiple-choice questions.
- Science10k: Science QA pairs with reasoning.
- BoolQ: Reading comprehension with boolean (yes/no) answers.
- Note: A subset of the Stanford Alpaca dataset was also used for general instruction-following comparisons.
Fine-tuning Method: Low-Rank Adaptation (LoRA) was applied with varying hyperparameters (rank $\{16, 64\}$ , target layers $\{[qv], [all-linear]\}$ ).
Mitigation Method: The study specifically analyzes SafeLoRA, a training-free method that projects LoRA weight updates onto a "safety subspace" (defined as the weight difference between aligned and unaligned models) if the similarity score falls below a threshold.

Evaluation Dimensions

Capability: Measured via accuracy on the test splits of the fine-tuning datasets.
Coherence: Measured via perplexity of generated outputs using a GPT-2 model. High perplexity indicates incoherent generations.
Safety: Evaluated using a comprehensive set of 1,840 safety prompts derived from four benchmarks (SORRY-Bench, BeaverTails-Eval, BeaverTails-Intent, XSTest).
- Judges: Two distinct evaluators were employed to test robustness:
  - LlamaGuard-8B: A content safety classifier outputting harmfulness rates.
  - SORRY-Bench Evaluator (Mistral-7b): A fine-tuned model judging refusal vs. compliance.

Key Results

1. Coherence and Automated Judgment Failure

Fine-tuning on tasks with restricted output formats (e.g., multiple-choice or yes/no) caused models to overfit, resulting in incoherent generations when presented with safety prompts.

Models fine-tuned on ARC and BoolQ often generated single-character or nonsensical responses to safety questions.
Critical Finding: Automated safety judges (specifically LlamaGuard) frequently misinterpreted these incoherent outputs as "unsafe" or "harmful." The paper demonstrates that lack of capability (incoherence) is often misclassified as a lack of safety by automated evaluators.

2. Instability of Safety Benchmarks

The conclusions regarding safety degradation or improvement are highly sensitive to the choice of benchmark and evaluator.

Benchmark Variance: For the same model checkpoint (e.g., Llama-8B fine-tuned on GSM8k), one benchmark (SORRY-Bench) might indicate a significant safety improvement over the base model, while another (BeaverTails-Intent) indicates a degradation.
Evaluator Divergence: There is often a weak or opposing correlation between "harmfulness" (LlamaGuard) and "compliance" (SORRY-Bench). Smaller models showed particularly divergent trends between these two metrics.

3. Safety-Capability Trade-offs (SafeLoRA Analysis)

The study compared standard LoRA against SafeLoRA:

Llama Models: SafeLoRA generally improved safety (lowered harmfulness) but consistently reduced model capability (accuracy). On the Science10k dataset, the safest SafeLoRA checkpoint lowered accuracy by nearly 10% compared to the best standard LoRA checkpoint.
Qwen Models: Results were mixed. On GSM8k, SafeLoRA improved both safety and accuracy. However, on Science10k, SafeLoRA caused marked degradation in both safety and accuracy.
Coherence Mitigation: On ARC and BoolQ, SafeLoRA appeared to mitigate the coherence degradation observed in standard LoRA fine-tuning, likely by preserving alignment weights that support free-form generation.

Significance and Claims

The paper claims that the field of LLM safety requires a paradigm shift from arbitrary empirical settings to capability-grounded evaluations.

Grounding is Essential: Anchoring fine-tuning to specific, verifiable capability goals is necessary to avoid arbitrary choices and enable consistent comparison of mitigation methods.
Measurement Validity: Safety metrics are brittle. The paper argues that without measuring coherence (e.g., via perplexity) and using multiple evaluators, safety assessments are unreliable. Incoherent outputs can artificially inflate safety scores or, conversely, be falsely flagged as harmful.
Contextual Interpretation: The authors conclude that the effectiveness of safety-preserving methods like SafeLoRA is not absolute; it varies significantly based on the model architecture, the specific fine-tuning task, and the measurement instruments used. Therefore, researchers and practitioners must adopt well-defined, grounded, and valid measurement methodologies to accurately assess the safety of fine-tuned models.

The authors explicitly state that their findings highlight the need for further study into the mechanisms of safety recovery in later training epochs and the development of more intrinsic measures of safety behavior (e.g., via mechanistic interpretability) rather than relying solely on external benchmarking.