💬 NLP

Trustworthiness Costs of Domain Adaptation in Small Language Models:A Cross-Architecture Empirical Study

This paper presents a systematic cross-architecture empirical study revealing that while domain adaptation of small language models using adversarially perturbed data improves performance without harming factual calibration, common safety-preserving strategies like Dark Experience Replay and Task Arithmetic LoRA fail to maintain adversarial robustness and can significantly increase susceptibility to harmful outputs.

Original authors: Ramesh B. Paramkusham

Published 2026-08-04

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ramesh B. Paramkusham

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant, super-smart robot that knows a little bit about everything—history, cooking, space, and jokes. This is what we call a "Large Language Model." But sometimes, this robot is too big and expensive to run on a regular laptop, and it might get confused when asked very specific questions, like "How do I treat a broken leg?" or "What does this legal contract mean?" To fix this, scientists create smaller, cheaper versions called "Small Language Models" (SLMs). Think of these as the robot's younger, faster cousins who are ready to learn a specific job.

The big question is: What happens when we teach these smart-but-small robots a new, specialized skill? It's like taking a general knowledge genius and sending them to medical school. We know they can learn the facts, but will they stay honest? Will they still be able to tell the difference between a real medical fact and a made-up one? And if someone tries to trick them with confusing questions, will they break? This is the world of "trustworthiness." It's not just about being smart; it's about being safe, accurate, and hard to fool. Researchers are worried that when we teach these robots a new job, we might accidentally break their "safety brakes" or make them start lying, even if they get better at the job itself.

This paper is like a giant, careful experiment where a researcher put three different small robot brains through a rigorous training camp to see exactly what happens to their honesty and safety when they learn new skills. The researcher tested three different robot models (TinyLlama, Gemma-2, and Llama 3.2) and taught them three very serious jobs: healthcare, law, and finance. They tried teaching them in two ways: with normal, clean textbooks (benign data) and with textbooks that had been secretly messed up with confusing tricks and lies (adversarial data). They also tried four different "teaching methods" to see if any of them could protect the robots' safety while they learned.

Here is what they found, and it's a bit surprising. First, when they just taught the robots their new jobs using the standard method, the robots didn't really get any worse at telling the truth. Their "honesty score" barely changed at all. It turns out that learning a new job doesn't automatically make them liars.

Second, the researcher tried a weird trick: they gave the robots textbooks that had been intentionally messed up with confusing facts and tricks. You might think this would confuse the robots, but actually, it helped them learn their new job better. It was like giving a student a practice test with tricky questions; when they took the real test, they did great. However, this didn't make them any better at resisting tricks or attacks; it just made them better at the job itself.

The most important part of the story is about the "safety guards." The researcher tried three special methods designed to keep the robots safe while they learned. One method was supposed to be a neutral safety net, and the other two were supposed to be super-protective shields. The results were a bit disappointing for the "super-protective" ideas. The neutral method didn't really do anything (it was like a safety net that was too loose to catch anything). But the two "super-protective" methods actually made the robots more vulnerable to being tricked! In some cases, these safety methods made the robots fail safety tests by a huge margin—up to 45% worse in some specific situations.

So, the big takeaway is this: Teaching a small, smart robot a new job doesn't automatically make it lie, but the fancy "safety shields" we invented to protect them while they learn might actually be making them weaker against tricks. The robots that were already safe before training stayed safe, but the ones that were supposed to be "super-protected" by these new methods ended up getting confused more easily. It seems that the safety tricks we use for general robots don't work the same way when the robot is learning a very specific, high-stakes job like medicine or law. The researcher suggests we need to rethink how we keep these specialized robots safe, because the current "magic shields" might be doing more harm than good.

Technical Summary: Trustworthiness Costs of Domain Adaptation in Small Language Models

Problem Statement

The rapid maturation of Small Language Models (SLMs), defined here as models with 100M–5B parameters, offers a practical pathway for deploying NLP systems in resource-constrained, high-stakes domains such as healthcare, legal services, and finance. While parameter-efficient fine-tuning (PEFT) methods like Low-Rank Adaptation (LoRA) and its quantized variant (QLoRA) enable cost-effective domain specialization, a critical gap remains in understanding the trustworthiness cost of this adaptation.

Existing literature extensively characterizes performance gains in task-specific benchmarks but largely overlooks how domain fine-tuning impacts trustworthiness properties, specifically factual calibration and adversarial robustness. Previous work (e.g., Qi et al., 2024) suggests that even benign fine-tuning can degrade safety alignment. This paper addresses the lack of empirical data quantifying whether domain adaptation compromises these safety properties and whether existing safety-preserving strategies (Safety-DPO, Dark Experience Replay, Task Arithmetic LoRA) effectively mitigate such degradation in SLMs.

Methodology

The study employs a systematic, cross-domain, and cross-architecture empirical design involving 216 experimental configurations across three random seeds (42, 123, 456).

1. Experimental Setup

Models: Three SLM architectures spanning the 1–2.6B parameter range:
- TinyLlama 1B: Utilizes grouped-query attention and a 3T token pre-training corpus.
- Gemma-2 2B: Features alternating local/global attention and knowledge distillation.
- Llama 3.2 1B: Benefits from multi-stage RLHF instruction tuning.
Domains: Three high-stakes domains with distinct linguistic characteristics:
- Healthcare: MedQA-USMLE (short MCQs, high reasoning).
- Legal: LegalBench (long declarative text, high hallucination risk).
- Finance: Finance-Alpaca + PhraseBank (mixed Q&A and sentiment).
Data Conditions:
- Benign: Standard domain text.
- Adversarial: Systematically perturbed via Domain Injection (DI), Concept Mixing (CM), and Anchored Obfuscation (AO).
Fine-Tuning Strategies:
- Baseline: Standard QLoRA (control).
- Safety-DPO: Preference optimization on domain vs. harmful pairs.
- Dark Experience Replay (DER): Interleaving safety-relevant pre-training samples to prevent catastrophic forgetting.
- Task Arithmetic LoRA (TA-LoRA): Merging a safety LoRA adapter with a domain adapter ( $\tau_{merged} = \tau_{domain} + 0.5 \cdot \tau_{safety}$ ).

2. Evaluation Metrics

Domain Adaptation Quality: Measured by evaluation loss (cross-entropy) and mean token accuracy.
Trustworthiness:
- Factual Calibration: TruthfulQA MC2 accuracy (higher is better).
- Adversarial Robustness: HarmBench Attack Success Rate (ASR) (lower is better; reported as $1 - ASR$ for composite analysis).
- Note: BBQ (bias) evaluation was noted as ongoing and not included in the final empirical results.

Key Contributions

First Cross-Domain/Cross-Architecture Study: This work provides the first systematic quantification of trustworthiness costs across three SLM architectures, three high-stakes domains, and four fine-tuning strategies.
High Reproducibility: A multi-seed experimental design demonstrated exceptionally low variance ( $\sigma < 0.001$ in evaluation loss), establishing a robust statistical foundation for comparisons.
Counter-Intuitive Finding on Adversarial Data: The study empirically demonstrates that adversarially perturbed training data consistently improves domain adaptation quality (reducing evaluation loss by $\Delta \approx -0.040$ ) without degrading trustworthiness benchmarks, suggesting an implicit data-augmentation effect.
Comparative Safety Strategy Evaluation: A direct comparison of Safety-DPO, DER, and TA-LoRA reveals that current alignment-preserving strategies do not effectively transfer to domain-adapted SLMs.

Key Results

1. Baseline Trustworthiness Costs

Contrary to the hypothesis that standard LoRA domain adaptation systematically degrades trustworthiness:

Factual Calibration: Baseline QLoRA produced minimal changes in TruthfulQA MC2 scores across all model-domain combinations (mean $|\Delta TQA| < 0.02$ ).
Adversarial Robustness: The baseline ASR was primarily determined by the pre-training safety alignment of the model rather than the domain fine-tuning itself.
- TinyLlama 1B: Exhibited near-ceiling vulnerability (ASR $\approx$ 0.97–1.00) regardless of strategy.
- Safety-Aligned Models (Gemma-2, Llama 3.2): Showed meaningful variance but maintained relatively low ASR in baseline conditions.

2. Impact of Adversarial Training

Adversarially perturbed training data consistently improved domain adaptation quality (lower evaluation loss) but did not yield consistent improvements or degradations in trustworthiness metrics. The improvement in adaptation quality appears orthogonal to adversarial robustness.

3. Performance of Safety-Preserving Strategies

None of the three safety-preserving strategies successfully reduced the trustworthiness cost or improved adversarial robustness:

Safety-DPO: Was effectively neutral regarding adversarial robustness (mean $\Delta ASR < 0.001$ ) and offered no significant safety benefit over the baseline.
Dark Experience Replay (DER) & TA-LoRA: Both strategies worsened adversarial robustness in safety-aligned models.
- DER: Increased mean HarmBench ASR by +0.171.
- TA-LoRA: Increased mean HarmBench ASR by +0.155.
- Worst Cases: Individual configurations showed degradation exceeding +0.45 (e.g., Gemma-2 2B in Healthcare with DER; Llama 3.2 1B in Legal with TA-LoRA).

Significance and Claims

The paper claims to challenge the prevailing assumption that replay-based (DER) and arithmetic-merge (TA-LoRA) strategies effectively transfer safety alignment to domain-adapted SLMs. The findings suggest that:

Negligible Calibration Cost: Standard domain adaptation does not materially alter factual calibration in SLMs.
Strategy Failure: Current safety-preserving fine-tuning strategies, developed for general-purpose settings, fail to preserve or improve adversarial robustness in domain-specific contexts and may actively degrade it.
Architecture Dominance: Pre-training safety alignment is the dominant predictor of adversarial robustness, not the fine-tuning strategy or domain.

The study concludes that practitioners deploying SLMs in high-stakes domains should be cautious regarding the use of DER and TA-LoRA for safety preservation, as these methods may inadvertently increase susceptibility to adversarial harm. Future work is directed toward mechanistic explanations for these degradations and exploring alternative safety-preservation approaches.