💻 computer science

Asymmetric Collapse in Model Merging: When Refusal Over- writes Recognition

This paper demonstrates that standard model merging techniques often cause an asymmetric collapse where safety recognition capabilities are lost while broad refusal behaviors are preserved, primarily because the refusal fine-tune induces significantly larger task-vector magnitudes that dominate the merging process despite the vectors being nearly orthogonal.

Original authors: Aarnav Choudhary, Matheus Fonseca Rocha, Jiwon Seo, Vasu Sharma, Maheep Chaudhary

Published 2026-07-31

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Aarnav Choudhary, Matheus Fonseca Rocha, Jiwon Seo, Vasu Sharma, Maheep Chaudhary

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart robot that can learn new tricks. Sometimes, you want to teach it one specific thing, like how to spot a fake news headline. Other times, you want it to learn a different trick, like how to say "no" to dangerous requests. But what if you wanted the robot to do both at the same time? In the world of artificial intelligence, there's a clever shortcut called "model merging." Instead of retraining the robot from scratch, scientists take two different versions of the robot—one trained for trick A and one for trick B—and mathematically blend their brains together. It's like mixing two different flavors of ice cream to get a new, perfect swirl. The big question is: when you mix these two "flavors" of safety training, do you get a robot that's good at both, or does one flavor completely drown out the other? This is the mystery a new study set out to solve, looking at whether we can keep a robot's ability to understand danger while also keeping its ability to refuse it.

The researchers behind this study, published at the COLM 2026 conference, decided to test this with a specific robot model called Gemma-3-1B-IT. They created two special versions of this robot. The first version, let's call it "Detective," was trained on a massive dataset of medical questions to become an expert at spotting how harmful a prompt is. It learned to give a graded answer, like saying, "This is a little risky," or "This is very dangerous." The second version, "Guardian," was trained on a huge collection of tricky, adversarial prompts designed to break the robot's rules. Guardian learned one simple, blunt lesson: if it looks dangerous, just say "No" and refuse to answer, no questions asked.

The team then took these two specialists and tried to merge them back into one robot using four different mathematical blending methods. They expected the result to be a robot that could both detect the level of harm and refuse the bad stuff. But instead, they found a strange and lopsided outcome. In every single case, the "Guardian" behavior took over completely. The merged robots became excellent at refusing bad prompts, keeping their refusal rates high (between 81% and 85%). However, the "Detective" skills vanished almost entirely. The robots forgot how to grade the harm; their ability to classify the severity of a prompt dropped to near zero (falling to as low as 0.6% or 12.9%, depending on the method). It was as if the robot learned to slam the door shut on everything, forgetting how to peek through the peephole to see what was actually there.

Why did this happen? The scientists looked closely at the "DNA" of the two robots to find the culprit. They discovered it wasn't because the two robots were fighting each other or trying to go in opposite directions. In fact, their brains were nearly at right angles to each other, meaning they were trying to do totally different things without clashing. The real problem was a matter of volume. The "Guardian" robot had been trained on a dataset that was roughly 29 times larger than the "Detective's" dataset. Because of this, the changes made to the Guardian's brain were much "louder" and stronger. When the scientists mixed the brains, the mathematical tools they used were like a volume knob that turned up the louder signal and turned down the quieter one. The "Guardian's" big, bold "No!" drowned out the "Detective's" subtle, nuanced "Maybe."

Even when the researchers tried to fix this by shrinking the Guardian's training data to match the Detective's size, the problem didn't fully go away for all the merging methods. This suggests that while the size of the data was a major factor, the way the merging algorithms handle these "loud" versus "quiet" updates is the real key. The study suggests that standard merging tools are currently biased toward keeping the big, blunt behaviors (like refusing everything) while accidentally deleting the fine-grained, detailed skills (like understanding why something is bad).

So, what does this mean for the future? The paper suggests that if we want to build safe AI that can both refuse bad things and understand the nuances of safety, we can't just mix and match models using current standard tools. We might be losing the "Detective" inside the "Guardian." The researchers conclude that future methods need to be smarter about balancing these different "volumes" of learning, ensuring that the robot doesn't just learn to say "no" to everything, but also learns to understand the difference between a little warning and a big danger.

Technical Summary: Asymmetric Collapse in Model Merging

Problem Statement
Model merging allows for the combination of separately fine-tuned models into a single model without additional training, operating on the assumption that the result approximates a balanced synthesis of source capabilities. However, it remains unclear whether standard merging methods can simultaneously preserve multiple, distinct safety-relevant behaviors. Specifically, this paper investigates whether merging a model fine-tuned for nuanced harm recognition with a model fine-tuned for broad adversarial refusal results in a balanced capability or a "collapse" where one behavior overwrites the other.

Methodology
The authors conducted a controlled case study using Gemma-3-1B-IT as the base model to create two distinct supervised fine-tuned (SFT) variants:

FT CARES: Fine-tuned on the CARES benchmark (18,000+ prompts) to perform structured, graded classification of medical prompt harm levels (0–3).
FT WJB: Fine-tuned on the WildJailbreak dataset (262k examples) to learn binary refusal behavior against adversarial jailbreak attempts.

These two models were merged using four standard algorithms from the mergekit library: Linear (uniform averaging), SLERP (spherical linear interpolation), TIES, and DARE-TIES. All merges utilized equal weighting ( $\alpha = 0.5$ ).

The merged models were evaluated on three metrics:

CARES Accuracy: Four-class classification accuracy for harm levels.
Attack Resistance: Refusal rate on adversarial prompts (WildJailbreak).
Benign Compliance: Non-refusal rate on benign prompts.

To isolate the cause of observed failures, the authors performed mechanistic analyses, including:

Vector Analysis: Measuring per-layer L2 norms and cosine similarity between task vectors (differences between fine-tuned and base weights).
Data Size Normalization: Retraining FT WJB on a 9k subsample to match the CARES dataset size, thereby reducing the magnitude disparity.
Attention Head Ablation: Learning binary necessity masks to determine which attention heads are critical for each task and tracking their retention or destruction post-merge.

Key Results
The experiments revealed a stark asymmetric collapse across all four merging methods:

Refusal Transfers, Recognition Collapses: Merged models retained high attack resistance (81.1%–85.3%), closely matching the FT WJB baseline (88.0%). Conversely, harm recognition accuracy plummeted. The best-performing merge (SLERP) achieved only 12.9% CARES accuracy, while TIES and DARE-TIES dropped to near-zero (0.6% and 0.8%, respectively), effectively reverting to the base model's performance or worse.
Orthogonality vs. Magnitude: The task vectors for refusal and recognition were found to be nearly orthogonal (cosine similarity of 0.011), ruling out opposing directions as the primary cause of interference. Instead, the FT WJB task vector exhibited 4–5× larger per-layer L2 magnitudes than the FT CARES vector.
Magnitude Sensitivity: Methods sensitive to magnitude (TIES, DARE-TIES) systematically favored the larger refusal updates. Even in the data-normalized experiment (9k subsample), where the magnitude gap was reduced, classification accuracy did not fully recover for Linear, SLERP, or TIES, though DARE-TIES showed significant improvement (20.0% accuracy), suggesting magnitude is a proximate but not sole driver.
Structural vs. Functional Preservation: Attention head analysis revealed that while 49–65% of the essential attention heads for CARES were structurally retained in the merged weights, their functional capability was corrupted. The merged weights within these heads no longer supported classification, a failure invisible to aggregate weight statistics.

Significance and Claims
The paper claims that standard model merging can inadvertently collapse fine-grained safety recognition into broad, coarse refusal when safety-relevant task vectors differ substantially in scale. The authors argue that this is not a result of conflicting task directions but rather a magnitude imbalance where larger updates (refusal) dominate smaller, more distributed updates (recognition) in the weight space.

The significance of this work lies in identifying a specific failure mode in merging safety-aligned models: the loss of nuanced judgment (e.g., grading harm levels) in favor of blanket refusal. The authors caution that any safety capability requiring graduated judgment (toxicity scoring, risk triage) may be similarly vulnerable to being overwritten by coarser behavioral objectives during standard merging processes. They conclude that future safety-aware merging methods must account for behavioral roles, scale imbalances, and the preservation of fine-grained recognition alongside refusal, rather than focusing solely on task-vector direction and sparsity.

Technical Summary: Asymmetric Collapse in Model Merging

More like this