🤖 machine learning

Stay Unique, Stay Efficient: Preserving Model Personality in Multi-Task Merging

This paper proposes Decomposition, Thresholding, and Scaling (DTS), a highly storage-efficient personalized model merging framework that preserves task-specific performance and generalizes to unseen tasks using only 1% extra storage per task by leveraging singular value decomposition and semantic similarity.

Original authors: Kuangpu Guo, Aijing Yu, Jian Liang, Yuhe Ding, Zilei Wang, Ran He, Tieniu Tan

Published 2026-06-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Kuangpu Guo, Aijing Yu, Jian Liang, Yuhe Ding, Zilei Wang, Ran He, Tieniu Tan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Smoothie" Effect

Imagine you have a master chef who is an expert at cooking eight different cuisines: Italian, Japanese, Mexican, Indian, and so on. To save space in the kitchen, you decide to mix all their recipes into one giant "Super Chef" cookbook.

You try to blend the recipes by simply averaging them out. The result? A disaster. The "Super Chef" can't make a perfect pizza because the Japanese spices ruin the dough, and the Mexican salsa clashes with the Indian curry. This is what happens in AI when we try to merge different models: the specific "personality" or knowledge of each task gets lost in the mix, leading to poor performance.

The Current Fix: The "Heavy Backpack"

Some recent solutions tried to fix this by giving the Super Chef a backpack. For every new cuisine, the chef carries a separate, heavy bag of specific ingredients (task-specific parameters). This works great—the chef can make perfect pizza and perfect sushi.

But there's a catch: The backpacks are too heavy. If you have 100 tasks, you need 100 heavy backpacks. This defeats the purpose of trying to save space and resources in the first place.

The Solution: DTS (Decomposition, Thresholding, and Scaling)

The authors propose a new method called DTS. Think of DTS as a "Smart Compression" technique that lets the chef keep their unique personality without needing a heavy backpack. It works in three clever steps:

1. Decomposition (The "Highlight Reel")

Instead of keeping the entire recipe book for a specific task, DTS uses a mathematical trick (called Singular Value Decomposition) to find the most important "highlights" of that recipe.

Analogy: Imagine you have a 3-hour movie. Instead of storing the whole thing, you extract only the 10 most critical scenes that define the plot. You throw away the boring filler. DTS keeps only the "top 10%" of the most important numbers that make the task unique.

2. Thresholding (The "Grouping Game")

Now that we have the highlights, they are still too detailed to store efficiently. DTS looks at these numbers and groups them into four simple categories:

Big Positive numbers
Small Positive numbers
Big Negative numbers
Small Negative numbers
Analogy: Instead of writing down every single person's exact height in a crowd (e.g., 5'10.23", 5'10.25"), you just put them into four bins: "Tall," "Medium," "Short," and "Very Short." You lose a tiny bit of precision, but you save a massive amount of space.

3. Scaling (The "Volume Knob")

To make sure the "bins" still sound like the original crowd, DTS assigns a simple "volume knob" (a scaling factor) to each group.

Analogy: If the "Tall" bin is too quiet, you turn the volume knob up for that group. This allows the system to reconstruct the original "flavor" of the task very accurately using just a few bits of data.

The Result: This process compresses the task-specific information so much that it only takes up 1% extra space per task. It's like shrinking a heavy backpack down to the size of a single key.

The Magic Trick: Guessing New Tasks (Generalization)

Usually, if you want the chef to cook a new cuisine they've never seen (like Ethiopian food), you need to train them or give them a new recipe.

DTS has a special "Data-Free" mode. It looks at the names or descriptions of the tasks.

Analogy: If the chef has mastered "Italian" and "Spanish," and you ask them to cook "Portuguese," DTS looks at the names. It knows "Portuguese" is linguistically similar to "Spanish." So, it automatically mixes the "Spanish" and "Italian" flavors in the right proportions to guess the Portuguese recipe.
It does this without needing any new data or training. It just uses the "semantic similarity" (how much the task names sound alike) to blend the existing compressed memories.

Why This Matters

The paper shows that DTS is the best of both worlds:

Performance: It keeps the model's "personality" intact, performing almost as well as if the model had been trained separately for each task.
Efficiency: It requires only 1% extra storage per task, whereas other methods might require 10% to 100% extra space.
Versatility: It works on images (like recognizing cars or handwritten numbers) and text (like understanding sentences or writing stories).

In short, DTS allows us to merge many AI models into one without them losing their individual skills, and it does so by keeping the "memory" of those skills incredibly small and lightweight.

Technical Summary: Stay Unique, Stay Efficient (DTS)

Problem Statement

Model merging offers a data-free and optimization-free paradigm to unify multiple task-specific models into a single representation, avoiding the prohibitive costs of fine-tuning and storing separate models. However, traditional basic merging methods (e.g., Weight-Averaging, Task-Arithmetic) suffer from severe parameter conflicts, leading to significant performance degradation even when merging models trained on semantically similar tasks.

While recent personalized model merging frameworks attempt to preserve task-specific information to mitigate these conflicts, they typically incur prohibitive storage overhead. Methods that store dense expert components, full model parameters, or complex routing networks negate the primary efficiency advantage of model merging, limiting their deployment in resource-constrained environments. The core challenge is to preserve essential task-specific information to maintain performance while minimizing the extra storage required per task.

Methodology: Decomposition, Thresholding, and Scaling (DTS)

The authors propose DTS, an approximation-based personalized merging framework designed to compress task-specific information efficiently. The method operates in two main stages: an approximation stage and an inference stage.

1. Task-Specific Information Representation

The framework defines task-specific information in two forms:

Task Vectors ( $\tau_n$ ): The difference between fine-tuned weights ( $\theta_n$ ) and pre-trained weights ( $\theta_0$ ), i.e., $\tau_n = \theta_n - \theta_0$ .
Difference Vectors ( $d_n$ ): A novel representation defined as the difference between fine-tuned weights and a coarsely merged baseline ( $\theta_m$ ), i.e., $d_n = \theta_n - \theta_m$ . This variant is particularly useful when original pre-trained weights are inaccessible and captures the personalized residual relative to a shared baseline.

2. The DTS Compression Pipeline

To store these vectors efficiently, DTS applies three sequential operations:

Decomposition (SVD): The target matrix (task vector or difference vector) undergoes truncated Singular Value Decomposition (SVD). Only the top- $r$ singular values and their corresponding vectors are retained, filtering out unstructured parameter noise based on the Eckart-Young-Mirsky theorem.
Thresholding: Instead of using a simple binary mask (which discards fine-grained information), DTS introduces a novel thresholding strategy. It partitions the elements of each singular vector into four groups:
1. Positive "Large" values
2. Positive "Small" values
3. Negative "Large" values
4. Negative "Small" values
  This is achieved by first encoding the sign (1 bit) and then comparing positive/negative values against their respective medians to determine magnitude levels.
Scaling: For each of the four groups, a single scaling factor is computed. This factor represents the arithmetic mean (calculated via L1 norm for efficiency) of the absolute values within that group.

3. Reconstruction and Storage Efficiency

During inference, the task-specific information is reconstructed by combining the approximated singular vectors and the scaling factors. The resulting representation requires only 2 bits per element (1 bit for sign, 1 bit for magnitude level), achieving a 16 $\times$ compression ratio. This results in an additional storage overhead of approximately 1% per task.

4. Generalization to Unseen Tasks (Data-Free Adaptation)

To handle unseen tasks without additional training data or labeled examples, the authors propose a variant that fuses task-specific information based on semantic similarity.

Task characteristics (e.g., class names for classification, instructions for generation) are encoded using a pre-trained text encoder to generate semantic embeddings.
The merging weights for an unseen task are computed dynamically based on the cosine similarity between the unseen task's embedding and the embeddings of seen tasks.
This approach is entirely data-free and requires no online training or backpropagation.

Key Contributions

DTS Framework: A lightweight, approximation-based personalized merging method that preserves task-specific information while minimizing storage overhead through SVD, novel 4-group thresholding, and scaling.
DTS-T and DTS-D Variants: The introduction of two specific implementations: one operating on standard task vectors and another on difference vectors (relative to a merged baseline), offering flexibility for different deployment scenarios.
Data-Free Generalization: A variant capable of adapting to unseen tasks by leveraging semantic similarity of task characteristics, eliminating the need for target-domain data or trainable routers.
Efficiency-Performance Trade-off: The method achieves state-of-the-art performance with only $\approx$ 1% extra storage per task, significantly outperforming existing personalized methods that often require 10–100% additional storage.

Experimental Results

The authors evaluated DTS on visual classification (ViT-B/32, ViT-L/14, ViT-B/16), natural language processing (RoBERTa, GPT-2), and natural language generation (Qwen-14B) tasks.

Standard Multi-Task Merging: DTS consistently outperforms state-of-the-art baselines (including T-Switch, FREE-Merging, and WEMOE) on standard benchmarks.
- On ViT-B/32, DTS-D achieved an average accuracy of 90.40%, surpassing the best baseline (T-Switch at 90.15%) while requiring only 3.68% additional memory (compared to T-Switch's 6.25% and WEMOE's 49.95%).
- On RoBERTa, DTS-D achieved 84.98% average accuracy with only 3.81% additional memory, recovering 99.06% of the performance of individually fine-tuned models.
- The authors note that by adaptively adjusting the sparsity coefficient ( $r$ ), they can reduce the additional memory rate to <1% (e.g., DTS-D* at 0.98% for ViT-B/32) with minimal performance loss.
Generalization to Unseen Tasks: In zero-shot scenarios where fine-tuned models for unseen tasks were not available during merging:
- DTS variants significantly outperformed all competing personalized methods. For instance, on ViT-B/32 unseen tasks, DTS-D achieved 55.41% average accuracy, whereas other personalized baselines (like WEMOE and T-Switch) dropped to the 40–50% range, often performing worse than basic merging methods like Ties-Merging.
- This demonstrates that the semantic similarity-based fusion effectively transfers task-specific knowledge without overfitting to seen tasks.

Significance and Claims

The paper claims that DTS addresses the fundamental trade-off in model merging between performance preservation and storage efficiency. By demonstrating that task-specific information can be compressed to a 2-bit representation without significant loss of fidelity, the authors argue that personalized merging can be deployed in resource-constrained environments previously inaccessible to such methods.

Furthermore, the work highlights that personalized merging methods often fail to generalize to unseen tasks because they over-optimize for specific seen tasks. The proposed data-free, semantic-similarity-based adaptation offers a scalable solution for "cold-start" multi-task scenarios, enabling models to handle new tasks without additional training data or computational overhead. The authors position DTS as a practical, efficient, and effective paradigm for next-generation multi-task model deployment.