💬 NLP

Dynamic Model Merging Made Slim

The paper introduces DiDi-Merging, a compact dynamic model merging framework that utilizes differentiable rank allocation and data-free refinement to achieve superior accuracy-efficiency trade-offs across vision, language, and multimodal tasks with significantly fewer parameters than existing methods.

Original authors: Guodong Du, Wanyu Lin

Published 2026-05-20

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Guodong Du, Wanyu Lin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a master chef who is incredibly talented at cooking. You ask them to learn ten different cuisines: Italian, Japanese, Mexican, Indian, and so on. To do this, you train a separate "specialist" version of the chef for each cuisine. Now, you have ten different chefs, each with their own unique set of notes and techniques.

The Problem:
If you want to run a restaurant that serves all ten cuisines, you can't just hire ten separate chefs; it's too expensive and takes up too much space.

The Old Way (Static Merging): You try to mix all ten chefs' notes into one giant book. But the notes often contradict each other (e.g., "add salt" vs. "don't add salt"), and the final book is a confusing mess where the chef forgets how to cook specific dishes well.
The Current "Dynamic" Way: You keep the master chef's base knowledge and add a small "cheat sheet" for each cuisine. When a customer orders Italian, you hand them the Italian cheat sheet. This works well, but if you have 50 cuisines, you need 50 cheat sheets. If the master chef's base knowledge is huge, and you keep it separate for every single cuisine, your "kitchen" (storage) becomes bloated and expensive.

The New Solution: DiDi-Merging
The authors of this paper, Guodong Du and Wanyu Lin, propose a smarter way to organize these chefs called DiDi-Merging. Think of it as a "Slim Dynamic Kitchen."

Here is how it works, using simple analogies:

1. The "Shared Knowledge" vs. "Specialist Tricks"

Instead of keeping the whole master chef's brain separate for every task (which is heavy) or throwing away the master chef and only keeping tiny notes (which loses quality), DiDi-Merging finds a balance.

It creates a small, shared "core" book that contains the common skills all cuisines share (like chopping vegetables or boiling water).
It then creates tiny, specific "pocket guides" for the unique parts of each cuisine (like the specific spice blend for Indian curry).

2. The "Smart Dimmer Switch" (Differentiable Rank Allocation)

This is the paper's secret sauce. Usually, when you compress information, you just cut off the top 10% or 20% of the data, like cutting the top of a cake. This is a "one-size-fits-all" approach.

DiDi-Merging uses a smart dimmer switch. It looks at every single piece of information and asks: "How important is this for this specific task?"

If a piece of knowledge is useful for all tasks, the dimmer switch keeps it bright in the Shared Core.
If a piece of knowledge is only useful for one task, the dimmer switch dims it down in the core and moves it to the Specialist Pocket Guide.
Crucially, the system learns exactly how much "brightness" (or rank) to give to each part automatically, without needing to see the original training data again. It's like a smart thermostat that learns exactly how much heat each room needs to stay comfortable without wasting energy.

3. The "Polish" (Data-Free Refinement)

When you compress information, you usually lose a little bit of flavor. To fix this, DiDi-Merging does a final "polish."

It takes the compressed, slim version and tweaks it slightly using the original "notes" (task vectors) it already has.
It does this without needing the original ingredients (the raw training data). It's like a chef tasting a reduced sauce and adding a pinch of salt to bring the flavor back, even though they don't have the original pot of soup anymore.

Why is this a big deal?

The paper claims that DiDi-Merging is the most efficient way to do this so far:

Tiny Footprint: It only takes up about 1.24 times the space of a single chef's notes. Previous methods needed 2 times or more space.
High Quality: Even though it's so small, it keeps 98% or more of the original cooking skills. It doesn't lose the flavor.
Versatile: It works for vision (seeing images), language (chatting), and even mixed tasks (like describing a video).

In Summary:
DiDi-Merging is like building a modular kitchen where you have one small, high-quality base station and tiny, custom attachments for every specific job. It uses a smart, learning system to decide exactly how big each part should be, ensuring you get the best performance with the least amount of storage space, all without needing to go back to the original training data.

Technical Summary: DiDi-Merging

Problem Statement
Model merging allows for the reuse of fine-tuned models without joint training or access to original data, addressing the high costs of deploying multiple task-specific models and the infeasibility of multi-task training due to privacy or compute constraints. While static merging fuses models into a single checkpoint, it suffers from task interference. Dynamic merging improves flexibility by routing inputs to task-relevant expert components, but existing methods face a critical storage bottleneck. Current dynamic approaches fall into two suboptimal extremes:

Heavy Shared: Maintaining a full-size shared model with tiny per-task experts (e.g., TALL-Mask, EMR-Merging), resulting in parameter counts no smaller than the original model.
Heavy Expert: Discarding the shared component and storing all knowledge in per-task expert vectors (e.g., TSV-C, SMILE), causing storage to scale linearly with the number of tasks.
Neither approach effectively balances the trade-off between shared knowledge and task-specific capacity, leading to excessive storage overhead (typically $\ge 2\times$ the size of a single fine-tuned model).

Methodology: DiDi-Merging
The authors propose DiDi-Merging (Dynamic Merging with Differentiable Rank), a slim dynamic merging framework that balances parameters between shared and task-specific experts through differentiable rank allocation. The method operates in three stages:

Low-Rank Decomposition: Task vectors ( $\tau_t = \theta_t - \theta_0$ ) are decomposed into a shared knowledge matrix ( $M_s$ ) obtained via static averaging, and per-task expert residuals ( $M_t = \tau_t - M_s$ ). Both matrices are decomposed via Singular Value Decomposition (SVD).
Differentiable Rank Optimization: Instead of fixed rank allocation, the method formulates parameter budgeting as a differentiable optimization problem. A smooth truncation function reweights singular values based on a learnable continuous rank parameter $r$ . The objective minimizes a task reconstruction loss (Mean Squared Error between reconstructed and original task vectors) while penalizing deviations from a target compression ratio. This allows the model to jointly optimize the ranks of shared and expert components, allocating more capacity to diverse tasks and less to similar ones without access to original training data.
Data-Free Refinement: After determining optimal ranks, the truncated low-rank parameters are refined using a data-free optimization step. Low-rank LoRA-style modules ( $A, B$ ) are initialized from the truncated SVD and updated to minimize the reconstruction loss against the stored task vectors, recovering performance lost during truncation.

Key Contributions

Identification of Failure Modes: The paper characterizes existing dynamic merging methods as suffering from "heavy shared" or "heavy expert" allocation extremes, both of which lead to suboptimal accuracy-efficiency trade-offs.
Differentiable Rank Allocation: DiDi-Merging introduces a mechanism to jointly optimize shared and per-task expert ranks via a differentiable SVD truncation objective, eliminating the need for manual rank tuning or data access.
Slim Dynamic Merging: The framework achieves near-lossless performance with significantly reduced storage. It matches prior dynamic baselines at 1.24 $\times$ the parameter count of a single fine-tuned model and surpasses them at 1.4 $\times$ , substantially outperforming methods requiring $\ge 2\times$ storage.

Experimental Results
DiDi-Merging was evaluated across vision, Large Language Models (LLMs), and Multimodal Large Language Models (MLLMs):

Vision: On an 8-task ViT-B/32 benchmark, DiDi-Merging-S (1.24 $\times$ budget) matched the strongest dynamic baseline (TSV-C at 2.08 $\times$ ) with 89.3% accuracy vs. 89.5%, while using less than 60% of the storage. At 1.4 $\times$ , it achieved the highest overall score (89.8%).
LLMs: Merging Llama-3.1-8B-Instruct and Qwen-2.5-7B-Instruct across four domains, DiDi-Merging-L (2.0 $\times$ budget) outperformed prior dynamic methods by an average of +5.4% on Llama-3.1-8B, with significant gains on instruction-following benchmarks like AlpacaEval-2.
MLLMs: In multimodal settings (vision, audio, video, point cloud), DiDi-Merging consistently outperformed data-free dynamic baselines (NaiveMC, MMER) while maintaining a slim parameter footprint. Notably, it achieved performance retention exceeding 100% on certain tasks by leveraging shared cross-modal knowledge.
Efficiency: The method requires modest computational resources (approx. 1.1 GB GPU memory) and runs in roughly 14 minutes for the optimization stage on ViT-B/32.

Significance and Claims
The paper claims that DiDi-Merging represents a significant step forward in making dynamic model merging practical for large-scale deployment. By moving away from the "heavy shared" or "heavy expert" dichotomy, it achieves a Pareto-optimal trade-off between parameter efficiency and accuracy. The authors emphasize that the method is fully data-free, relying solely on stored task vectors for refinement, which preserves privacy and reduces the barrier to entry for merging specialized models. The framework is presented as a general solution applicable across vision, language, and multimodal domains, enabling the deployment of expert ensembles without the storage overhead of maintaining multiple full models or the performance degradation of static merging.

1. The "Shared Knowledge" vs. "Specialist Tricks"

2. The "Smart Dimmer Switch" (Differentiable Rank Allocation)

3. The "Polish" (Data-Free Refinement)

Why is this a big deal?

More like this