Navigating the Accuracy-Size Trade-Off with Flexible Model Merging

1. Problem Statement

The paper addresses the fundamental tension in deploying Large Language Models (LLMs) and Vision Transformers (ViTs) for multi-task scenarios:

• The Storage vs. Accuracy Dilemma:
  • Individual Fine-Tuning: Deploying a separate fine-tuned model for each task yields the highest accuracy but incurs prohibitive storage and deployment costs (linear scaling with the number of tasks, $M$).
  • Single-Model Merging: Traditional model merging combines all $M$ tasks into a single model (size $1\times$). While storage-efficient, this often results in a significant accuracy gap due to parameter interference (conflicts between task-specific weights).
• Limitations of Existing Solutions:
  • Data-Free Merging: Most current data-free methods (e.g., Task Arithmetic, TIES-Merging) are restricted to producing a single merged model ($1\times$ size). They lack the ability to generate intermediate models that balance storage and accuracy.
  • Data-Dependent Merging: Methods that use validation data to improve merging are often impractical due to privacy constraints or proprietary data restrictions.
  • Rigid Granularity: Existing attempts to merge into larger sizes (e.g., Channel Merging) often rely on fixed clustering (e.g., K-Means) that cannot generate fractional model sizes (e.g., $1.75\times$) or adaptively determine the optimal grouping per layer.

Core Research Questions:

1. How can we derive merged models across the full spectrum of sizes (from $1\times$ to $M\times$, including fractional sizes) in a data-free manner?
2. What is the nature of the accuracy-size trade-off exhibited by different data-free merging algorithms?

---

2. Methodology: FLEXMERGE

The authors propose FLEXMERGE, a novel, data-free framework that enables flexible model merging by treating models as sequences of blocks and performing greedy, bottom-up merging.

A. Core Concept: Block-Level Granularity

Instead of merging entire models at once, FLEXMERGE decomposes a model into $B$ sequential blocks (e.g., individual Transformer layers, attention heads, or MLP layers).

• Input: $M$ fine-tuned task vectors $\{\tau_1, \dots, \tau_M\}$, where each $\tau_i$ is decomposed into block-level vectors $\{\tau_i^b\}_{b=1}^B$.
• Goal: Reduce the total deployed size from $M \times$ (all models) to a target size $S_{target}$ (where $1 \le S_{target} \le M$) while maximizing accuracy.

B. The Greedy Merging Algorithm

FLEXMERGE employs a bottom-up, greedy approach (Algorithm 1):

1. Initialization: Start with $M$ distinct task groups for every block.
2. Similarity Metric: In each iteration, the algorithm calculates the similarity between all pairs of task groups within every block. The similarity between two groups $g_i$ and $g_j$ is defined as the minimum cosine similarity between any pair of original task vectors across the two groups:
   $$\text{SIMILARITY}(g_i, g_j) = \min_{k_1 \in g_i, k_2 \in g_j} \cos(\tau_{k_1}^b, \tau_{k_2}^b)$$
   Rationale: The "min" strategy ensures that merging only occurs when even the most dissimilar tasks in the groups are sufficiently compatible, preventing the introduction of high-interference conflicts.
3. Selection: Identify the block $b^*$ and the pair of groups $(g_{i^*}, g_{j^*})$ with the highest similarity score.
4. Merge:
   • Union the task sets: $T_{uni} = g_{i^*} \cup g_{j^*}$.
   • Apply a chosen data-free merging algorithm $F$ (e.g., Task Arithmetic, TIES-Merging, Consensus) to the block vectors of these tasks: $\tau_{uni}^b = F(\{\tau_k^b \mid k \in T_{uni}\})$.
   • Replace the two groups with the new merged group.
5. Termination: Repeat until the total parameter count reaches the target size $S_{target}$.

C. Compatibility with Existing Algorithms

FLEXMERGE is a unified framework that can wrap any existing data-free merging algorithm:

• Standard Methods (TA, TIES, PCB, TSV-M): These generate a single unified vector $\tau_{uni}^b$. The final block weights are $\theta_{uni}^b = \theta_{pre}^b + \tau_{uni}^b$.
• Mask-Based Methods (Consensus, EMR-Merging): These generate a unified vector plus task-specific masks. FLEXMERGE stores the unified vector and masks for the merged group, reconstructing task-specific weights at inference time. This allows FLEXMERGE to support these high-accuracy methods while offering size flexibility.

D. Complexity

• Time Complexity: $O(B M^2 d_{max})$, dominated by pre-computing pairwise similarities.
• Efficiency: The merging process is extremely fast (e.g., ~20 seconds for 30 tasks on ViT-L/14), making it practical for generating models of various sizes on demand.

---

3. Key Contributions

1. First Flexible Size Framework: FLEXMERGE is the first method to generate merged models across the entire spectrum of sizes, including non-integer sizes (e.g., $1.75\times$), bridging the gap between single-model merging and full ensemble deployment.
2. Unified Algorithm Support: It provides a single framework to apply diverse data-free merging algorithms (from simple averaging to complex mask-based methods) at the block level.
3. Systematic Characterization of Trade-offs: The paper systematically maps the accuracy-size curve, revealing that:
   • Steep Initial Gains: Modest increases in model size (e.g., doubling from $1\times$ to $2\times$) yield massive accuracy improvements (up to 13.5%).
   • Diminishing Returns: Accuracy approaches the full fine-tuning limit well before the maximum size ($M\times$).
   • Ranking Instability: The relative performance of algorithms is not consistent across sizes. A method that is inferior at $1\times$ (e.g., simple averaging) can outperform sophisticated methods (e.g., TIES-Merging) at larger sizes ($>2\times$).
4. Efficiency: The framework introduces negligible inference-time overhead, even for larger merged models, by utilizing efficient tensor sharing and task-specific block routing.

---

4. Experimental Results

The authors evaluated FLEXMERGE on extensive benchmarks:

• Vision: 8-task and 30-task benchmarks using ViT-B/32 and ViT-L/14 (CLIP).
• NLP: 11-task PEFT benchmark using (IA)3 on T0-3B, and 7-task Full-Parameter Fine-Tuning (FFT) on T5-Base/Large.
• Multi-modal: BEiT-3-base on COCO, ImageNet, etc.

Key Findings:

• Accuracy Gains:
  • On 8 vision tasks (ViT-B/32), FLEXMERGE + Task Arithmetic (TA) improved accuracy from 67.5% ($1\times$) to >80% ($2\times$), a 13.5% gain.
  • On 30 tasks, doubling the size from $\approx 3\times$ to $6\times$ with Consensus yielded an 8.5% gain.
  • Near-perfect fine-tuning accuracy was achieved at sizes significantly smaller than $M$ (e.g., $\approx 6\times$ for 8 tasks, $\approx 23.5\times$ for 30 tasks).
• Algorithm Rankings Shift:
  • At $1\times$, advanced methods like TIES-Merging or PCB-Merging often outperform simple averaging.
  • At $>2\times$, the performance gap narrows significantly, and simpler methods (like Averaging or TA) often rival or surpass complex methods. This suggests that increasing model size naturally mitigates parameter interference, reducing the need for complex interference-reduction mechanisms.
• Comparison with Channel Merging: FLEXMERGE consistently outperformed Channel Merging (a recent $>1\times$ method) because FLEXMERGE's greedy, pairwise approach allows for variable cluster sizes per block, whereas Channel Merging enforces fixed clustering.
• OOD Performance: FLEXMERGE models with sizes $>1\times$ consistently outperformed the best single in-domain expert on Out-of-Distribution (OOD) tasks, suggesting improved generalization.

---

5. Significance and Implications

• New Design Dimension: The paper argues that the field should move beyond the "single-model limit" ($1\times$). Practitioners should evaluate and develop merging algorithms across the full size spectrum, as algorithm rankings are size-dependent.
• Practical Deployment Strategy: FLEXMERGE offers a "sweet spot" for deployment. Organizations can achieve near-full accuracy with only a fractional increase in storage (e.g., $2\times$ or $3\times$ instead of $M\times$), avoiding the massive costs of storing $M$ separate models while avoiding the accuracy penalty of a single merged model.
• Flexibility for Mask-Based Methods: By allowing larger sizes, FLEXMERGE reduces the reconstruction latency overhead for mask-based methods (like Consensus), as fewer blocks need to be reconstructed at inference time.
• Scalability: The paper demonstrates that scaling model size proportionally to the number of tasks ($M$) is crucial for maintaining accuracy as the task count grows, a scaling law that fixed-size models fail to satisfy.

In conclusion, FLEXMERGE redefines model merging not as a binary choice between "one model" and "many models," but as a continuous trade-off space where accuracy can be tuned precisely by adjusting the deployed model size, offering a highly practical solution for multi-task deployment in resource-constrained environments.