ACE-Merging: Data-Free Model Merging with Adaptive Covariance Estimation

1. Problem Statement

Model Merging aims to combine multiple task-specific expert models (fine-tuned from a shared pre-trained base) into a single unified model without requiring access to the original training data. This is crucial for consolidating knowledge in scenarios where data is private, unavailable, or too large to retrain on.

The Core Challenge:

• Inter-Task Interference: When merging models trained on different objectives, their parameter updates often conflict, leading to significant performance degradation.
• The Data-Free Limitation: Existing solutions fall into two categories:
  • Data-dependent methods: Require access to task data to estimate statistics (e.g., Fisher information), which is often impractical due to privacy or storage constraints.
  • Heuristic/Iterative methods: Operate solely on weights (e.g., Task Arithmetic, Ties-Merging) but rely on simple averaging or complex iterative optimization (gradient descent) without a theoretical guarantee of optimality. They often fail to address the root cause of interference: the mismatch in underlying data distributions.

The Gap: There is a lack of a principled, closed-form, data-free solution that can theoretically estimate the necessary statistical structure of task data solely from model weights.

---

2. Methodology: ACE-Merging

The authors propose ACE-Merging (Adaptive Covariance Estimation), a framework that theoretically bridges the gap between parameter-space updates and data-level statistics.

A. Theoretical Foundation

The paper establishes a key theoretical insight: The input covariance of a task ($\Sigma_t$) is implicitly encoded in the parameter displacements ($\Delta W_t$) of its fine-tuned model.

• Under a linear approximation of the Transformer layers ($f(W, x) \approx Wx$), the optimal merged weights $\bar{W}$ that minimize the discrepancy with all experts follow a closed-form solution:
  $$\bar{W} = \left( \sum_{t} W_t \Sigma_t \right) \left( \sum_{t} \Sigma_t \right)^{-1}$$
• Theorem 1: The authors prove that $\Sigma_t \propto \text{Cov}(\Delta W_t)$. This means the input covariance can be estimated directly from the fine-tuning updates ($\Delta W_t = W_t - W_0$) without any data.

B. The Three-Stage Algorithm

ACE-Merging implements this theory through three specific components to handle real-world heterogeneity:

1. Adaptive Covariance Normalization:

   • Problem: Tasks often have vastly different "energy scales" (magnitude of weight updates). High-energy tasks dominate the merge, suppressing low-energy tasks.
   • Solution: The method calculates a heterogeneity metric ($\gamma$) based on the variance of the log-Frobenius norms of task vectors.
   • If $\gamma$ exceeds a threshold (indicating high heterogeneity), it normalizes each task's covariance matrix by its trace ($\text{Tr}(\hat{\Sigma}_t)$) to equalize energy scales. It also applies Tikhonov regularization scaled by the trace to ensure numerical stability during matrix inversion.

2. Collective Structural Prior (CSP):

   • Problem: Standard regularization (e.g., $+\epsilon I$) is isotropic and ignores the geometry of the feature space.
   • Solution: The method constructs a Collective Structural Prior ($C_{agg}$) by aggregating the column-wise statistics of all scaled covariance matrices. This acts as an anisotropic regularizer that amplifies dimensions of shared importance across tasks, effectively integrating inter-task structural consensus into the optimization.

3. Spectral Refinement:

   • Problem: Even with normalization, the preliminary closed-form solution often suffers from spectral ill-conditioning (extreme concentration of energy in top singular values), making the model sensitive to noise.
   • Observation: The principal directions (eigenvectors) of the preliminary solution are usually correct, but the singular value distribution is unbalanced.
   • Solution: The method computes a structural residual and performs a low-rank SVD. It re-weights the top-$k$ principal directions using their mean singular value to restore a stable, expressive energy profile while preserving the correct structural subspace.

---

3. Key Contributions

1. Foundational Theory: The paper provides the first formal proof linking fine-tuning parameter changes directly to input covariance matrices, enabling purely data-free model merging with a theoretical basis.
2. Unified Framework: It reinterprets prior methods (like simple averaging or WUDI-Merging) as implicit, often coarse, estimators of input covariance, clarifying why they work and where they fail.
3. Closed-Form Solution: Unlike iterative gradient-based merging methods, ACE-Merging offers a principled, non-iterative, closed-form solution, ensuring computational efficiency and stability.
4. State-of-the-Art Performance: Extensive experiments show ACE-Merging outperforms all existing data-free baselines and even surpasses some data-dependent and test-time adaptation methods.

---

4. Experimental Results

The authors evaluated ACE-Merging on diverse vision and language benchmarks:

• Vision Benchmarks (ViT):

  • Tested on ViT-B/32, ViT-B/16, and ViT-L/14 across 8, 14, and 20-task sets.
  • Result: ACE-Merging achieved State-of-the-Art (SOTA) in all 9 settings. On the most challenging 20-task ViT-L/14 setting, it achieved 89.5% accuracy, surpassing the next best method (TSV-M) by nearly 2 absolute points.

• Language Benchmarks:

  • GPT-2: Achieved an average score of 74.1% on the GLUE benchmark, outperforming the previous SOTA (Ties-Merging) by 4.1% and TSV-M by 3.9%.
  • RoBERTa: On RoBERTa-Base, it achieved 90.4% (normalized), beating WUDI-Merging by 5.1%. On RoBERTa-Large, it reached 91.7%, outperforming WUDI-Merging by 2.9%.
  • LLaMA-3: Demonstrated strong out-of-domain generalization, recovering significant performance on multilingual and coding tasks where simple averaging failed.

• Efficiency:

  • The method has a computational complexity of $O(L \cdot T \cdot n^3)$, matching SVD-based methods but avoiding the iterative overhead ($K$ steps) of gradient-based methods like WUDI-Merging.

---

5. Significance

• Practicality: ACE-Merging solves the "data-free" dilemma, allowing organizations to merge proprietary or private models without needing access to the original datasets.
• Scalability: The closed-form nature makes it highly scalable to large models (e.g., LLaMA-3, ViT-L) and large numbers of tasks, where iterative optimization becomes prohibitively expensive.
• Theoretical Advancement: By establishing the link between weight displacements and input covariance, the paper shifts model merging from heuristic engineering to a theoretically grounded optimization problem.
• Robustness: The adaptive mechanisms (normalization and spectral refinement) ensure the method works robustly across diverse architectures (Transformers, ViTs) and varying degrees of task heterogeneity.

In conclusion, ACE-Merging represents a significant leap forward in model merging, offering a theoretically sound, efficient, and high-performing solution for consolidating specialized AI models without data access.