Technical Summary: PACT – Preserving Anchored Cores in Task-Vectors for Model Merging

Problem Statement

Model merging has emerged as a training-free alternative to multi-task learning, aiming to combine multiple task-specific fine-tuned models into a single multi-task model. The dominant paradigm, Task Arithmetic, decomposes fine-tuned weights into pre-trained parameters ($\theta_0$) and task vectors ($\Delta_t = \theta_t - \theta_0$), performing merging exclusively in the task-vector space. This approach implicitly assumes that all task-specific knowledge is encoded within the task vectors, while the pre-trained model serves merely as a shared initialization point.

The authors challenge this assumption, arguing that it does not hold due to the intrinsic task preferences of pre-trained models. They identify a phenomenon where task-critical knowledge remains "anchored" in the pre-trained weights rather than being fully transferred into the task vectors. These dimensions, termed Load-Bearing Wall (LBW) dimensions, exhibit negligible updates during fine-tuning (small $|\Delta_t|$) but are essential for maintaining task performance. Existing merging methods, by operating solely on task vectors, inadvertently overwrite or distort these LBW dimensions when aggregating multiple tasks, leading to performance degradation.

Methodology: PACT (Preserve Anchored Cores)

To address the vulnerability of LBW dimensions, the authors propose PACT, a data-free framework designed to protect task-specific knowledge anchored in the pre-trained model during the merging process.

1. Identification of LBW Dimensions

The paper characterizes LBW dimensions from two perspectives:

• Scalar-Weight Perspective: Using Fisher Information and enrichment scores, the authors identify specific parameters that have small magnitudes in the task vector but high sensitivity to the task loss.
• Subspace Perspective: Using Singular Value Decomposition (SVD), the authors define the LBW subspace ($V_{t,LBW}$) for a task $t$. This is derived by projecting the principal subspace of the pre-trained weights ($V_{0,K}$) onto the orthogonal complement of the task's active update subspace ($V_{t,k}$). Formally:
  $$V_{t,LBW} = \text{orth}\left((I - V_{t,k}V_{t,k}^\top)V_{0,K}\right)$$
  This subspace captures pre-trained feature directions critical for the task that remain untouched during fine-tuning.

2. The PACT Algorithm

PACT operates by filtering task vectors before applying standard merging algorithms (e.g., Task Arithmetic, TSV-M, Iso-C). The core mechanism involves constructing a Global Orthogonal Shield for each task to prevent its updates from interfering with the LBW subspaces of other tasks.

• Shield Construction: For a specific task $t$, the algorithm aggregates the LBW subspaces of all other tasks ($V_{j,LBW}$ for $j \neq t$) and constructs a unified orthonormal basis $V_{t}^{protect}$ using QR decomposition. This shield represents the union of critical pre-trained dependencies required by all other tasks.
• Filtering: The task vector $\Delta_t$ is projected onto the orthogonal complement of this shield to remove components that would overwrite the "sacred spaces" of other tasks:
  $$\tilde{\Delta}_t = \Delta_t - \Delta_t V_{t}^{protect}(V_{t}^{protect})^\top$$
• Merging: The filtered vectors $\{\tilde{\Delta}_t\}$ are then merged using existing algorithms.

3. Efficient Variant (Randomized SVD)

To improve scalability, the authors propose an efficient variant using Randomized SVD (RSVD). Since PACT primarily requires the extraction of top-$k$ singular vectors, RSVD approximates these components with significantly reduced computational complexity ($O(mnk)$ vs. $O(mn^2)$ or $O(n^3)$) and memory footprint, while acting as an implicit regularizer that smooths high-frequency noise.

Key Contributions

1. Revisiting Task Arithmetic Assumptions: The paper empirically demonstrates that task vectors do not fully capture task-specific knowledge. It identifies and causally validates the existence of LBW dimensions—parameters that receive negligible updates but are critical for performance.
2. Causal Validation: Through controlled perturbation experiments (scalar and subspace levels), the authors show that:
   • Modifying LBW dimensions (even with small magnitudes) causes catastrophic performance drops.
   • Standard merging overwrites these dimensions, and restoring them recovers performance.
   • Subspace ablation of LBW components leads to performance degradation orders of magnitude larger than random subspace ablation.
3. PACT Framework: The proposal of a data-free, geometric framework that constructs orthogonal shields to filter interfering updates, seamlessly integrating with existing scalar-weight and subspace-based merging methods.
4. Theoretical Analysis: The authors provide theoretical bounds showing that the performance degradation caused by PACT's filtering is controlled by the projection of the task gradient and update onto the protection space, ensuring that self-performance is preserved if the task update is concentrated in its active subspace.

Experimental Results

Extensive experiments were conducted on vision benchmarks (ViT-B/32, ViT-B/16, ViT-L/14) across 8, 14, and 20 tasks, covering both full fine-tuning and LoRA fine-tuning settings.

• Performance Gains: PACT consistently enhances mainstream merging methods (Task Arithmetic, TSV-M, Iso-C).
  • PACT-Iso-C achieves State-of-the-Art (SOTA) performance across all 9 evaluated scenarios (3 backbones $\times$ 3 task counts).
  • In the 20-task setting with ViT-L/14, PACT-Iso-C reaches 95.1% absolute accuracy and 98.3% normalized accuracy, surpassing the previous best (Iso-CTS) by a significant margin.
• LoRA Adaptation: PACT also improves merging performance for LoRA-fine-tuned models, demonstrating that LBW effects persist even when pre-trained parameters are frozen during fine-tuning (as the LBW knowledge resides in the base model).
• Efficiency: The RSVD variant achieves performance comparable to full SVD (and sometimes superior due to implicit denoising) while drastically reducing computational complexity.
• Ablation Studies:
  • Static vs. Dynamic: A static baseline (S-PACT) that only filters against the pre-trained core without considering other tasks' LBW subspaces yields limited gains, highlighting the necessity of dynamic, mutual protection.
  • Layer-wise Depth: Filtering early foundational layers (where intrusion energy is highest) yields the most significant improvements, though global filtering is required for high task loads.

Significance and Claims

The paper claims that model merging is not merely about resolving conflicts among task updates but also about actively respecting and shielding the critical pre-trained foundations. By identifying the "Load-Bearing Wall" dimensions, the authors reveal a fundamental blind spot in existing task-vector-based methods: they manipulate what is explicitly encoded in task vectors but overlook knowledge implicitly stored in the pre-trained model.

The significance of PACT lies in its ability to:

• Provide a geometric perspective on model merging that complements existing algebraic approaches.
• Offer a modular, data-free solution that can be integrated into any existing task-vector merging pipeline without requiring access to original training data.
• Establish new SOTA performance across diverse vision tasks and model scales, proving that protecting anchored cores is essential for robust multi-task synthesis.

The authors conclude that while PACT introduces some computational overhead (mitigated by the RSVD variant), it offers a necessary paradigm shift for handling the structural dependencies inherent in pre-trained models during merging. Future work is suggested to extend this framework to Large Language Models (LLMs) and explore adaptive dimensionalities for the protection shields.