Technical Summary: BiCo – Bilinear Coordinate Alignment for Training-Free Task-Vector Transfer

Problem Statement

The prevailing paradigm in modern machine learning involves adapting large-scale pre-trained models to specialized downstream tasks via fine-tuning. However, as pre-trained model families expand with new variants differing in scale, pre-training data, or training recipes, expertise acquired through fine-tuning becomes tied to the specific parameterization of the original model. Consequently, when a new model version is released, the fine-tuned knowledge cannot be directly reused, necessitating costly and repetitive fine-tuning processes.

Existing approaches to transfer this "task-vector" expertise (defined as the parameter difference $\tau = \theta_{ft} - \theta_{pre}$) across different models face significant limitations. Methods relying on parameter-space alignment (e.g., re-basin via permutation symmetries) require strict architectural compatibility and matching dimensions. Recent training-free methods that utilize task-specific signals like activations or gradients (e.g., THESEUS, GradFix) provide partial correspondences but still exhibit a substantial performance gap relative to direct fine-tuning of the target model. The core challenge is to transfer task vectors across models with differing widths, depths, and pre-training configurations without additional parameter updates.

Methodology: BiCo

The authors propose BiCo (Bilinear Coordinate alignment), a training-free framework that reframes task-vector transfer as a dual-space alignment problem.

1. Theoretical Insight: Bilinear Formation of Task Vectors

The authors revisit the mechanistic formation of a task vector. By analyzing the gradient descent update rule, they demonstrate that a task vector $\tau^{(l)}$ for a layer $l$ is the accumulation of bilinear interactions between input-side activations ($x_t$) and output-side gradients ($g_t$) over the fine-tuning trajectory:
$$\tau^{(l)} = -\sum_{t=1}^{T} \eta_t (g_t^{(l)})^\top x_t^{(l)}$$
This formulation implies that a task vector is not merely a parameter offset but a coupling of two distinct coordinate spaces: the input activation space and the output gradient space. The authors argue that successful transfer requires aligning both spaces to preserve the underlying geometry of the update, rather than aligning only activations or only parameters.

2. The BiCo Framework

BiCo estimates orthogonal Procrustes mappings for both the input and output spaces using a small calibration set ($N$ samples) and a single forward–backward pass through frozen pre-trained models. No target model parameters are updated.

• Input-Side Alignment: The method assumes that pre-trained activations serve as effective proxies for the trajectory activations. It computes the cross-covariance between the source pre-trained activations ($\bar{X}_{A,pre}$) and target pre-trained activations ($\bar{X}_{B,pre}$). An orthogonal Procrustes solution ($R_{in}$) is derived via Singular Value Decomposition (SVD) to map the source input coordinates to the target input space while preserving relational geometry (measured by Centered Kernel Alignment, CKA).
• Output-Side Alignment: Similarly, the method computes gradients with respect to the layer outputs at the pre-trained initialization ($\bar{G}_{A,pre}$ and $\bar{G}_{B,pre}$). A second orthogonal Procrustes solution ($R_{out}$) is estimated to align the output gradient coordinates. This step is crucial because the output dimensions often differ between source and target models, and the "task-sensitive" directions may be encoded in different bases.
• Task-Vector Transfer: The source task vector $\tau_A^{(l)}$ is transformed into the target-compatible update $\hat{\tau}_B^{(l)}$ by applying the learned mappings to both sides of the bilinear structure:
  $$\hat{\tau}_B^{(l)} = (R_{out}^{(l)})^\top \tau_A^{(l)} R_{in}^{(l)}$$
  The final target model is updated as $\hat{\theta}_{B,ft} = \theta_{B,pre} + \hat{\tau}_B$.

3. Handling Structural Mismatches

• Width Mismatch: The Procrustes formulation naturally handles cases where input/output dimensions differ (e.g., narrow-to-wide or wide-to-narrow transfer) by finding the optimal orthogonal projection.
• Depth Mismatch: For models with different numbers of layers, a simple discrete layer-index matching strategy is employed, mapping target layer $j$ to the nearest source layer $i(j)$.

Key Contributions

1. Revisiting Task-Vector Formation: The paper establishes that task vectors are accumulations of bilinear interactions between input activations and output gradients, motivating a dual-space alignment approach rather than single-space or parameter-space alignment.
2. BiCo Framework: Introduction of a training-free method that uses orthogonal Procrustes alignment on both input activations and output gradients to transfer task vectors. It requires only a single forward–backward pass on a small calibration set and involves no parameter updates or hyperparameter tuning.
3. Empirical Validation: Extensive experiments across computer vision (8-Vision benchmark) and natural language processing (GLUE suite) benchmarks demonstrate that BiCo consistently outperforms existing training-free baselines (e.g., THESEUS, GradFix, TransFusion) and narrows the performance gap to direct fine-tuning, even across models with varying widths, depths, and pre-training distributions.

Results

• Width Scaling: In transferring from a ViT-B/16 to a wider ViT-B/16-plus, BiCo significantly outperforms naive padding and THESEUS. With only 100 calibration samples, BiCo achieves an average accuracy gain of +18.88% over the zero-shot baseline, compared to +3.81% for THESEUS.
• Identical Architecture: Even when architectures match but pre-training data differs (Datacomp-XL to LAION-2B), naive transfer degrades performance. BiCo improves the average accuracy by +17.81% over the zero-shot baseline with 5 shots per class, substantially outperforming GradFix and THESEUS.
• Depth and Width Scaling: In challenging scenarios involving both depth and width mismatches (ViT-B/16 to ViT-L/14), BiCo improves the zero-shot baseline by +12.89%, demonstrating structural generalizability.
• NLP Generalization: Applied to T5 models (T5-3B to T5-Large), BiCo shows robust improvements in linear probing accuracy across GLUE tasks, scaling effectively with the number of calibration samples.
• Ablation Studies: Analysis confirms that aligning only the input side or only the output side is insufficient. The combination of both, specifically using gradients for the output side, is critical for performance. Input-only alignment can even degrade performance below the zero-shot baseline in deep layers.

Significance and Claims

The paper claims that BiCo addresses the inefficiency of re-fine-tuning models by enabling the direct reuse of task-specific expertise across diverse model variants. By interpreting task vectors as bilinear interactions, the authors argue that preserving the geometry of both the input activation space and the output gradient space is essential for effective transfer.

The significance of this work lies in its ability to:

• Reduce Computational Overhead: Eliminate the need for iterative fine-tuning of target models, requiring only a single forward–backward pass for calibration.
• Enable Scalable Reuse: Facilitate the transfer of knowledge across models with different widths, depths, and pre-training configurations, which is critical as foundation model families evolve.
• Bridge the Performance Gap: Substantially narrow the gap between training-free transfer and direct fine-tuning, offering a practical, plug-and-play alternative for model adaptation.

The authors maintain modesty regarding limitations, noting that the method assumes sufficiently compatible internal structures (e.g., within the same backbone family) and that a small gap to direct fine-tuning remains, suggesting that linear coordinate alignment alone may not capture all task-specific information. They also note that the method does not inherently correct biases present in the source task vector or calibration data.