HeBA: Heterogeneous Bottleneck Adapters for Robust Vision-Language Models

1. Problem Statement

Vision-Language Models (VLMs) like CLIP have achieved remarkable zero-shot generalization but face significant challenges when adapting to downstream tasks with limited data (few-shot learning). The primary issues identified are:

• Architectural Homogeneity: Existing Parameter-Efficient Fine-Tuning (PEFT) methods (e.g., adapters, prompt learning) often treat visual and textual tokens as uniform 1D vectors. This ignores the distinct structural nature of the modalities: images possess spatial locality (2D correlations), while text possesses semantic density (global relationships). Treating them uniformly leads to "spatial amnesia," discarding critical structural cues like textures and shapes.
• Stability-Plasticity Dilemma: Naive fine-tuning causes catastrophic forgetting, while current adapter designs often rely on "Inverse Bottlenecks" (expanding dimensions, e.g., $4\times$) which increase parameter counts and overfitting risks in data-scarce regimes.
• Initialization Limitations: Most PEFT methods use zero-initialization for adapter weights to preserve the pre-trained backbone's identity. The authors argue this induces vanishing gradients in the early training stages, delaying convergence and hindering the model's ability to adapt to severe distribution shifts.

2. Methodology: HeBA Framework

HeBA (Heterogeneous Bottleneck Adapter) is a unified framework that introduces modality-specific structural inductive biases directly into the adapter architecture. It freezes the pre-trained CLIP backbone and injects lightweight, specialized adapters.

A. Heterogeneous Inductive Biases

Instead of a single architecture for both modalities, HeBA bifurcates the processing:

• Visual Stream (Spatial-Aware): Utilizes 2D depthwise-separable convolutions. Input tokens are reshaped into a 2D grid ($B \times D \times \sqrt{N} \times \sqrt{N}$). The adapter applies a $1\times1$ convolution for channel compression, followed by a $3\times3$ depthwise convolution to aggregate local spatial context, and finally flattens the output. This explicitly models spatial locality (textures, shapes).
• Textual Stream (Semantic-Preserving): Utilizes dense linear projections. Since text lacks spatial structure, the adapter uses standard linear layers to preserve global semantic integrity without unnecessary spatial operations.

B. Structural Regularization via Bottlenecks

Unlike prior methods that expand feature dimensions (Inverse Bottlenecks), HeBA employs a compressive bottleneck ($D \rightarrow D/4$).

• Mechanism: The adapter forces the model to compress features before projecting them back.
• Benefit: This acts as an intrinsic structural regularizer, forcing the model to learn low-rank, compact representations of domain shifts. It physically filters out task-irrelevant noise and reduces overfitting risks without needing complex external regularizers.

C. Active Gradient Initialization

HeBA challenges the standard zero-initialization paradigm.

• Strategy: It uses Active Kaiming Initialization (He Normal distribution) for the up-projection weights ($W_{up}$).
• Rationale: Zero-initialization creates a state of vanishing gradients in the adapter subspace at $t=0$. Active initialization ensures sufficient initial gradient magnitude, allowing the model to rapidly adapt to the downstream distribution while the frozen backbone preserves pre-trained knowledge.

D. Optimization & Regularization

To balance the active initialization and prevent divergence:

• Dynamic Slow-Fast Schedule: A stochastic scaling mechanism amplifies the adapter's output scale factor $s$ with a certain probability to escape local saddle points.
• Label Smoothing: Replaces standard Cross-Entropy with Label Smoothing Cross-Entropy (LSCE) to prevent overconfident predictions on few-shot examples.

3. Key Contributions

1. Heterogeneous Architecture: The first unified adapter framework to explicitly decouple visual and textual processing pipelines, using 2D convolutions for vision and linear projections for text to respect their intrinsic structural differences.
2. Compressive Bottleneck Design: Replaces parameter-heavy expansion designs with a $D \rightarrow D/4$ compression strategy, serving as a structural regularizer that enhances generalization in few-shot settings.
3. Active Initialization Paradigm: Proposes Kaiming initialization over zero-initialization, demonstrating that active gradient flow accelerates convergence and improves the stability-plasticity trade-off without catastrophic forgetting.
4. State-of-the-Art Performance: Establishes new benchmarks across 11 diverse datasets, particularly excelling in structure-sensitive domains (textures, satellite imagery).

4. Experimental Results

The authors evaluated HeBA on 11 diverse image classification datasets (e.g., ImageNet, EuroSAT, DTD, Oxford-Pets) under Base-to-Novel generalization, Cross-Dataset, and Domain Generalization settings.

• Base-to-Novel Generalization:
  • HeBA achieved a new State-of-the-Art (SOTA) Harmonic Mean (HM) of 81.35%, surpassing the previous best (LwEIB at 81.21%) and strong baselines like MaPLe (78.55%).
  • It showed superior Novel Accuracy (78.62%), indicating better generalization to unseen classes compared to CoOp (63.22%) and LwEIB (78.21%).
• Structure-Sensitive Domains:
  • DTD (Textures): +2.37% improvement over LwEIB (70.20% vs 67.83%).
  • EuroSAT (Satellite Imagery): Achieved 88.16% HM, significantly outperforming LwEIB (86.86%). This validates the efficacy of 2D spatial modeling for non-object-centric domains.
• Cross-Dataset & Domain Generalization:
  • Achieved the highest average accuracy (68.71%) across 10 target datasets when trained on ImageNet.
  • Demonstrated robustness on adversarial examples (ImageNet-A), achieving 51.36% accuracy, outperforming MMA and LwEIB.
• Ablation Studies:
  • Removing the spatial bias (using 1D sequences) dropped HM to 81.25%.
  • Replacing depthwise convolutions with identity mapping dropped HM to 81.20%.
  • Switching from Active Kaiming Init to Zero-Init reduced Base accuracy and overall HM, confirming the benefit of active gradient flow.

5. Significance

HeBA represents a paradigm shift in adapting VLMs by moving away from "one-size-fits-all" adapter designs. Its significance lies in:

• Theoretical Insight: It proves that respecting the inductive biases of specific modalities (spatial for vision, semantic for text) is crucial for robust few-shot learning.
• Efficiency: By using compressive bottlenecks rather than expansive ones, it achieves higher performance with fewer parameters and lower overfitting risks.
• Robustness: The combination of structural regularization and active initialization creates a more stable optimization landscape, making VLMs more reliable for real-world applications involving domain shifts (e.g., medical imaging, satellite analysis) where data is scarce and distribution changes are frequent.

The code is publicly available, facilitating further research into modality-specific architectural designs for large-scale models.