SeMoBridge: Semantic Modality Bridge for Efficient Few-Shot Adaptation of CLIP

1. Problem Statement

While Contrastive Language-Image Pretraining (CLIP) excels at zero-shot tasks due to strong inter-modal alignment (image-text), it struggles in few-shot classification. The core limitation is intra-modal misalignment:

• The Modality Gap: CLIP's training objective focuses on pulling paired image-text samples together across the modality gap. Consequently, the internal semantic structures within the image modality and within the text modality remain uncalibrated.
• The Consequence: Direct image-to-image comparisons (intra-modal) are unreliable. A query image may be embedded closer to the centroid of an incorrect class than its true class, leading to misclassification.
• Limitations of Existing Solutions:
  • Logit Refinement (e.g., Tip-Adapter, APE): Avoid direct image-image comparison by relying on text prompts, but this limits the capture of fine-grained visual details.
  • Optimization-based Methods (e.g., Cross the Gap): Map images to text via per-sample iterative optimization. While effective, this is computationally expensive and lacks inference efficiency.

Goal: Design a method that resolves intra-modal misalignment without the high computational cost of per-sample optimization.

2. Methodology: SeMoBridge

The authors propose SeMoBridge, a lightweight approach that maps image embeddings into the text modality to enable robust inter-modal comparisons (bridged-image vs. original-image or bridged-image vs. bridged-image).

A. Core Mechanism: Closed-Form Projection

SeMoBridge leverages the pre-trained alignment of CLIP to create a unified projection. It is based on the observation that normalized image embeddings ($f_{img}$) and their corresponding (unknown) text embeddings ($\hat{f}_{txt}$) point in similar directions in the shared space.

1. Pseudo-EOS Token Derivation: Using the Moore-Penrose pseudo-inverse of CLIP's text projection matrix ($W_{txt}^+$), the method back-projects the image embedding to estimate a "pseudo" End-of-Sequence (EOS) token ($\hat{f}_{eos}$).
2. Scaling: The magnitude of the projected vector is rescaled to match the average norm of genuine EOS tokens derived from class descriptions.
3. Final Bridged Embedding: The pseudo-EOS is projected back through $W_{txt}$ to obtain the final bridged embedding ($\hat{f}_{txt}$).
   • Formula: $\hat{f}_{txt} \approx \frac{\|T_{eos}\|}{\|W_{txt}^+ f_{img}\|} f_{img}$
   • This transformation effectively converts an image embedding into a "text-like" vector while preserving its semantic content, allowing for reliable comparison within the text space.

B. Inference Strategies

The paper proposes two variants:

1. SeMoBridge (Training-Free):

   • Uses the fixed pseudo-inverse of the pre-trained $W_{txt}$.
   • Logit Blending: The final prediction is a weighted sum of three signals:
     • $z_1$: Standard CLIP zero-shot logit (Image vs. Text Prompts).
     • $z_2$: Bridged Query vs. Original Few-Shots (Inter-modal).
     • $z_3$: Original Query vs. Bridged Few-Shots (Inter-modal).
   • This combines the robustness of zero-shot priors with the precision of few-shot bridged comparisons.

2. SeMoBridge-T (Trained):

   • A lightweight trainable variant that fine-tunes the bridge parameters ($\hat{W}_{txt}^+$) and adds a Class-Specific Bias (CSB) term ($\hat{\tau}$) to capture nuanced semantic differences across many classes.
   • Multi-Modal Supervision: The model is trained using a composite loss function:
     • $L_{img}$: Ensures bridged embeddings retain visual semantics (Image-to-Image consistency).
     • $L_{txte}$ & $L_{txtp}$: Aligns bridged embeddings with text prompts (Image-to-Text consistency).
     • $L_{cons}$: Encourages consistency among bridged few-shots of the same class.
     • $L_{bias}$: Regularizes the norm of CSB vectors to prevent class imbalance.
   • Efficiency: Only the bridge and bias parameters are updated; the massive CLIP backbone remains frozen.

3. Key Contributions

1. SeMoBridge Architecture: A lightweight, closed-form solution that directly addresses intra-modal misalignment by mapping images to the text modality, eliminating the need for expensive per-sample optimization.
2. Multi-Modal Supervision Strategy: A novel training approach for SeMoBridge-T that combines image and text alignment losses, ensuring the bridged embeddings retain semantic knowledge from both modalities without backpropagating through CLIP's encoders.
3. State-of-the-Art Efficiency: Demonstrates that high performance in few-shot learning can be achieved with minimal training time (seconds) and parameters, outperforming methods that require hours of training.

4. Experimental Results

The method was evaluated on 11 diverse datasets (including ImageNet, OxfordPets, Flowers102, etc.) using CLIP ViT-B/16 and ResNet-50 backbones.

• Accuracy:
  • SeMoBridge-T achieves state-of-the-art (SOTA) performance, particularly in low-shot scenarios (1, 2, and 4 shots).
  • It outperforms strong baselines like APE, Tip-Adapter-F, LDC, and CLIP-LoRA.
  • On average across 11 datasets, SeMoBridge-T achieves 78.15% accuracy in 16-shot settings, surpassing CoOp and PromptSRC.
• Efficiency:
  • Training Time: SeMoBridge-T requires only 27 seconds of training time on ImageNet (16-shot), compared to hours for methods like CoOp (10h) or PromptSRC (1h 42m).
  • Parameters: It introduces only 0.77M trainable parameters, significantly fewer than LDC (69M).
• Robustness:
  • Out-of-Distribution (OOD): SeMoBridge-T shows superior generalization on ImageNet-V2 and ImageNet-Sketch, outperforming existing methods by up to +0.71%.
  • Transferability: A model trained on ImageNet few-shot data transfers effectively to other datasets without retraining, proving the bridge learns a general geometric projection rather than dataset-specific statistics.
• Ablation Studies:
  • Text supervision is critical for 1-shot scenarios where visual data is scarce.
  • The Class-Specific Bias (CSB) and its regularization ($L_{bias}$) are essential for maintaining balance across classes, especially for visually distinct categories.

5. Significance

SeMoBridge represents a paradigm shift in few-shot adaptation for vision-language models. Instead of treating the modality gap as an obstacle to be bypassed via indirect logit refinement or expensive optimization, it exploits the gap by transforming the problem into a well-calibrated inter-modal comparison.

• Practical Impact: The method makes high-accuracy few-shot learning accessible in resource-constrained environments due to its negligible training time and computational footprint.
• Theoretical Insight: It validates that the intra-modal misalignment in CLIP is a structural issue that can be resolved via a simple, closed-form linear projection, suggesting that the "modality gap" is largely a geometric shift rather than a complex non-linear divergence.
• Versatility: The approach extends beyond classification to retrieval tasks (Image-Image and Text-Text), consistently outperforming intra-modal baselines.

In summary, SeMoBridge offers a highly efficient, theoretically grounded, and empirically superior solution for adapting CLIP to few-shot tasks, setting a new benchmark for the trade-off between accuracy and computational cost.