Unlocking Few-Shot Capabilities in LVLMs via Prompt Conditioning and Head Selection

1. Problem Statement

Despite the remarkable success of Large Vision-Language Models (LVLMs) in zero-shot tasks like image captioning and Visual Question Answering (VQA), they significantly underperform compared to CLIP-based methods in image classification, particularly in few-shot and zero-shot settings.

• The Paradox: Many LVLMs utilize CLIP-pretrained vision encoders. However, while CLIP excels at classification by matching images to class names, LVLMs often fail to leverage these internal representations effectively for classification.
• The Limitation: In CLIP, vision and text encoders are independent, leading to a bias toward simple class-name matching. In LVLMs, a shared LLM decoder processes vision and text tokens jointly. The authors hypothesize that while the LVLM's final output (summary token) is weak for classification, its internal representations (specifically attention heads) contain rich, discriminative features that are currently being ignored or underutilized.
• Goal: To bridge the performance gap between training-free CLIP-based methods and LVLM-based methods without requiring extensive fine-tuning.

2. Methodology: Head Ensemble Classifiers (HEC)

The authors propose HEC, a training-free framework that conditions LVLMs via prompts and extracts discriminative features from specific attention heads. The method consists of three main components:

A. Prompt Conditioning

Instead of using a generic prompt, the authors introduce a hierarchical conditioning strategy to guide the LVLM's internal feature distribution:

1. Task Conditioning: A general prompt (e.g., "What is in this image?") to steer the model toward discriminative representations.
2. Domain Conditioning: A domain-specific prompt (e.g., "What breed is that dog?") to extract features relevant to the specific visual domain.
3. Class Conditioning: Appending a list of candidate classes to the prompt (e.g., "Between: boxer, yorkshire...") to further refine class separability.

• Impact: Experiments show that adding domain and class conditioning significantly improves the class separability of the internal distributions, a capability CLIP lacks due to its independent encoders.

B. Head Selection via Gaussian Discriminant Analysis (GDA)

The core innovation is the selection of a sparse set of "best" attention heads rather than relying on the final summary token.

• Mechanism: The authors treat the attention vectors of individual heads as features. They assume these features follow a class-conditional Gaussian distribution.
• Ranking:
  • Vision-Heads (for Few-Shot): Using a labeled support set, they apply GDA to rank heads based on their ability to separate classes. The score is derived from the log-probability of the features under the Gaussian model.
  • Text-Heads (for Zero-Shot): Heads are ranked based on their zero-shot soft accuracy (dot product between image and text embeddings) on the support set.
• Insight: The paper demonstrates that a small subset of heads (e.g., top 20) often outperforms the full model's summary token by a significant margin (up to 10% gain).

C. Ensemble Classifiers

The selected heads are combined to form three distinct classifiers:

1. HEC-V (Vision-Few-Shot): Averages the class probabilities of the top-ranked vision-heads. Used when class names are unknown.
2. HEC-T (Text-Zero-Shot): Averages the class probabilities of the top-ranked text-heads. Used for zero-shot classification.
3. HEC-VT (Vision-Text-Few-Shot): A weighted combination of HEC-V and HEC-T probabilities, bridging the gap for scenarios where both support sets and class names are available.

3. Key Contributions

1. Prompt Conditioning for Feature Refinement: Demonstrated that prompt conditioning (Task, Domain, Class) significantly improves the class separability of LVLM internal representations, enabling domain adaptation without fine-tuning.
2. Sparse Head Selection: Introduced a method to identify and select a sparse set of highly discriminative attention heads (Vision-Heads and Text-Heads) using GDA, proving that these heads outperform the model's final output token.
3. Training-Free SOTA Performance: Proposed HEC, a training-free classifier that achieves state-of-the-art results in few-shot and zero-shot classification across 12 diverse datasets, effectively closing the gap between LVLMs and CLIP-based methods.
4. Complementarity: Showed that HEC is complementary to fine-tuning approaches and can be applied to any LVLM architecture.

4. Experimental Results

The method was evaluated on 12 datasets (including EuroSAT, UCF101, OxfordPets, StanfordCars, Flowers102, Food101, etc.) using Qwen2-VL and LLaVA-OV models.

• Vision-Few-Shot (HEC-V):

  • Achieved an average accuracy of 82.4% across 12 datasets (4-shot setting).
  • Outperformed all other LVLM-based methods (including SAVs) and surpassed strong non-LVLM baselines (like DINOv3 and DFN) on 9 out of 12 datasets.
  • Notably, it improved upon the base LVLM's linear probing by ~4% using Domain conditioning.

• Text-Zero-Shot (HEC-T):

  • Improved Qwen2-VL's zero-shot accuracy by +10.1% on average, surpassing its own CLIP backbone (DFN) by 1.5%.
  • Consistently outperformed the standard summary token (ST) baseline and CLIP-based zero-shot methods on most datasets.

• Vision-Text-Few-Shot (HEC-VT):

  • Achieved an average accuracy of 83.0%, outperforming all LVLM baselines by >3% and beating the best CLIP-based baselines on average.
  • Showed particular strength on non-object-centric datasets (textures, scenes, actions) where CLIP often struggles.

• Ablation Studies:

  • Confirmed that Domain Conditioning yields the highest performance gains.
  • Demonstrated that Head Ensembling (averaging probabilities) is robust and superior to voting or logit averaging.
  • Showed that top heads are transferable across tasks and domains, though selecting them on-the-fly from the support set is nearly as effective as oracle selection.

5. Significance and Future Work

• Paradigm Shift: The paper challenges the notion that LVLMs are inherently poor at classification. It suggests that the "weakness" lies in the final decoding layer, while the intermediate attention layers hold superior discriminative power.
• Efficiency: HEC is training-free, requiring no gradient updates or parameter tuning, making it highly efficient for deployment.
• Generalizability: The approach is model-agnostic and can be applied to any LVLM with a transformer decoder.
• Limitations: The method requires access to intermediate attention vectors (which may be restricted in API-based models) and is computationally heavier than CLIP due to the LLM inference. Class conditioning is also limited by the number of classes that fit in the context window.

In conclusion, HEC unlocks the latent few-shot capabilities of LVLMs by leveraging prompt conditioning to refine features and selecting specific attention heads to maximize classification performance, offering a powerful, training-free alternative to current CLIP-based and fine-tuned approaches.