Technical Summary: U-shaped Multi-granularity Learning for Vision-Language Models (UPrompt)

1. Problem Statement

Vision-Language Models (VLMs) adapted via prompt learning face a fundamental granularity dilemma. Existing methods typically optimize at a single, fixed granularity, creating an inherent trade-off:

• Global Prompts: Strategies like CoOp capture broad context but lack fine-grained semantic awareness, failing to encode local features necessary for detailed reasoning (e.g., underperforming on the FGVCAircraft dataset).
• Local Prompts: Finely structured prompts preserve local details but struggle to integrate sufficient global context or model cross-region compositional relationships.

This limitation constrains adaptation performance and generalization across diverse tasks such as cross-modal retrieval, few-shot classification, and base-to-novel generalization. While hierarchical architectures like U-Net have successfully unified multi-level representations in dense prediction tasks (operating on spatial pixel grids), transferring these principles to prompt learning is non-trivial. Prompt learning operates in an abstract embedding space where textual prompts lack inherent geometric structure, making the construction of meaningful multi-granularity representations and bidirectional information flow a significant challenge.

2. Methodology: UPrompt

The authors propose UPrompt, a U-shaped multi-granularity prompt learning framework inspired by U-Net. It constructs parallel multi-granularity representations in both visual and textual modalities, connected by bidirectional information flow mechanisms.

2.1 U-Shaped Multi-Granularity Construction

The framework extends CLIP encoders to multi-granularity versions ($k \in [1, K]$), spanning from coarsest to finest levels.

• Visual Modality: Constructs nested patch hierarchies via progressive spatial pooling in the embedding space. Starting from the finest patch tokens ($E^{(K)}_{patch}$), coarser representations are derived recursively ($E^{(k)}_{patch} = Pool^{(k)}(E^{(k+1)}_{patch})$). Learnable prompts are concatenated with these pooled embeddings before being fed into the vision encoder.
• Textual Modality: Constructs semantic hierarchies via progressive enrichment. Core semantics are extracted ($T^{(1)}_{embed}$), and subsequent levels are generated by iteratively adding granularity-specific elaborations ($T^{(k)}_{embed} = T^{(k-1)}_{embed} \oplus \Phi^{(k)}_{refine}(t)$). These are instantiated using Large Language Models (LLMs) to generate hierarchical text descriptions (e.g., from abstract concepts to detailed descriptions).

2.2 Bidirectional Connection Mechanisms

To ensure cross-level consistency and complementary learning, UPrompt introduces two key mechanisms analogous to U-Net's skip connections:

1. Coarse-to-Fine Cascaded Enhancement (CE):

   • Goal: Address context deficiency where fine-grained embeddings lack global guidance.
   • Mechanism: Injects coarse-grained contextual information into finer embeddings using cross-granularity attention. The enhanced embedding $\hat{X}^{(k)}$ is computed by attending to the enhanced representation of the coarser level $\hat{X}^{(k-1)}$.
   • Effect: Allows fine-grained features to extract contextually relevant information from global representations, improving local modeling.

2. Fine-to-Coarse Hierarchical Supervision (HS):

   • Goal: Address optimization inconsistency and semantic drift at coarse granularities.
   • Mechanism: Uses the finest-level features (which possess optimal cross-modal correspondence) as "teacher signals" to supervise coarser levels via knowledge distillation. The KL divergence between the coarse-level similarity distribution and the detached fine-level distribution is minimized.
   • Effect: Ensures semantic consistency across scales, preventing coarse representations from drifting while maintaining their complementary nature.

2.3 Overall Objective

The training objective combines contrastive alignment loss across all $K$ granularity levels with the hierarchical supervision loss ($L_{guide}$). During inference, the system can flexibly select granularity levels to balance performance and efficiency, defaulting to the finest level for optimal accuracy.

3. Key Contributions

• UPrompt Framework: A U-Net-inspired framework that introduces structured hierarchy into vision-language adaptation, overcoming single-scale limitations through parallel multi-granularity pathways.
• Bidirectional Information Flow: The introduction of Coarse-to-Fine Cascaded Enhancement (injecting global context into local details) and Fine-to-Coarse Hierarchical Supervision (regularizing coarse granularities with fine-level alignment) to ensure semantic consistency and complementary learning.
• Comprehensive Validation: Demonstration of superior performance across 17 benchmarks, including cross-modal retrieval, few-shot classification, base-to-novel generalization, and out-of-distribution scenarios.

4. Experimental Results

Extensive experiments validate the effectiveness of UPrompt:

• Cross-Modal Retrieval:
  • On MSCOCO, UPrompt outperforms MAMET and VPKE by 4.1 and 7.3 rSum, respectively.
  • On Flickr30K, it achieves an rSum of 571.1, surpassing recent fine-tuning methods like DoPL, MAMET, and VPKE.
• Base-to-Novel Generalization:
  • Achieves a 82.12% Harmonic Mean (HM) across 11 datasets, outperforming CoCoA-Mix by 5.09% and surpassing TAP and SurPL-G.
  • Notable gains are observed on challenging datasets like FGVCAircraft (+0.79% HM).
• Few-Shot Classification:
  • Achieves 85.13% average accuracy on 11 datasets at 16-shot, outperforming GalLoP, ProVP-Ref, and MMA.
• Efficiency and Adaptability:
  • Adaptive Inference: Using a cascaded early-exit strategy, UPrompt matches fine-grained accuracy while reducing computational cost to 48%, achieving a 2.1× speedup.
  • Cost-Performance: The medium-grained configuration (UPrompt-M) matches the accuracy of PSRC with only 1/3 of the FLOPs.
• Robustness:
  • Demonstrates strong Out-of-Distribution (OOD) generalization (61.22% average on ImageNet variants), outperforming methods like GalLoP and HiCroPL.
  • Performance remains stable across different LLMs (Qwen3, Llama3) used for text hierarchy generation.

5. Significance and Claims

The paper claims that UPrompt provides a principled solution for vision-language adaptation by bridging hierarchical representation learning principles from U-Net to prompt-based VLMs.

• Resolution of Trade-offs: It successfully resolves the limitation of single-granularity approaches by establishing a unified framework that achieves both global understanding and local precision through bidirectional information flow.
• Semantic Consistency: The framework ensures that semantic consistency is maintained across representation scales, preventing the degradation often seen in isolated multi-granularity modules.
• Flexibility: The hierarchical design enables flexible performance-efficiency trade-offs, allowing the model to adapt to resource constraints without significant loss in accuracy.

The authors acknowledge that while effective, the current hierarchical depth is limited, and future work will explore deeper multi-granularity architectures to model richer semantic structures.