Prompt Disentanglement via Language Guidance and Representation Alignment for Domain Generalization

Technical Summary: Prompt Disentanglement via Language Guidance and Representation Alignment for Domain Generalization (PADG)

Problem Definition
Domain Generalization (DG) aims to train models that perform robustly on unseen target domains using only source domain data. A primary challenge in DG is the "domain shift," where training and testing data follow different distributions, violating the standard independent and identically distributed (i.i.d.) assumption. While recent Visual Foundation Models (VFMs) like CLIP show promise, existing prompt tuning methods often focus on task-specific adaptation rather than disentangling domain-invariant features from domain-specific ones. Consequently, these methods struggle to generate representations that generalize effectively across diverse, unseen domains. Furthermore, purely text-based guidance often fails to capture complex domain-specific variations (e.g., viewpoint or background changes) that are difficult to describe linguistically, leading to incomplete disentanglement.

Methodology
The authors propose PADG (Prompt Disentanglement via Language Guidance and Representation Alignment), a unified framework that leverages the flexibility of VFMs and Large Language Models (LLMs) to achieve robust domain generalization. The framework operates through two training stages and one inference strategy:

1. Cross-Modal Prompt Disentanglement (CMD): This stage addresses the separation of features across modalities.

   • GPT-Assist Text Disentanglement (GAT): The authors utilize an LLM (GPT-3) to generate fine-grained descriptions for each class within specific domains. Through latent semantic analysis, the LLM synthesizes these into domain-invariant descriptions (e.g., structural shape) and domain-specific descriptions (e.g., color or background). These descriptions are used to guide the learning of disentangled textual prompts via distillation and contrastive learning losses.
   • Image Disentanglement Guided by Text (IMT): The disentangled textual embeddings serve as semantic guides for the visual encoder. The visual encoder is fine-tuned using learnable prompts to produce corresponding domain-invariant and domain-specific visual features. This process is regularized by minimizing the distance between the learned visual features and the original CLIP embeddings while enforcing domain confusion for the invariant features.

2. Worst Explicit Representation Alignment (WERA): To compensate for the limitations of text-only guidance in capturing visual variations, this module enhances visual invariance through distributional robustness.

   • The method models a worst-case distribution within a Wasserstein ball centered on the source domain.
   • Instead of distorting image semantics, it employs a stylization strategy that adjusts the mean and variance (statistics) of intermediate visual features using learnable prompts.
   • An iterative optimization process searches for the worst-case stylized samples that maximize empirical risk. The model then aligns the representations of the original and stylized samples, ensuring the learned visual features remain stable under bounded distributional shifts.

3. Domain-Specific Prototype Learning (DSPL): During inference, the authors argue that relying solely on domain-invariant features is suboptimal. They introduce an ensemble strategy that combines domain-invariant predictions with domain-specific predictions.

   • Prototypes are constructed for each class within each source domain using the domain-specific visual encoder.
   • For a test sample, the model calculates the similarity between the query and these prototypes to generate a domain-specific prediction.
   • The final output is a weighted combination of the domain-invariant prediction and the domain-specific prototype-based prediction, leveraging knowledge from similar seen domains to aid unseen target classification.

Key Contributions

• PADG Framework: A novel prompt-tuning framework that fully harnesses the text modality to achieve cross-modal disentanglement, separating invariant and specific components in both text and visual spaces.
• WERA Module: A new module that adversarially stylizes visual features within a Wasserstein ball and enforces alignment between original and perturbed distributions. The authors provide theoretical analysis guaranteeing generalization bounds under this optimization.
• DSPL Strategy: A prototype-based ensemble mechanism that integrates domain-specific statistics with domain-invariant predictions to improve inference reliability.
• Comprehensive Evaluation: Extensive experiments demonstrating the method's efficacy across multiple benchmarks.

Experimental Results
The authors evaluated PADG on five mainstream DG benchmarks: PACS, VLCS, OfficeHome, DomainNet, and TerraIncognita.

• Multi-Source DG: PADG achieved state-of-the-art (SOTA) performance across all datasets, with an average accuracy of 78.73%, outperforming previous SOTA methods like SIMPLE+ (78.10%) and SMOS (68.30%). Notably, on the challenging TerraIncognita dataset, PADG improved upon the ERM baseline by 6.71% and surpassed CLIP-LoRA by 3.31%.
• Single-Source DG: In the more difficult single-domain generalization setting (training on one source, testing on multiple targets), PADG significantly outperformed existing methods, including UDIM and CLIP-based approaches, with improvements of up to 16.79% on OfficeHome.
• Ablation Studies: The authors confirmed the statistical significance of each component (GAT, IMT, WERA, DSPL). The results indicated that text disentanglement provides rich semantic cues, while WERA effectively reduces source-target divergence, and DSPL further refines inference by utilizing domain-specific knowledge.

Significance and Claims
The paper claims that PADG successfully reframes prompt tuning as a disentanglement problem, effectively addressing the limitations of existing methods that conflate task-specific and domain-invariant features. By combining the semantic richness of LLMs with the distributional robustness of Wasserstein-based learning, the method achieves robust performance even under significant style shifts and limited source data. The authors assert that their approach provides a lightweight yet effective fine-tuning framework that validates the hypothesis that explicit disentanglement of invariant and specific factors is crucial for generalization. The theoretical analysis further supports the claim that the proposed optimization guarantees robustness against distributional shifts within a defined radius.