Technical Summary: DOME - Learning Transferable Domain Variables from Sparse Supervision for Test-Time Adaptation

1. Problem Statement

Test-Time Adaptation (TTA) aims to align a pre-trained model with shifting test domains using only unlabeled streaming data. Existing TTA methods predominantly rely on an "implicit search" paradigm, where they attempt to infer a single, global statistical distribution to represent the entire target domain. The authors argue this approach is fundamentally flawed for real-world scenarios because:

• Multidimensionality: Real-world domain shifts are multidimensional and sample-specific (e.g., an image can simultaneously belong to an "artistic" domain and an "animal" domain).
• Information Loss: Forcing all target data into a single unified representation eliminates sample-specific domain cues.
• Instability: The conflict between multidimensional sample realities and the objective of learning a single global distribution leads to unstable optimization and local minima.
• Fragility: Current methods often require complex heuristics or intricate adaptation algorithms to mitigate these issues, yet they remain fragile under diverse shifts.

The paper poses a central question: Can we explicitly model each sample's domain instead of relying on implicit search?

2. Methodology: DOME and the Sparse-to-Dense (S2D) Framework

The authors propose DOME (DOMain Encoder), a zero-shot domain encoder that explicitly generates domain representations for downstream samples. DOME is trained via a Sparse-to-Dense (S2D) learning framework, which consists of two synergistic components:

A. Sparse-Label Domain Disentanglement (SDD)

Since dense domain annotations are unavailable, SDD learns to isolate domain-specific factors from category semantics using sparse supervision.

• Momentum Sparse Bank: A domain bank stores sparse, domain-related features updated via a momentum strategy ($m=0.9$). This reinforces shared domain information while suppressing instance-specific noise.
• Disentanglement Mechanism:
  1. Intermediate tokens from a CLIP vision encoder have positional encodings removed to suppress category-related cues.
  2. A lightweight Dictionary Decoder maps the image representation into a high-dimensional, overcomplete space.
  3. Sparsity Regularization: An $\ell_1$ loss encourages the model to activate only a sparse subset of interpretable features (canonical domain attributes).
  4. JumpReLU: A learnable threshold suppresses residual weak activations, yielding a clean, sparse domain signal ($z_{dom}$).

B. Dense Domain Representations Learning (DDRL)

This component extrapolates the sparse annotations into a continuous, dense domain space to enable zero-shot generalization.

• Vision-Language Pretraining: DOME is initialized from frozen CLIP image and text encoders. During training, partially masked domain text guides the extraction of domain variables, reducing ambiguity in the multidimensional space.
• Reparameterization: The model predicts channel-wise mean ($\hat{\mu}$) and standard deviation ($\hat{\sigma}$) to represent domain statistics.
• Training Objectives: The dense features ($\hat{z}$) are trained to align with the clean sparse signals ($z_{dom}$) via:
  • MSE Loss: Regresses dense projections toward sparse prototypes.
  • Auxiliary Loss: Aligns projections with raw domain statistics.
  • InfoNCE Loss: Encourages intra-domain cohesion and inter-domain separation.

C. Test-Time Adaptation (TTA) Integration

At inference, DOME operates as a frozen module:

1. Domain Injection: DOME processes the test image (and its weak augmentation) to generate domain statistics.
2. Adapter Modulation: A lightweight MLP adapter converts these statistics into affine parameters ($g, b$).
3. Feature Modulation: These parameters modulate the features of a frozen ViT backbone via $x = x \cdot (1 + g) + b$ after LayerNorm.
4. Optimization: Only the adapter and normalization layers are updated using a standard entropy-minimization objective (conditional and marginal entropy) over a single forward-backward pass. No complex self-training or pseudo-labeling is required.

3. Key Contributions

1. Explicit Domain Injection: DOME is the first general-purpose domain-specific feature extractor in TTA literature, transforming the paradigm from implicit domain search to explicit domain injection.
2. Sparse-to-Dense Learning Framework: A novel training strategy that uses a momentum-updated sparse bank for disentangled supervision and vision-language pretraining to extrapolate dense, continuous domain representations for zero-shot generalization.
3. Simplicity and Performance: The authors demonstrate that robust adaptation stems from explicit, structured domain representation rather than complex algorithms. Even a naive entropy-minimization strategy combined with DOME achieves state-of-the-art results.

4. Experimental Results

The method was evaluated on ImageNet-C, ImageNet-R, and ImageNet-Sketch using various backbones (ViT-B, ViT-L, DINOv2).

• ImageNet-C: DOME + Entropy Minimization achieved 66.7% average accuracy, outperforming complex baselines like DeYO (66.4%) and FOA (66.3%). It showed particular strength in Motion Blur, Zoom Blur, and Elastic Transform.
• ImageNet-R: Achieved 68.7% top-1 accuracy, matching the best-performing complex method (DeYO) and effectively handling style shifts.
• ImageNet-Sketch: Set a new state-of-the-art at 53.9%, significantly surpassing DeYO (50.3%), demonstrating robustness when texture cues are absent.
• Generalization: In offline TTA settings across unseen artistic domains, DOME consistently outperformed a baseline using raw CLIP statistics (DomStat), improving accuracy from 52.81% to 68.34% on ViT-B/16.
• Efficiency: DOME achieves SOTA performance with 4$\times$ fewer trainable parameters than full-model adaptation methods (e.g., CoTTA) while maintaining real-time latency.

5. Significance and Claims

The paper claims that the primary driver of robust TTA is not the complexity of the adaptation algorithm (e.g., self-training, prototype alignment), but the quality of the domain representation.

• Paradigm Shift: By explicitly modeling domain variables, the inference process becomes data-dependent and customizable to each sample's multidimensional domain, rather than being optimized for a potentially inaccurate global distribution.
• Disentanglement: The S2D framework successfully disentangles domain cues from semantic information, allowing the model to focus on semantic information during adaptation while being guided by explicit domain cues.
• Zero-Shot Capability: Leveraging vision-language pretraining allows DOME to generalize to unseen domains without requiring specific domain labels during the downstream TTA phase.

The authors conclude that explicit, structured domain representation is the key to solving the fragility of current TTA methods, enabling simple strategies to achieve state-of-the-art performance across diverse and continuous domain shifts.