PAS: Estimating the target accuracy before domain adaptation

1. Problem Statement

Unsupervised Domain Adaptation (UDA) aims to transfer knowledge from a labeled source domain to an unlabeled target domain. However, the success of UDA is highly dependent on two critical choices:

1. Source Domain Selection: Choosing a source domain that is compatible with the target. An inappropriate choice can lead to negative transfer, where performance degrades below the baseline.
2. Pre-trained Feature Extractor Selection: Choosing a pre-trained model (backbone) that captures invariant features suitable for the target task.

The Challenge: In real-world scenarios, target labels are unavailable. Existing transferability estimation methods (e.g., H-score, LEEP) require target labels and are therefore inapplicable. Furthermore, evaluating every combination of source domains and pre-trained models via full domain adaptation training is computationally prohibitive. There is a lack of a metric that can estimate transferability before training, specifically accounting for the asymmetric nature of domain shifts (transferring from easy-to-hard vs. hard-to-easy).

2. Methodology: Potential Adaptability Score (PAS)

The authors propose PAS, a novel, unsupervised metric designed to estimate the potential success of a specific source domain and pre-trained model combination for a target task.

Core Assumptions

PAS relies on the assumption that a high-quality pre-trained model extracts invariant discriminative features. Consequently:

• Samples from the same class (even across different domains) should be close in the embedding space.
• Samples from different classes should be distant.
• Ideally, a target sample should be closer to the centroid of its true source class than to any other class centroid.

Algorithm

1. Embedding Generation: A pre-trained feature extractor $f_\theta$ maps source samples ($x^S$) and target samples ($x^T$) into an embedding space $Z$.
2. Centroid Calculation: For each source class $c$, a centroid $\mu_c$ is computed as the normalized sum of the embeddings of all source samples in that class:
   $$\mu_c = \frac{\sum_{x^S_i \in S^S_c} f_\theta(x^S_i)}{\|\sum_{x^S_i \in S^S_c} f_\theta(x^S_i)\|}$$
3. Distance Calculation: For each target sample $x^T_i$, the cosine distance to all source class centroids is calculated:
   $$\text{dist}(f_\theta(x^T_i), \mu_c) = 1 - (f_\theta(x^T_i) \cdot \mu_c)$$
4. PAS Computation: For each target sample, let $d_{1i}$ be the distance to the closest centroid and $d_{2i}$ be the distance to the second closest centroid. The PAS score is the average of the normalized margin across all target samples:
   $$\text{PAS}(\theta, D_S, D_T) = \frac{1}{|D_T|} \sum_{i} \frac{d_{2i} - d_{1i}}{d_{2i}}$$

Key Design Choices:

• Asymmetry: Unlike symmetric metrics (MMD, Wasserstein), PAS is asymmetric. It measures how well the target fits into the source structure.
• Cosine Distance: Used instead of Euclidean distance to ignore magnitude shifts (e.g., lighting changes) and focus on angular differences, mitigating the "curse of dimensionality."
• Unsupervised Nature: It does not require target labels; it assumes the closest source cluster represents the "true" class for the target sample during estimation.

3. Key Contributions

1. Novel Metric (PAS): The first proposal for transferability estimation specifically tailored for the domain adaptation setting (where target labels are missing).
2. Selection Framework: A practical framework to select the optimal source domain and pre-trained model from a pool of candidates before running expensive domain adaptation algorithms.
3. Efficiency: PAS is computationally cheap (linear complexity) and robust to sample size reduction, allowing for rapid screening of candidates.
4. Empirical Validation: Extensive experiments demonstrating that PAS correlates strongly with final target accuracy, outperforming existing symmetric metrics.

4. Experimental Results

The authors evaluated PAS on four standard benchmarks: Office-Home, Office-31, ImageCLEF, and DomainNet, using various pre-trained backbones (ResNet, DeiT, ViT, Swin) and domain adaptation methods (DANN, MCC, etc.).

• Correlation with Accuracy:
  • PAS achieved a Spearman rank correlation of 0.88 with target accuracy across all scenarios, significantly outperforming MMD (0.37) and A-distance (-0.16).
  • In specific target domains (e.g., Office-Home "C" target), PAS reached correlations as high as 0.81.
• Source Selection: PAS successfully identified the source domain that yielded the highest accuracy in the majority of cases.
• Backbone Selection: PAS correctly identified which pre-trained model (e.g., ViT-Base vs. ResNet-50) would perform best for a specific adaptation task.
• Robustness: The relative ranking of PAS scores remained stable even when using only a subset of the data (e.g., 10% of samples), making it scalable.
• Limitations: The metric showed lower correlation on the ImageCLEF dataset, particularly for the "Pascal" domain. The authors attribute this to multi-object images where the target sample is visually similar to a source class centroid but semantically belongs to a different object in the scene.

5. Significance and Impact

• Cost Reduction: By predicting the best source/model combination before training, PAS drastically reduces the computational overhead and time required for hyperparameter and architecture search in domain adaptation.
• Avoiding Negative Transfer: It provides a safety mechanism to avoid selecting source domains that would harm target performance.
• Practical Utility: The method is "plug-and-play," requiring only the pre-trained model and the datasets (no target labels or training), making it highly applicable to real-world scenarios where data is scarce or expensive to label.
• Future Directions: The authors suggest extending PAS to multi-source domain adaptation and validating it on non-vision modalities (e.g., NLP, audio).

In summary, PAS bridges the gap between theoretical transferability estimation and practical domain adaptation deployment by offering a fast, label-free, and highly correlated predictor of adaptation success.