Technical Summary: RADAR (Relative Angular Divergence Across Representations)

Problem Statement

In modern machine learning, expanding training datasets with auxiliary source domains is a standard strategy to address data constraints. However, this practice does not guarantee improved downstream performance and can lead to negative transfer, where incorporating source information actively degrades target performance. Identifying which source domains will yield positive transfer is difficult because empirically evaluating each candidate requires costly full-model retraining. Existing transferability metrics often rely on static representations at a single layer or require pre-training on the target dataset, failing to capture the dynamic structural evolution of representations across the model hierarchy.

Methodology: RADAR

The authors propose RADAR (Relative Angular Divergence Across Representations), a geometrically grounded metric designed to estimate cross-domain transferability using only frozen features from foundation models. The method operates by analyzing the layer-wise evolution of representations rather than static snapshots.

Core Mechanism

1. Layer-wise Trajectory Extraction: For a pre-trained foundation model $\Phi$, intermediate feature representations $h^{(l)}(x)$ are extracted at every layer $l$.
2. Geometric Triangle Construction: For a given anchor sample $x$ and a secondary sample $x'$ (either within the same domain or across domains), the method constructs a geometric triangle of displacement vectors spanning layers $l$ and $l+1$:
   • Separation ($v_{sep}$): The vector between $h^{(l)}(x)$ and $h^{(l)}(x')$.
   • Detour ($v_{detour}$): The vector from $h^{(l)}(x')$ to $h^{(l+1)}(x)$.
   • Trajectory ($v_{traj}$): The direct displacement from $h^{(l)}(x)$ to $h^{(l+1)}(x)$.
   • Note: The paper proves that $v_{sep} + v_{detour} = v_{traj}$ exactly.
3. Descriptor Computation: The method computes two key descriptors for these triangles:
   • Angular Alignment ($\theta$): The cosine similarity between the separation and detour vectors.
   • Relative Distance ($d$): The normalized excess path length of the two-step detour compared to the direct trajectory ($d = \frac{\|v_{sep}\| + \|v_{detour}\| - \|v_{traj}\|}{\|v_{traj}\|}$).
   • Topological Adaptation: While the metric can be defined on geodesic distances for hyperspherical spaces (e.g., DINO features), the authors adopt a unified Euclidean space formulation as it proved most robust across modalities, particularly for text.
4. Distribution Modeling:
   • The method samples pairs to compute empirical distributions of $\theta$ and $d$ for within-domain pairs and cross-domain pairs.
   • To ensure computational feasibility, a stratified sampling strategy is used, weighting pairs based on their distance to class centroids (capturing both inlier-inlier and inlier-outlier dynamics).
   • Gaussian Mixture Models (GMMs) are fitted to these distributions to model the continuous probability densities of the within-domain and cross-domain dynamics.
5. Divergence Measurement: The final transferability score is derived from the symmetric Kullback-Leibler (KL) divergence between the two GMMs. A lower divergence indicates higher structural similarity and, consequently, higher predicted transferability.
6. Extended Trajectories: To capture holistic structural shifts, the metric concatenates descriptors across a window of layers (e.g., $l-6$ to $l+6$), creating a high-dimensional feature vector before fitting the GMM.

Key Contributions

1. Dynamic, Layer-Wise Structural Evolution: RADAR is the first metric to evaluate transferability by measuring the divergence between dynamic, continuous distributions of geometric trajectories (angles and relative distances) across network layers, rather than static feature distributions.
2. Highly Competitive Predictive Performance: The metric achieves top-three performance in 7 out of 10 configurations across vision and text benchmarks, securing the absolute best performance in 6 of them. It consistently outperforms or matches state-of-the-art baselines like LEEP, LogME, NCE, TransRate, and S-OTDD.
3. Robust and Efficient Estimation: Through extensive ablation studies, the authors demonstrate that RADAR is robust to hyperparameter choices (sample size, window radius, temperature) and computationally efficient, often outperforming baselines in CPU-only environments due to its reliance on frozen features and subsampling.

Experimental Results

The authors evaluated RADAR on:

• Vision Modalities: CLIP and DINOv3 backbones on DomainNet, OfficeHome, and PACS.
• Text Modalities: Qwen3-Embedding and EmbeddingGemma on EuroEval and AmazonReviews.

Key Findings:

• Performance: RADAR achieved the best Mean Correlation Improvement (MCI) on DomainNet for both CLIP and DINOv3, and on OfficeHome for CLIP. In text, it achieved the best results on EuroEval for EmbeddingGemma (outperforming the next best baseline by >30 percentage points) and was competitive on AmazonReviews.
• Domain Separation Regimes: RADAR excels when domain transitions are smooth (low corruption/separation) or cleanly separated (high corruption/separation). Its performance degrades in "intermediate" regimes (e.g., PACS dataset) where domains exhibit scattered, non-uniform separation that is neither smooth enough for trajectory alignment nor distinct enough for clean divergence estimation.
• Modality Differences: Text representations strictly favor Euclidean space and angular features, while vision representations show robustness to both Euclidean and pseudo-Cartesian spaces.
• Robustness to Corruption: On ImageNet-C, RADAR demonstrated exceptional robustness at extreme corruption levels (Severity 5), where static statistical metrics failed, though it showed vulnerability to moderate synthetic noise (Severity 3).

Significance and Claims

The paper claims that layer-wise geometric structure carries a meaningful transferability signal that static, single-layer metrics fail to capture. By measuring the divergence of how representations evolve across layers, RADAR provides a principled, computationally efficient way to predict transfer gain rankings without retraining models.

The authors emphasize that their approach is motivated by the Perceptual Straightening Hypothesis and the observation that the most informative representations in foundation models are not always at the final layer. They argue that RADAR offers a cross-modal, unified framework for transfer learning decisions, particularly effective when domain shifts are well-defined. However, they modestly acknowledge limitations, noting that the metric's predictive power degrades in scenarios with scattered, overlapping domain structures and that the reliance on GMMs presents a computational bottleneck for real-time filtering.