Quantifying Data Similarity Using Cross Learning

1. Problem Statement

Measuring dataset similarity is a fundamental challenge in machine learning, critical for transfer learning, domain adaptation, few-shot learning, and multi-task learning.

• Limitations of Existing Methods: Most current approaches (e.g., Maximum Mean Discrepancy, $f$-divergences, Wasserstein distance) quantify similarity based solely on the input feature distributions ($P(X)$). They often neglect label information ($Y$) and the feature-response relationship ($P(Y|X)$).
• The Core Issue: Two datasets may have identical feature distributions but fundamentally different prediction tasks (e.g., different decision boundaries). Relying only on feature similarity can lead to negative transfer, where knowledge from a source dataset degrades performance on a target dataset.
• Goal: Develop a metric that directly quantifies the similarity of the underlying feature-response relationships to predict whether transferring knowledge between datasets will be beneficial.

2. Methodology: The Cross-Learning Score (CLS)

The authors propose the Cross-Learning Score (CLS), a metric that measures dataset similarity through bidirectional generalization performance.

2.1 Theoretical Definition

Let $D^{(t)}$ be the target dataset and $D^{(s)}$ be the source dataset. Let $f^*_t$ and $f^*_s$ be the optimal Bayes rules (learners) for the target and source tasks, respectively.
The CLS is defined as the symmetric average of the directional discrepancies in predictive performance:
$$\text{CLS} = \frac{1}{2} (\Delta_{s \to t} + \Delta_{t \to s})$$
Where:

• $\Delta_{s \to t} = E_{(X,Y) \sim P^{(t)}}[\ell(f^*_s(X), Y)] - E_{(X,Y) \sim P^{(t)}}[\ell(f^*_t(X), Y)]$

  • Interpretation: How much worse the source-optimal predictor performs on the target data compared to the target-optimal predictor.

• $\Delta_{t \to s}$ is defined symmetrically for the source distribution.

• Interpretation: A low CLS indicates that the optimal predictors for both tasks behave similarly across domains (high similarity). A high CLS indicates structural differences in the decision boundaries (low similarity).

• Range: For classification with 0-1 loss, CLS $\in [-1, 1]$.

2.2 Theoretical Foundation

The paper establishes a geometric interpretation of CLS under canonical linear models (Probit Regression and Linear Discriminant Analysis):

• Theorem 1 & 2: CLS is shown to be closely related to the cosine similarity between the decision boundaries (parameter vectors $\beta$ or mean vectors $\mu$) of the target and source tasks.
• Geometric Insight: CLS essentially measures the "angular misalignment" between the two tasks' decision boundaries. If the boundaries are aligned (cosine similarity $\approx 1$), CLS approaches 0. If they are orthogonal or opposed, CLS increases.

2.3 Computational Algorithms

Since the true Bayes rules are unknown, the authors propose empirical estimators ($\widehat{\text{CLS}}$):

1. Single-Model Estimator: Train a model $M$ on $D^{(t)}$ and evaluate on $D^{(s)}$ (and vice versa), subtracting the in-domain cross-validation error to isolate the transfer gap.
2. Robust Ensemble Estimator: To handle model misspecification, the authors propose aggregating scores from multiple models (e.g., Logistic Regression, SVM, XGBoost).
   • Weighted Average: Weights models based on their cross-validation performance.
   • Weighted Ensemble: Creates a single ensemble predictor with dataset-specific weights derived via cross-validation. This method consistently showed the lowest deviation from the "Oracle CLS" in experiments.

2.4 Transferability Zones

To make CLS actionable, the authors define three zones based on statistical thresholds ($\tau_1, \tau_2$) derived from the standard error of baseline errors:

• Positive Transfer (PT): $\text{CLS} \le \tau_1$. High similarity; transfer is likely to improve performance.
• Ambiguous Zone (AZ): $\tau_1 < \text{CLS} < \tau_2$. Uncertain benefit; transfer may or may not help.
• Negative Transfer (NT): $\text{CLS} \ge \tau_2$. Low similarity; transfer is likely to degrade performance.

2.5 Extension to Deep Learning (Encoder-Head CLS)

Standard CLS fails in deep learning because it does not account for fine-tuning. In deep transfer learning, the encoder is often shared and adapted.

• Solution: The authors introduce an Encoder-Head CLS framework:
  1. Stage 1: Jointly train a shared encoder on both $D^{(t)}$ and $D^{(s)}$ to learn aligned representations.
  2. Stage 2: Freeze the encoder and train task-specific "heads" for cross-domain evaluation.
• This isolates representation learning from task adaptation, providing a more accurate measure of transferability in deep learning pipelines.

3. Key Results

The paper validates CLS through extensive synthetic and real-world experiments.

3.1 Synthetic Experiments

• Monotonicity: CLS demonstrates a perfect monotonic relationship (Spearman correlation = 1.0) with controlled similarity levels (cosine similarity of parameters) across linear and nonlinear settings (Logistic, Probit, LDA, Mixture Gaussians, Nonlinear Regression).
• Comparison: Unlike KL Divergence, Wasserstein Distance, or OTDD (Optimal Transport Dataset Distance), CLS remains sensitive to changes in the label structure even when feature distributions are identical. Other metrics often fail (correlation $\approx 0$) when only the decision boundary changes.
• Estimation Accuracy: The Ensemble Estimator consistently achieved the lowest deviation from the Oracle CLS across diverse data-generating mechanisms, outperforming single-model estimators.

3.2 Real-World Applications

• eICU Mortality Prediction:
  • Task: Predicting post-ICU mortality across different hospitals.
  • Result: The CLS framework correctly identified 6 out of 10 source hospitals as "Positive Transfer" candidates. Empirical results confirmed that transfer learning improved performance for these hospitals, while "Ambiguous" zones showed mixed results.
• Canine Image Classification (Dogs vs. Wolves):
  • Task: Distinguishing dogs from wolves using deep learning (ResNet-18).
  • Source Data:
    1. Dogs vs. Wolves (Kaggle): Classified as Ambiguous. Empirically, it improved target performance but hurt source performance (mixed transfer).
    2. Cats vs. Dogs: Classified as Negative Transfer. Empirically, it degraded performance in both directions.
    3. Horses vs. Camels: Classified as Negative Transfer. Empirically, it failed to transfer.
  • Conclusion: The Encoder-Head CLS successfully predicted the direction and magnitude of transfer effects in complex vision tasks.

4. Key Contributions

1. Novel Metric (CLS): Introduced a metric that directly measures the similarity of feature-response relationships rather than just feature distributions.
2. Theoretical Justification: Proved that CLS is geometrically linked to the cosine similarity of decision boundaries in linear settings, providing a rigorous theoretical basis.
3. Robust Estimation: Developed an ensemble-based estimation procedure that is robust to model misspecification and scalable to high dimensions without density estimation.
4. Deep Learning Adaptation: Extended the concept to modern Encoder-Head architectures, addressing the specific challenges of fine-tuning in deep transfer learning.
5. Actionable Framework: Proposed a statistical framework to categorize datasets into Positive, Ambiguous, and Negative transfer zones, aiding in dataset selection and curation.

5. Significance

• Solving Negative Transfer: By focusing on the alignment of decision boundaries rather than just data distribution, CLS offers a more reliable way to avoid negative transfer, a major bottleneck in practical transfer learning.
• Computational Efficiency: Unlike Optimal Transport methods (OTDD), which are computationally expensive and sensitive to dimensionality, CLS bypasses explicit density estimation, making it scalable.
• Practical Utility: The "Transferable Zones" framework provides a principled, data-driven method for practitioners to decide which source datasets to use for a specific target task, optimizing resource allocation in multi-source learning and federated learning scenarios.
• Broad Applicability: The method is validated across regression, classification, linear, nonlinear, and deep learning settings, making it a versatile tool for the broader AI community.