Minimax optimal adaptive structured transfer learning through semi-parametric domain-varying coefficient model

1. Problem Statement

The paper addresses the challenge of transfer learning in settings where data is collected across multiple heterogeneous environments (domains). The core difficulty lies in conditional distributional drift: the relationship between covariates ($X$) and responses ($Y$) varies systematically across domains, indexed by a domain identifier $U$.

• The Goal: Improve inference for a specific target domain ($u_0$) by leveraging data from $K$ source domains.
• The Challenge:
  • Negative Transfer: Blindly pooling data from unrelated source domains can degrade performance compared to using only target data.
  • Heterogeneity Modeling: Existing methods often assume fixed coefficients (invariant) or fully heterogeneous coefficients. There is a lack of frameworks that explicitly model the smooth variation of coefficients across domains to interpolate between these regimes.
  • Inference: Most transfer learning methods focus on point estimation and lack valid uncertainty quantification (e.g., confidence intervals) because the estimators involve complex, data-adaptive pooling and shrinkage.

2. Methodology

The authors propose a Semi-Parametric Domain-Varying Coefficient Model (DVCM) and a two-step adaptive estimator.

A. The Model: Domain-Varying Coefficient Model (DVCM)

Instead of assuming a fixed coefficient vector $\theta$ or a completely distinct $\theta_k$ for each domain, the authors model the coefficient as a smooth function of the domain identifier $U$:
$$E[Y | X, U] = g^{-1}(X^\top \theta(U))$$
where $g$ is a link function (identity for linear, logit for logistic, etc.), and $\theta(\cdot)$ belongs to a Hölder smoothness class $\mathcal{H}(\beta, L)$. This structure allows information sharing between domains with similar $U$ values.

B. The Estimator: Two-Step Adaptive Procedure

The proposed estimator, $\hat{\theta}_{TL}(u_0)$, combines nonparametric pooling with ridge regularization:

1. Step I: Nonparametric Pilot Estimation

   • Construct a pilot estimator $\hat{\theta}_{DVCM}(u_0)$ using local polynomial regression on the pooled data from all domains (source and target), weighted by the proximity of their domain identifiers $U_k$ to the target $u_0$.
   • This step effectively borrows strength from nearby domains but is susceptible to bias if the smoothness assumption is violated or if the bandwidth is misspecified.

2. Step II: Adaptive Fine-Tuning (Ridge Regularization)

   • Refine the pilot estimator using only the target domain data via a ridge-regularized regression:
     $$\hat{\theta}_{TL}(u_0) = \arg\min_{\alpha} \left( \frac{1}{n_0} \sum_{i \in I_0} \ell(X_{0i}^\top \alpha, Y_{0i}) + \frac{1}{2} \|\alpha - \hat{\theta}_{DVCM}(u_0)\|_Q^2 \right)$$
   • Key Innovation: The penalty matrix $Q$ is chosen data-adaptively based on the inverse of the Mean Squared Error (MSE) of the pilot estimator.
   • Mechanism:
     • If the pilot estimator is precise (low MSE), $Q$ is large, causing strong shrinkage toward the pooled estimate.
     • If the pilot estimator is noisy or biased (high MSE), $Q$ is small, causing the estimator to revert to the target-only solution.
   • This design guarantees robustness against negative transfer: the risk of $\hat{\theta}_{TL}$ is never worse than the target-only estimator.

C. Implementation Details

• Data Splitting: To ensure theoretical independence between the pilot and the fine-tuning steps, the target data is split into two halves.
• Estimation of $Q$: The authors provide a fully data-driven recipe to estimate the unknown MSE components (bias and variance) of the pilot estimator using plug-in and sandwich estimators.

3. Key Theoretical Contributions

A. Minimax Optimality

The paper establishes that the proposed estimator achieves the minimax-optimal convergence rate for estimating $\theta(u_0)$. The rate is characterized by a phase transition depending on:

• $n_0$: Target sample size.
• $n$: Total sample size (target + sources).
• $K$: Number of source domains.
• $\beta$: Smoothness of the coefficient function $\theta(\cdot)$.
• $\gamma$: Dispersion of domain identifiers (proximity of sources to target).

The optimal rate is:
$$\text{Rate} \asymp n_0^{-1} \wedge \max\left( (K/\gamma)^{-2\beta}, (n/\gamma)^{-\frac{2\beta}{2\beta+1}}, n^{-1} \right)$$

• Implication: The estimator automatically adapts to the "best" regime. If sources are close and smooth, it achieves the faster nonparametric rate. If sources are distant or uninformative, it defaults to the parametric target-only rate ($n_0^{-1}$).

B. Asymptotic Normality and Inference

A major contribution is the derivation of the asymptotic distribution of the adaptive estimator. Despite the data-adaptive selection of the penalty matrix, the authors prove that under appropriate undersmoothing conditions:
$$\hat{\Sigma}_{TL}^{-1/2} (\hat{\theta}_{TL}(u_0) - \theta(u_0)) \xrightarrow{d} N(0, I_p)$$
This enables the construction of valid confidence intervals and hypothesis tests that correctly account for the uncertainty introduced by the transfer learning process.

C. Negative Transfer Guarantee

Theoretical proofs demonstrate that the estimator is immune to negative transfer. Its Mean Squared Error (MSE) is bounded above by the minimum of the MSEs of the target-only estimator and the pooled nonparametric estimator.

4. Results and Validation

• Simulation Studies:
  • Adaptivity: The estimator successfully tracks the better of the target-only or pooled estimators across varying levels of domain heterogeneity ($\gamma$) and number of sources ($K$).
  • Phase Transitions: Log-log plots of MSE vs. parameters ($K, \gamma, n$) confirm the theoretical phase transitions and convergence rates.
  • Normality: Histograms of standardized estimators align with the standard normal distribution, validating the inference procedures.
• Real-Data Applications:
  1. SLID-Ontario Dataset: Predicting hourly wages based on demographics, with "years of employment" as the domain identifier. The method improved prediction accuracy in data-sparse regions by borrowing from nearby employment cohorts.
  2. US Adult Income Dataset: Predicting income >$50k. The adaptive transfer learning estimator consistently outperformed or matched the baselines, demonstrating robustness in a binary classification setting.

5. Significance and Impact

• Bridging Parametric and Nonparametric: The DVCM framework provides a structured, interpretable way to handle domain drift, moving beyond rigid invariance assumptions or unstructured black-box adaptation.
• Theoretical Rigor: It is one of the few works to provide matching minimax lower and upper bounds for transfer learning with structured heterogeneity, proving that the proposed method is statistically optimal.
• Practical Utility: By solving the "negative transfer" problem and providing valid uncertainty quantification, the method makes transfer learning safer and more reliable for high-stakes applications (e.g., healthcare, economics) where over-reliance on source data can be detrimental.
• Generalizability: The methodology extends from linear models to Generalized Linear Models (GLMs), covering regression, logistic regression, and Poisson regression.

In summary, this paper provides a theoretically grounded, computationally efficient, and adaptive solution for transfer learning in the presence of smooth domain drift, ensuring that leveraging external data never harms performance while enabling rigorous statistical inference.