Nonparametric Goodness-of-fit Testing under Covariate Shift

Technical Summary: Nonparametric Goodness-of-fit Testing under Covariate Shift

Problem Formulation
This paper addresses the problem of nonparametric goodness-of-fit testing in the presence of covariate shift. In this setting, labeled data $\mathcal{D} = \{(X_i, Y_i)\}_{i=1}^n$ are drawn from a source distribution $P$, but the goal is to evaluate the performance of a regression function $f^*$ under a target distribution $Q$. The marginal distributions of the covariates differ ($P_X \neq Q_X$), while the conditional distribution of the response remains invariant ($P_{Y|X} = Q_{Y|X}$).

The primary inferential objective is to test the null hypothesis:
$$H_0: \|f_0 - f^*\|_{L^2(Q)} = 0 \quad \text{vs.} \quad H_1: \|f_0 - f^*\|_{L^2(Q)} > 0,$$
where $f_0$ is a candidate function and the norm is defined with respect to the target measure $Q_X$. The challenge lies in constructing valid confidence sets for $f^*$ in the $L^2(Q)$ geometry using only source-domain samples, particularly when the target-to-source density ratio $\rho(x) = dQ_X/dP_X$ is unbounded or exhibits heavy tails, which can destabilize standard importance-weighting estimators.

Methodology
The proposed framework combines truncated importance-weighted kernel ridge regression (IWKRR) with a multiplier bootstrap procedure.

1. Truncated IWKRR Estimator:
   Standard importance weighting (IW) estimates the target risk by weighting source samples by $\rho(x)$. However, large values of $\rho(x)$ can cause high variance. The authors employ a truncated weight function $w_n(x) = \min\{\rho(x), \tau_n\}$, where $\tau_n$ is a pre-specified truncation level. The estimator $\hat{f}$ minimizes the truncated empirical risk:
   $$\hat{f} = \arg\min_{f \in \mathcal{H}} \frac{1}{n} \sum_{i=1}^n w_n(X_i)(Y_i - f(X_i))^2 + \lambda_n \|f\|_{\mathcal{H}}^2,$$
   where $\mathcal{H}$ is a Reproducing Kernel Hilbert Space (RKHS) and $\lambda_n$ is the regularization parameter.

2. Multiplier Bootstrap for Confidence Sets:
   To construct a $100(1-\alpha)\%$ confidence ball $CS_\alpha(\hat{f}) = \{f \in \mathcal{H} : \|\hat{f} - f\|_{L^2(Q)} \leq \hat{c}_{\alpha,n}\}$, the authors utilize a multiplier bootstrap.

   • Bias-Variance Decomposition: The error $\sqrt{n}(\hat{f} - f^*)$ is decomposed into a bias term (negligible under under-smoothing) and a stochastic fluctuation term.
   • Bahadur Representation: The stochastic term is shown to admit a Bahadur representation involving influence functions $U_i$.
   • Bootstrap Statistic: A computable bootstrap statistic $B$ is constructed using anti-symmetric multipliers $h_{ij} \sim N(0,1)$ and empirical residuals. The radius $\hat{c}_{\alpha,n}$ is determined by the empirical quantiles of $\|B\|_{L^2(Q)}$.
   • Computational Efficiency: The method leverages the matrix structure of the RKHS to compute $\|B\|_{L^2(Q)}$ efficiently via quadratic forms, avoiding the need to re-fit the model in every bootstrap iteration.

Theoretical Framework and Assumptions
The validity of the inference relies on several key conditions:

• Compatibility Conditions: The paper introduces "regularized transfer coverage" (Assumption 4.4) and "non-degenerate covariance" (Assumption 4.5). These conditions ensure that the geometry of the weighted source distribution (controlled by the truncated weights) is sufficiently compatible with the target $L^2(Q)$ geometry to allow for valid error transfer and anti-concentration properties.
• Density Ratio Conditions: The theory accommodates three regimes for the density ratio $\rho(x)$:
  1. Bounded density ratio.
  2. Sub-exponential tail conditions (Bernstein-type moment conditions).
  3. Heavy-tailed distributions with bounded higher-order moments (requiring appropriate truncation).
• Spectral Conditions: The results depend on the spectral decay of the kernel integral operator $L_Q$, covering both polynomial decay ($\mu_k \asymp k^{-\beta}$) and exponential decay.

Key Contributions and Results

1. Nonasymptotic Validity: The paper establishes nonasymptotic bounds on the coverage error of the bootstrap confidence sets. The error rate depends on the bias, Gaussian approximation error, and bootstrap approximation error.
2. Explicit Error Rates: Under specific conditions on the density ratio and kernel spectral decay, the authors derive explicit convergence rates for the miscoverage probability $|P(f^* \notin CS_\alpha(\hat{f})) - \alpha|$. For instance, under bounded density ratios and polynomial spectral decay, the error rate is shown to be of order $O(n^{-\gamma} \log^3 n)$ for a specific exponent $\gamma$ determined by the smoothness and dimension parameters.
3. Operator-Theoretic Treatment: A novel technical contribution is the use of unitary isomorphisms generated by the square roots of the target and weighted-source integral operators. This allows the comparison of stochastic fluctuations in different geometries ($L^2(Q)$ vs. RKHS) within a common operator calculus, capturing both the bias of the truncated estimator and its covariance structure.
4. Empirical Validation:
   • Simulations: Numerical experiments demonstrate that the bootstrap radius is well-calibrated and that the confidence sets achieve empirical coverage close to the nominal level, even when the density ratio is large.
   • Real Data Application: The method is applied to the Survey of Consumer Finances (SCF) to test hypotheses regarding household net worth and the monotonicity of stock equity holding probabilities with respect to income. The results show the ability to reject specific candidate functions (e.g., source-only KRR) and test composite hypotheses (e.g., monotonicity) in a survey-weighted setting.

Significance and Claims
The paper claims to provide a general inference framework for evaluating regression functions under covariate shift, a problem that has been less understood than the estimation problem itself. By combining truncation to stabilize variance with a multiplier bootstrap to quantify uncertainty, the authors offer a procedure that is:

• Robust: Applicable even when the density ratio has heavy tails, provided the truncation level is chosen appropriately.
• Theoretically Grounded: Proven to have nonasymptotic coverage guarantees under explicit compatibility and spectral conditions.
• Practically Feasible: Computationally efficient, requiring only matrix-vector multiplications after the initial estimator is computed.

The authors emphasize that their approach allows for the construction of confidence sets in the target distribution's geometry, addressing the critical issue that a function close to the truth in the source distribution may not be close in the target distribution. The work bridges the gap between robust estimation under covariate shift and valid statistical inference.