Towards regularized learning from functional data with covariate shift

Technical Summary: Towards Regularized Learning from Functional Data with Covariate Shift

Problem Statement
This paper addresses the challenge of vector-valued regression under the covariate shift assumption, specifically within the context of functional data where both input and output variables are functions. In standard statistical learning, it is assumed that training and testing data are drawn from the same distribution. However, in many practical scenarios—such as structural health monitoring or medical imaging—the input distributions of the source (training) and target (testing) domains differ, while the conditional distribution of the output given the input remains invariant.

The authors focus on the setting where the source distribution is $p(x, y) = p(y|x)p_X(x)$ and the target distribution is $q(x, y) = p(y|x)q_X(x)$. The goal is to learn a regression function $f^*: X \to Y$ that minimizes the expected risk over the target marginal distribution $q_X$, despite having access only to labeled source samples $\{(x_i, y_i)\}$ and unlabeled target inputs $\{x'_j\}$. This problem is particularly complex when the output space $Y$ is an infinite-dimensional Hilbert space (functional data), a scenario less explored in existing literature compared to finite-dimensional settings.

Methodology
The proposed framework utilizes Vector-Valued Reproducing Kernel Hilbert Spaces (vRKHS) to handle functional outputs. The methodology proceeds through several key stages:

1. Operator Learning Formulation:
   The authors leverage the isomorphism between the vRKHS $\mathcal{G}$ and the space of Hilbert-Schmidt operators $\mathcal{S}_2(\mathcal{H}, Y)$, where $\mathcal{H}$ is the scalar RKHS induced by a scalar multiplicative kernel $K(x, t) = k(x, t)I_Y$. This allows the regression problem to be recast as finding an optimal operator $C \in \mathcal{S}_2(\mathcal{H}, Y)$ such that $f(\cdot) = C\phi(\cdot)$.

2. Importance Weighted Regularized Least Squares (IWRLS):
   To account for the covariate shift, the empirical risk is reweighted using a function $\beta(x)$ such that $dq_X(x) = \beta(x)dp_X(x)$. The empirical risk functional is expressed in terms of the source data and importance weights. The authors derive a regularized solution using a general family of spectral filter functions $g_\lambda$, leading to an estimator $\hat{C}_{\lambda}^\beta$ defined via the empirical covariance operator $\hat{C}_X^\beta$.

3. Theoretical Analysis:
   The paper establishes optimal convergence rates for the proposed estimator in both the $L^2(q_X, Y)$ norm and the vRKHS norm. The analysis relies on:

   • General Source Conditions: The target operator $C^*$ is assumed to satisfy $C^* = C_0 \phi(C_X)$, where $\phi$ is a general index function characterizing the smoothness of the solution.
   • Spectral Decay: Assumptions are made regarding the polynomial decay of the eigenvalues of the covariance operator $C_X$.
   • Concentration Inequalities: Theoretical bounds are derived using concentration inequalities for Hilbert-space-valued random variables to control the deviation between empirical and population operators.

4. Aggregation Strategy:
   Recognizing the difficulty of selecting optimal regularization parameters ($\lambda$) and kernels ($k$) in practice, the authors propose an aggregation-based approach. This method constructs a linear combination of estimators generated by different regularization parameters and/or different kernels. The weights for this linear combination are determined by solving a system of linear equations involving empirical estimates of the Gram matrix and the cross-covariance vector, approximated using the source data and importance weights.

Key Contributions

• Operator Learning Algorithm: The paper proposes the first systematic operator-learning algorithm for vector-valued regression under covariate shift within the vRKHS framework, specifically tailored for functional data.
• Optimal Convergence Rates: The authors establish optimal convergence rates for general regularization schemes under general source conditions. These rates are proven to be minimax optimal, coinciding with known results for scalar-valued settings when specialized to the functional case.
• Aggregation Framework: A novel aggregation strategy is introduced to mitigate the sensitivity of kernel-based learning to the choice of tuning parameters and kernels. Theoretical guarantees are provided showing that the aggregated estimator performs at a level comparable to the best individual estimator in the ensemble.
• Extension to Functional Data: The work bridges the gap between existing covariate shift literature (mostly finite-dimensional) and functional data analysis, providing a theoretical foundation for learning in infinite-dimensional output spaces under distributional shifts.

Experimental Results
The authors validate their method using a real-world face image dataset (The 10k US Adult Faces Database). The task involves reconstructing non-distorted face images from their sinogram representations (Radon transforms), where the target domain involves blurred sinograms (covariate shift).

• Setup: Two experimental cases were tested: (1) blurring applied to the sinograms, and (2) blurring applied to the face images prior to sinogram generation.
• Performance: The proposed method was evaluated using Mean Squared Error (MSE), Relative Error, and Peak Signal-to-Noise Ratio (PSNR).
• Findings:
  • The regularized learning approach successfully recovered substantial facial information despite severe input degradation, outperforming or matching standard inverse Radon transform (iradon) baselines.
  • The aggregation method yielded results very close to the best-performing individual regularization parameter found via a "lambda sweep," demonstrating its effectiveness in avoiding the need for exhaustive parameter tuning.
  • In multiple kernel learning experiments (combining Gaussian, Cauchy, Exponential, and IMQ kernels), the aggregated predictor achieved performance comparable to the best individual kernel, confirming the robustness of the aggregation strategy against the risk of selecting an inadequate hypothesis space.

Significance and Claims
The paper claims to provide the first systematic treatment of vector-valued covariate-shift regression, effectively bridging theoretical development with practical application in high-dimensional settings. By establishing optimal convergence rates under general source conditions, the work offers a rigorous theoretical foundation for regularized learning in this domain. Furthermore, the proposed aggregation approach addresses a critical practical bottleneck—the selection of tuning parameters and kernels—making the method more viable for real-world applications where prior knowledge of optimal hyperparameters is unavailable. The authors position their work as a significant step forward in applying machine learning to functional data problems where input distributions shift due to environmental or operational factors.