Infinite-Dimensional Spherical Kernel ridge Regression

Technical Summary: Infinite-Dimensional Spherical Kernel Ridge Regression

Problem Statement
The paper addresses the regression problem where the response variable $Y$ lies on a sphere $S$ of arbitrary dimension (finite or infinite) within a Hilbert space $Y$, and the covariate $X$ resides in a Polish space. This setting arises in functional data analysis, statistical shape analysis, and the modeling of probability density functions (via square-root transformations).

Existing approaches for manifold-valued regression face significant limitations:

1. Tangent-space regression: Lifts responses to a tangent space $T_o S$ to apply linear methods. While computationally convenient, this violates intrinsic spherical distances, introducing curvature-induced bias because the conditional mean of the log-transformed data ($E[\text{Log}_o(Y)|X]$) differs from the conditional Fréchet mean ($\mu(x) = \arg\min_q E[d_S^2(q, Y)|X=x]$).
2. Geodesic regression: Typically restricted to finite-dimensional manifolds and often assumes parametric forms or specific predictor structures.
3. Infinite-dimensional challenges: Extending these methods to infinite-dimensional spheres (e.g., for density functions) is theoretically difficult because estimators lack closed-form solutions, and standard empirical process techniques often implicitly assume finite dimensionality.

Methodology
The authors propose an intrinsic regression framework that models the conditional Fréchet mean $\mu(x)$ directly on the sphere without lifting the raw responses to a tangent space for the loss calculation.

1. Model Formulation:
   The model decomposes the conditional mean into a base point (intercept) $o \in S$ and a linear predictor function $f: X \to T_o S$. The conditional mean is reconstructed via the exponential map:
   $$\mu(x) = \text{Exp}_o(f(x))$$
   The function $f$ is assumed to belong to a Vector-Valued Reproducing Kernel Hilbert Space (VVRKHS) $\mathcal{H}$ of functions mapping $X$ to the tangent space $T_o S$. The authors utilize Simple Multiplicative Operator (SMO) kernels of the form $K(x, x') = k(x, x')\text{Id}$, where $k$ is a scalar kernel and $\text{Id}$ is the identity on $T_o S$.

2. Estimation:
   The estimator $\hat{f}$ is obtained by minimizing a regularized empirical risk:
   $$R_n(f, \lambda) = \frac{1}{n} \sum_{i=1}^n d_S^2(\text{Exp}_o(f(X_i)), Y_i) + \lambda^2 \|f\|_{\mathcal{H}}^2$$
   Unlike standard kernel ridge regression, the loss term $d_S^2$ is non-convex over the entire Hilbert space due to the periodicity of the exponential map.

3. Optimization Algorithm:
   Since no closed-form solution exists, the authors develop a BFGS-based optimization algorithm combined with a dual low-rank approximation.

   • Representer Theorem: A novel representer theorem (Theorem 4.1) proves that the minimizer lies in a finite-dimensional subspace spanned by the kernel functions evaluated at the data points and the log-transformed responses.
   • Efficiency: To handle high dimensionality, the algorithm projects the problem onto a low-rank subspace derived from the leading eigenvectors of the kernel Gram matrix and the response Gram matrix. This reduces the parameter space from $O(n^2)$ to a manageable size while maintaining numerical stability.

Key Theoretical Results
The paper establishes rigorous theoretical guarantees under specific regularity assumptions, primarily a bounded support condition (Assumption 4.6): $d_S(o, Y) \leq \pi/4$ almost surely. This ensures the conditional mean lies within a geodesic ball where the squared distance function is locally convex.

1. Representer Theorem (Theorem 4.1): Proves that the infinite-dimensional optimization problem reduces to a finite-dimensional one, even when the sphere and the function space are infinite-dimensional.
2. Convergence Rates:
   • Minimal Assumptions: For arbitrary Polish covariate spaces and infinite-dimensional spheres, the estimator achieves a convergence rate of $O_P(n^{-1/2})$ in $L^2(P_X)$ norm (Theorem 4.8).
   • Smoothness-Dependent Rates: When the scalar kernel $k$ has specific eigenvalue decay properties (finite-rank, polynomial, or stretched exponential), the authors derive faster rates (e.g., $O_P(n^{-p/(p+1)})$ for polynomial decay) (Theorem 4.13). These rates match or improve upon known results for linear vector-valued kernel ridge regression and functional regression.
3. Strong Consistency: Under compactness and continuity assumptions, the estimator is shown to be outer almost surely consistent (Theorem 4.17).

Empirical Evaluation

1. Simulation Study: The authors simulate data on an infinite-dimensional sphere ($L^2([0,1])$) with varying noise levels.

   • Results: The proposed method outperforms tangent-space regression, particularly as sample size increases and data spread widens on the sphere. The method demonstrates robust performance even when the theoretical bounded support assumption is violated (diameter $4\pi/5$).
   • Rates: Observed convergence rates align with theoretical predictions (e.g., $O(\log(n)^2/n)$ for Gaussian kernels).

2. Application: Urban Pollution Density Regression:

   • Data: Aerosol particle light absorption coefficients (AAC) from 14 monitoring stations in the Netherlands. The response is the square-root density of AAC measurements.
   • Comparison: The method is compared against Fréchet regression (2-Wasserstein space), Bayes Hilbert regression, and a lognormal-normal convolution model.
   • Performance: The spherical kernel ridge regression method achieves competitive prediction errors in spherical distance and Bayes-space distance. Crucially, it is orders of magnitude faster than the Bayes Hilbert and lognormal-normal models, offering superior computational efficiency and stability.

Significance and Contributions
The paper claims several specific contributions to the literature:

1. Bridging Communities: It introduces VVRKHS regression methods to the functional data analysis community for non-linear response spaces.
2. Theoretical Novelty: It provides a new representer theorem that reduces infinite-dimensional spherical regression to finite-dimensional optimization, a result not previously established for this specific non-linear setting.
3. Intrinsic Geometry: By avoiding the tangent-space lift for the loss function, the method avoids the geometric bias inherent in linearizing curved spaces, aligning with the intrinsic geometry of the sphere.
4. Practical Utility: It provides a computationally efficient algorithm (via BFGS and low-rank approximation) and a ready-to-use R package (sphereg), making infinite-dimensional spherical regression feasible for real-world applications like density estimation.

The authors remain modest regarding the bounded support assumption, noting that while restrictive, it is comparable to assumptions in existing metric space regression literature. They suggest that the assumption can be relaxed via numerical arguments or a "patchwork" approach (partitioning the covariate space) for future work.