Limits of spectral learning under noise

Technical Summary: Limits of Spectral Learning under Noise

Problem Statement
Learning functional relationships from noisy data is a central challenge in scientific inference. While spectral methods—approximating unknown functions via expansions in a chosen basis (e.g., Fourier, Legendre)—offer interpretable decompositions into modes of increasing complexity, their stability under additive label noise remains poorly understood. Specifically, it is unclear how noise perturbs the geometry of the coefficient space and degrades the inferred spectral decomposition. Existing literature has highlighted that noise constrains the recovery of analytic structure, but the specific mechanisms by which noise induces instability in spectral representations require further elucidation.

Methodology
The authors investigate supervised regression with additive label noise, where observed labels are given by $y_i = f(x_i) + \epsilon_i$, with $\epsilon_i \sim \mathcal{N}(0, \sigma^2)$. The target function $f(x)$ is approximated using a sparse spectral expansion:
$$f_\sigma(x) = a(\sigma) + \sum_{k=1}^p c_k(\sigma) \phi_k(x)$$
where $\{ \phi_k(x) \}$ are basis functions and $c(\sigma)$ represents the spectral coefficients.

The core methodological approach involves:

1. Empirical Feature Geometry: The authors analyze the design matrix $\Phi$ formed by sampled basis functions. They define the empirical Gram matrix $G_c = \Phi_c^\top \Phi_c / N$ (where $\Phi_c$ contains centered features), which dictates the natural geometry of the coefficient space.
2. Whitening Transformation: To restore an approximately Euclidean geometry and decouple coordinates, the authors apply a whitening transformation using $A = (G_c + \rho I)^{-1/2}$. This transforms the features to $\tilde{\Phi} = \Phi_c A$ and coefficients to $\tilde{c} = A \tilde{c}$, such that the empirical Gram matrix in the new coordinates approximates the identity matrix ($\tilde{G} \approx I$).
3. Perturbation Analysis: In these whitened coordinates, the noise-induced deviation $\delta \tilde{c}$ is treated as a stochastic drift. The authors derive that the expected squared norm of this drift scales with the effective number of active spectral modes ($p_{\text{eff}}$) and the noise amplitude $\sigma$.
4. Universal Scaling: By comparing the perturbation magnitude to the signal norm, they define an intrinsic noise scale $\sigma^*$ and derive a closed-form expression for the overlap (cosine similarity) between noisy and noiseless coefficient vectors.

Key Contributions and Theoretical Results

• Intrinsic Noise Scale ($\sigma^*$): The paper derives a characteristic noise scale $\sigma^* = \|\tilde{c}_0\| \sqrt{N/p_{\text{eff}}}$, where $\|\tilde{c}_0\|$ is the norm of the noiseless coefficient vector, $N$ is the sample size, and $p_{\text{eff}}$ is the number of significantly excited directions. This scale governs the transition between stable learning and noise-dominated regimes.
• Universal Degradation Curve: The authors provide a parameter-free analytic prediction for the overlap $q(\sigma)$ between noisy and noiseless coefficient vectors:
  $$q(\sigma) \approx \left( 1 + \left( \frac{\sigma}{\sigma^*} \right)^2 \right)^{-1/2}$$
  This expression predicts a universal degradation curve where the overlap depends solely on the ratio of noise amplitude to the intrinsic noise scale.
• Coefficient Drift: The normalized distance between noisy and noiseless coefficients is shown to scale linearly with $\sigma/\sigma^*$, confirming that noise induces a predictable drift in the coefficient space.

Experimental Results
Numerical experiments were conducted across multiple dimensions (1D and 2D) and various orthogonal bases, including Fourier, Legendre, Bessel, Haar, Jacobi, and Chebyshev.

• Spectral Broadening: As noise increases, the learned spectral coefficients deviate from the expected power-law decay. The spectral weight spreads across modes, leading to a sharp increase in the normalized spectral entropy, moving from a sparse distribution toward a uniform distribution.
• Data Collapse: When empirical results for overlap and coefficient distance are plotted against the raw noise amplitude $\sigma$, they vary across different bases and dimensions. However, when plotted against the normalized scale $\sigma/\sigma^*$, all curves collapse onto the single theoretical prediction given by Eq. (6).
• RMSE Behavior: The test Root-Mean-Square Error (RMSE) follows the same scaling. At low noise, RMSE is dominated by baseline approximation errors (bias). Once $\sigma \gtrsim \sigma^*$, the noise-induced term dominates, and RMSE grows consistently with the predicted drift.

Significance and Claims
The paper claims to identify a fundamental limit on recovering functional structure from noisy data using spectral methods. The primary significance lies in demonstrating that spectral learning exhibits a sharp noise threshold ($\sigma^*$) beyond which coefficient estimates become unstable.

• Geometric Interpretation: The degradation is interpreted as a stochastic drift in a whitened coefficient space, governed by the interplay between the signal norm, sample size, and the effective dimensionality of the model ($p_{\text{eff}}$).
• Comparison to Symbolic Regression: The authors contrast their findings with symbolic regression. While symbolic regression (which seeks closed-form models) may exhibit abrupt transitions due to the gapped topology of its representation space (analogous to a Cantor set), spectral expansions with orthogonal bases exhibit continuous degradation.
• General Trade-off: The results highlight a universal trade-off between intrinsic signal scale, sample size, and model complexity. Specifically, the tolerable noise level increases with $\sqrt{N}$ but decreases with the square root of the effective model dimension.

The work concludes that these scaling laws apply broadly to spectral learning methods based on orthogonal function expansions, providing a theoretical framework to predict when noise will fundamentally compromise the recovery of spectral structure.