Functional Liu Regression for Scalar-on-Functional Models in High-Dimensional Settings

Technical Summary: Functional Liu Regression for Scalar-on-Functional Models in High-Dimensional Settings

Problem Statement
Scalar-on-function regression, a core component of Functional Data Analysis (FDA), models a scalar response dependent on one or more functional predictors defined over a continuum (e.g., time or space). While flexible, this approach faces significant challenges when predictors are discretized or expanded into basis functions. The smooth structure of functional data induces strong multicollinearity among the resulting regressors, leading to ill-conditioned or singular design matrices. In high-dimensional settings where the number of basis coefficients ($m$) approaches or exceeds the sample size ($n$), Ordinary Least Squares (OLS) estimates become unstable with inflated variance. While ridge regression and roughness penalties (e.g., Tikhonov regularization) offer stabilization, they apply uniform shrinkage or specific smoothness constraints that may not optimally balance bias and variance in all directions. Furthermore, standard tuning criteria like Generalized Cross-Validation (GCV) and Leave-One-Out Cross-Validation (LOO-CV) have been observed to fail in underdetermined regimes, yet the theoretical reasons for this failure in Liu-type contexts remain under-explored.

Methodology
The paper proposes a Functional Liu-type shrinkage estimator (fLiu) designed to address multicollinearity and instability in scalar-on-function regression. The methodology proceeds as follows:

1. Model Formulation: The scalar-on-function model is approximated via a truncated basis expansion (e.g., B-splines or Fourier basis), transforming the infinite-dimensional problem into a finite-dimensional linear model $y = Zb + \epsilon$, where $Z$ is the design matrix and $b$ represents the basis coefficients.
2. Hybrid Penalty Structure: To handle both singularity and smoothness, the authors define a hybrid penalty matrix $Q_{\lambda, \alpha} = \lambda(\alpha I + (1-\alpha)R)$, where $R$ is a roughness penalty matrix (penalizing curvature) and $I$ is the identity matrix. The parameter $\lambda$ controls overall regularization magnitude, while $\alpha$ balances isotropic ridge shrinkage against basis-dependent roughness.
3. The fLiu Estimator: The proposed estimator minimizes a centered penalized criterion:
   $$J_{\lambda, d, \alpha}(b) = \|y - Zb\|^2 + (b - d\hat{b}_{LS})^\top Q (b - d\hat{b}_{LS})$$
   This yields the closed-form solution:
   $$\hat{b}_{fLiu} = (S + Q)^{-1}(Z^\top y + dQ\hat{b}_{LS})$$
   where $S = Z^\top Z$, $\hat{b}_{LS}$ is the OLS estimator, and $d$ is the Liu-type biasing parameter.
   • Role of Parameters: $\lambda$ and $\alpha$ govern the geometry and magnitude of the penalty (stability and smoothness), while $d$ controls the bias-variance trade-off relative to OLS. When $d=1$, the estimator reduces to OLS; when $d=0$, it reduces to the generalized ridge estimator.
4. Parameter Selection:
   • Overdetermined Case ($n \ge m$): The authors employ continuous optimization (using SLSQP and automatic differentiation) to minimize GCV or PRESS/LOO-CV criteria over the three parameters $(\lambda, d, \alpha)$.
   • Underdetermined Case ($n < m$): The paper derives a theoretical result showing that GCV and PRESS become constant with respect to $d$ in this regime, rendering them uninformative for tuning $d$. To address this, a plug-in tuning rule is proposed, deriving an explicit optimal $d$ by minimizing the Mean Squared Error (MSE) decomposition using empirical estimates of bias and variance.

Key Contributions

• Theoretical Extension: The study extends the classical Liu estimator (Liu 1993) to the functional setting, integrating directional shrinkage with smoothness regularization.
• Risk Analysis: An explicit MSE decomposition is derived, proving that the risk of the fLiu estimator is a convex quadratic function of the parameter $d$. This establishes the existence of a unique optimal $d$ that dominates OLS in terms of MSE under specific conditions.
• High-Dimensional Insight: The paper provides a theoretical explanation for the failure of GCV and LOO-CV in underdetermined functional settings. It proves that for full row-rank design matrices, these criteria are independent of the Liu parameter $d$, explaining why existing Liu-type literature has predominantly focused on overdetermined regimes.
• Practical Implementation: The authors developed the fLiu.py Python package (and utilized R's fda package) to implement the estimator, utilizing automatic differentiation to efficiently solve the non-convex optimization problem for parameter selection.

Empirical Results
The methodology was evaluated using the Canadian weather dataset (35 stations, predicting annual precipitation from temperature curves).

• Overdetermined Setting ($K=11$ basis functions): The fLiu estimator achieved the lowest Generalized Cross-Validation (GCV) value and the lowest test set prediction loss (0.0203) compared to OLS, Ridge, classical Liu, and Generalized Ridge. It demonstrated a favorable bias-variance trade-off, producing stable coefficient estimates that captured seasonal patterns without overfitting.
• Underdetermined Setting ($K=35$ basis functions): In this high-dimensional regime, the GCV curve was observed to be flat with respect to $d$, confirming the theoretical degeneracy. The plug-in tuning rule successfully provided a data-driven choice for $d$, outperforming arbitrary choices (like $d=0$) and demonstrating the necessity of alternative tuning strategies when $n < m$.

Significance and Claims
The paper claims that the fLiu estimator offers a principled, flexible, and stable alternative to existing penalized functional regression methods. By decoupling the enforcement of smoothness/stability ($\lambda, \alpha$) from the bias-variance adjustment ($d$), the method provides additional control over the estimation process.

The authors modestly assert that the study:

1. Clarifies the theoretical behavior of Liu-type estimators in functional settings, specifically regarding risk decomposition and parameter selection.
2. Identifies and explains the limitations of standard cross-validation criteria in high-dimensional functional regression, motivating the use of plug-in rules in such scenarios.
3. Demonstrates through empirical analysis that the proposed method yields competitive predictive accuracy and interpretable coefficient functions, particularly in the presence of strong multicollinearity.

The paper concludes that while the fLiu estimator is a significant addition to the class of penalized functional regression methods, future work could explore extensions to function-on-function regression, theoretical analysis of the plug-in estimator's properties, and the incorporation of sparsity for multiple correlated predictors.