Technical Summary: Out-of-Distribution Generalization of Quantile Regression with Heavy Tailed Inputs: An SVM Approach

Problem Statement

The paper addresses the challenge of extrapolation in supervised regression, specifically when the goal is to predict a response variable $Y$ given covariates $X$ that take unusually large values (i.e., $\|X\| \ge t$ for large $t$). Standard machine learning algorithms, which typically minimize empirical risk over the entire distribution, often fail in these regimes because they prioritize the "bulk" of the data distribution, neglecting the tail regions where extreme events occur.

While Extreme Value Theory (EVT) provides a framework for modeling such extremes, existing statistical learning approaches for extrapolation face significant limitations:

1. Boundedness Assumptions: Many theoretical guarantees require the response variable $Y$ to be bounded, which is unrealistic in extreme value analysis where responses are often heavy-tailed and unbounded.
2. Complexity Constraints: Previous non-asymptotic guarantees for extrapolation (e.g., Clémencçon et al., 2025) rely on the predictor class having a finite Vapnik–Chervonenkis (VC) dimension. This restricts the hypothesis space to linear or low-complexity models, making them ill-suited for high-dimensional or nonlinear settings.
3. Transformation Issues: Attempts to enforce boundedness via nonlinear transformations of the target variable often interact poorly with conditional expectations, limiting the applicability of theoretical guarantees to the original untransformed variable.

The specific objective of this work is to develop a method for extreme quantile regression that accommodates unbounded response variables and high-dimensional, nonlinear settings while providing rigorous finite-sample learning guarantees.

Methodology

Theoretical Framework

The authors operate under regular variation assumptions on the joint distribution of $(X, Y)$. Specifically, they assume the pair is "first-component regularly varying," meaning that as the norm of the covariate $\|X\|$ grows, the distribution of the scaled pair $(X/t, Y)$ converges to a limiting distribution $(X_\infty, Y_\infty)$.

Key theoretical insights include:

• Angular Decomposition: In the extreme regime, the behavior of the conditional quantile function depends primarily on the angular component $\theta(X) = X/\|X\|$ rather than the radial component $\|X\|$. The extreme conditional quantile $q_{P_\infty}$ can be expressed as $h_{P_\infty} \circ \theta$, where $h_{P_\infty}$ is a continuous function on the unit sphere $S$.
• Asymptotic Risk: Instead of minimizing the standard risk $R(f)$, the method targets the minimization of an asymptotic conditional risk $R_\infty(f) = \limsup_{t \to \infty} E[\ell'_\tau(Y, f(X)) \mid \|X\| \ge t]$.
• Modified Pinball Loss: To handle unbounded $Y$, the authors utilize a modified pinball loss $\ell'_\tau(y, z) = \ell_\tau(y, z) - \ell_\tau(y, 0)$. This loss retains the property of eliciting the conditional quantile even when $Y$ is unbounded, provided the quantile function itself satisfies a mild integrability condition.

Proposed Algorithms

The paper proposes two algorithms for estimating the extreme quantile function:

1. Unpenalized Empirical Risk Minimization (ERM) (Algorithm 1):

   • Selects the top $k$ observations with the largest covariate norms.
   • Minimizes the empirical modified pinball risk over a function class $\mathcal{F}$ defined on the sphere $S$.
   • Limitation: Theoretical guarantees for this approach rely on the class $\mathcal{F}$ having finite VC dimension and being uniformly bounded, restricting it to linear or low-complexity models.

2. Support Vector Machine (SVM) Approach (Algorithm 2):

   • Formulates the problem as minimizing a penalized empirical risk within a Reproducing Kernel Hilbert Space (RKHS) $\mathcal{H}$.
   • The objective is: $\min_{h \in \mathcal{H}} \left( \frac{1}{k} \sum_{i=1}^k \ell'_\tau(Y_{(i)}, h(\theta(X_{(i)}))) + \lambda \|h\|_\mathcal{H}^2 \right)$.
   • Advantage: By leveraging the RKHS structure and regularization, this approach avoids the restrictive finite VC dimension assumption and does not require the target variable $Y$ to be bounded. It naturally handles high-dimensional and nonlinear settings.

Key Contributions

1. Novel SVM Framework for Extreme Quantile Regression: The paper introduces an SVM-based method tailored for extrapolation. Unlike previous works, it does not rely on VC-dimension bounds, allowing for rich, nonlinear predictor classes (e.g., Gaussian kernels) in high dimensions.
2. Relaxation of Boundedness Assumptions: The authors establish theoretical guarantees for unbounded response variables. By using the modified pinball loss and the specific structure of the asymptotic risk, they remove the need for artificial transformations or boundedness constraints on $Y$.
3. Finite-Sample Learning Guarantees:
   • For the ERM setting, they derive bias-variance decompositions and error bounds under VC-dimension assumptions, extending prior results to quantile regression.
   • For the SVM setting, they provide non-asymptotic, high-probability bounds on the excess asymptotic risk. These bounds are derived under mild regularity assumptions (regular variation and continuity of the extreme quantile) and explicitly account for the approximation error of the RKHS and the variance of the estimator.
4. Theoretical Unification: The work unifies ideas from statistical learning (SVMs, RKHS) and multivariate extreme value theory, demonstrating that the asymptotic conditional risk can be effectively minimized by learning an angular function.

Results and Empirical Validation

Theoretical Results

• Error Decomposition: The excess asymptotic risk for the SVM estimator is decomposed into a threshold bias term, an asymptotic approximation error (dependent on the RKHS), and a variance term.
• Convergence Rates: Under standard assumptions on the approximation error (e.g., if the true function lies in the RKHS or an interpolation space), the authors show that the error converges at a rate of $O(k^{-\beta/(\beta+1)})$, where $k$ is the number of extreme observations used for training.
• Stability: The analysis utilizes the stability of SVM solutions to control the deviation between the empirical minimizer and the theoretical minimizer, even in the presence of heavy-tailed data.

Empirical Study

The methodology is evaluated on a real-world dataset of river flow measurements from the Danube basin (31 gauging stations).

• Task: Predicting the conditional median of river flow at a target station given extreme flow measurements at other stations.
• Preprocessing: Data undergoes marginal standardization to unit-Pareto margins to satisfy the regular variation assumptions.
• Comparison: The proposed SVM approach (Algorithm 2) is compared against:
  • An unpenalized linear ERM (Algorithm 1).
  • Naive baselines using standard mean-deviation standardization.
• Findings:
  • The SVM method outperforms the linear ERM, particularly in high-dimensional settings where the "curse of dimensionality" affects the unpenalized approach.
  • The proposed angular-based approach provides accurate predictions in the extreme regime, whereas naive baselines fail to capture the dependence structure of extreme events.
  • The results demonstrate a clear bias-variance trade-off regarding the number of extreme observations ($k$) used for training, with optimal performance found when $k/n$ is between 0.05 and 0.1.

Significance and Claims

The paper claims to provide a tractable and theoretically grounded approach to extrapolation that bridges the gap between statistical learning theory and extreme value analysis. Its primary significance lies in:

• Practical Relevance: Offering a solution for predicting extreme events (e.g., floods) where data is scarce and covariates are heavy-tailed, a critical need in fields like climate science.
• Theoretical Advancement: Overcoming the historical trade-off between model flexibility (nonlinearity, high dimension) and theoretical guarantees in extrapolation tasks. By removing the boundedness and finite VC-dimension constraints, the method extends the applicability of rigorous learning guarantees to a broader class of real-world problems involving unbounded, heavy-tailed variables.

The authors maintain a modest stance, acknowledging that while their theoretical framework relies on regular variation and specific stability conditions, the empirical robustness of the method on the Danube dataset suggests its utility even when strict theoretical conditions (like the existence of a limiting density) might be challenging to verify in practice. They defer a rigorous treatment of standardization errors and more complex dependency structures to future research.