Tube Loss: A Novel Approach for Prediction Interval Estimation

Technical Summary: Tube Loss for Prediction Interval Estimation

Problem Statement
In regression tasks, providing a point prediction without quantifying uncertainty is often insufficient for high-stakes decision-making. While Prediction Intervals (PIs) offer a range $[\hat{\mu}_1(x), \hat{\mu}_2(x)]$ with a specified confidence level $1-\alpha$, existing methods face significant limitations. Traditional distribution-based approaches (e.g., Mean-Variance Estimation) often fail under skewed or heavy-tailed noise. Distribution-free methods like Lower Upper Bound Estimation (LUBE) rely on non-differentiable loss functions involving step functions, preventing the use of standard gradient descent (GD). Other approaches, such as Simultaneous Quantile Regression (SQR), require solving complex optimization problems for multiple quantiles, while Relaxed Quantile Regression (RQR) suffers from high sensitivity to outliers and potential crossing of interval bounds. Furthermore, many existing methods struggle to produce tight intervals when the conditional response distribution is asymmetric.

Methodology: The Tube Loss Function
The paper introduces Tube Loss, a novel loss function designed for the simultaneous estimation of lower and upper PI bounds via a single optimization problem.

• Definition: Let $u_1 = y - \mu_1(x)$ and $u_2 = y - \mu_2(x)$ be the prediction errors for the lower and upper bounds, respectively, with the constraint $\mu_2 \ge \mu_1$ (implying $u_2 \le u_1$). For a target coverage $1-\alpha$ and a tunable parameter $r \in (0, 1)$, the Tube Loss $L^{(1-\alpha)}_r(u_1, u_2)$ is defined piecewise:

  • $(1-\alpha)u_2$ if $u_2 > 0$ (observation above the upper bound).
  • $-\alpha u_2$ if $u_2 \le 0, u_1 \ge 0$, and $ru_2 + (1-r)u_1 \ge 0$ (observation inside the "upper" part of the tube).
  • $\alpha u_1$ if $u_2 \le 0, u_1 \ge 0$, and $ru_2 + (1-r)u_1 < 0$ (observation inside the "lower" part of the tube).
  • $-(1-\alpha)u_1$ if $u_1 < 0$ (observation below the lower bound).

• Key Mechanisms:

  1. Gradient Descent Compatibility: Unlike LUBE or Quality-Driven (QD) losses which involve non-differentiable step functions or require sigmoid approximations, Tube Loss is a linear function of errors (almost everywhere differentiable), enabling direct optimization via standard GD.
  2. Shiftability via Parameter $r$: The parameter $r$ controls the "midpoint" of the tube relative to the data density. When $r=0.5$, the tube is centered (optimal for symmetric distributions). By tuning $r$, the interval can be shifted up or down along the response distribution's support. This is particularly effective for skewed distributions, allowing the interval to capture high-density regions and reduce the Mean Prediction Interval Width (MPIW) without sacrificing coverage.
  3. Recalibration via Parameter $\delta$: If the empirical coverage (PICP) on a validation set exceeds the nominal level $1-\alpha$, a linear penalty term $\delta|\mu_2(x) - \mu_1(x)|$ can be added to the loss. This allows for further narrowing of the interval (reducing MPIW) while maintaining the desired coverage, a process less sensitive to outliers than quadratic penalties used in other methods.

Theoretical Properties
The paper provides a theoretical analysis (Lemma 1 and Proposition 1) demonstrating that as the sample size $n \to \infty$, the empirical coverage of the intervals derived from Tube Loss asymptotically attains the nominal confidence level $1-\alpha$. The analysis partitions the feature space into four regions based on the loss function's conditions, showing that the ratio of data points falling outside the interval to those inside converges to $\alpha / (1-\alpha)$.

Experimental Results
The authors evaluated Tube Loss across various frameworks and datasets:

1. Kernel Machines: On synthetic datasets with symmetric and positively skewed noise, Tube Loss with tuned $r$ significantly reduced MPIW compared to centered intervals and standard quantile regression, while maintaining target coverage. It also demonstrated faster training times compared to quantile-based kernel machines.
2. Neural Networks (NN):
   • vs. QD Loss: Tube Loss produced smoother and more compact PIs with lower Sum of Mean Squared Errors (SMSE) compared to QD loss, avoiding the sensitivity to the softening parameter required by QD.
   • vs. RQR: In the presence of outliers, Tube Loss maintained stable bounds, whereas RQR exhibited bound crossings and degraded coverage.
   • Benchmarking: Across 12 benchmark datasets, Tube Loss-based NNs achieved superior or comparable performance (in terms of PICP and MPIW) to RQR, QRNN, SQR, and QD baselines in 9 out of 12 cases.
3. Deep Probabilistic Forecasting: Applied to LSTM networks on time-series datasets (e.g., Electric, Sunspots, Wind Speed), the Tube Loss-based LSTM (T-LSTM) outperformed Quantile-based LSTM (Q-LSTM) in 5 of 6 datasets. Crucially, T-LSTM required only a single training cycle, whereas Q-LSTM requires training two separate models, resulting in training time reductions of 33% to 63%.
4. Conformal Prediction: When integrated into a Conformalized Quantile Regression (CQR) framework (replacing quantile bounds with Tube bounds), the resulting Tube-based Conformal Regression (TCR) achieved comparable coverage and width to CQR but with significantly reduced computational runtime.
5. Semantic Textual Similarity (STS): In the STS task, an ensemble of Tube Loss NNs outperformed existing uncertainty quantification baselines (including QRGB, QRNN, HTS, and MDN) in generating high-quality PIs.

Significance and Claims
The paper claims that Tube Loss offers a robust, efficient, and flexible alternative for uncertainty quantification in regression. Its primary significance lies in:

• Simultaneous Optimization: Solving for both bounds in a single, differentiable optimization problem, unlike methods requiring separate quantile estimations or non-gradient-based solvers.
• Adaptability to Skewness: The ability to shift intervals via $r$ to align with asymmetric data distributions, yielding tighter intervals where traditional centered methods fail.
• Robustness: Reduced sensitivity to outliers compared to RQR and elimination of the need for step-function approximations found in QD/LUBE.
• Efficiency: Significant reductions in training time for deep learning applications compared to dual-model quantile approaches.

The authors conclude that Tube Loss is a superior method for generating high-quality prediction intervals across kernel machines, neural networks, and deep probabilistic forecasting frameworks, particularly in scenarios involving skewed noise or computational constraints.