Technical Summary: Two-stage Odd Residual Flows for Mean-Preserving Probabilistic Time Series Forecasting

Problem Statement

Probabilistic time series forecasting (TSF) is critical for risk-sensitive decision-making in domains such as energy management and finance. However, existing approaches face a fundamental trade-off between distributional flexibility (the ability to model complex, non-Gaussian uncertainties) and mean prediction accuracy.

1. Parametric Methods (e.g., Mean Variance Estimation - MVE): These methods assume a fixed distributional family (typically heteroscedastic Gaussian) and optimize parameters via Negative Log-Likelihood (NLL). While they provide an analytical mean, joint NLL training creates an inherent conflict between learning the mean and the variance. As shown in prior work (Seitzer et al., 2022), the NLL gradient is scaled by the inverse variance ($1/\sigma^2$), causing high-variance regions to contribute less to mean learning. This often degrades point prediction accuracy, especially in long-horizon settings.
2. Flexible Generative Models (e.g., Diffusion, Normalizing Flows, VAEs): These models sidestep rigid distributional assumptions and can capture complex densities. However, they typically optimize distributional objectives (e.g., ELBO or log-likelihood) rather than mean-focused losses. Consequently, the predictive mean is an implicit by-product of fitting the full density and is rarely directly supervised. Furthermore, recovering the mean often requires expensive Monte Carlo sampling, which introduces estimation errors that worsen in high dimensions. Empirical results indicate that state-of-the-art probabilistic models (e.g., K2VAE) consistently trail deterministic point-predictors (e.g., SimpleTM) in mean accuracy.

Methodology: Two-stage Odd Residual Flows (TORF)

To decouple mean forecasting from uncertainty estimation, the authors propose TORF, a two-stage framework that guarantees mean preservation while modeling flexible residual distributions.

Stage 1: Deterministic Mean Estimation

A pre-trained, off-the-shelf point-prediction model (e.g., SimpleTM) is used to generate the conditional mean estimate $\hat{Y} = f_{\theta_1}(X)$. This stage is trained independently under a Mean Squared Error (MSE) objective. The model parameters $\theta_1$ are frozen after training, ensuring that the mean prediction is not distorted by subsequent uncertainty modeling.

Stage 2: Restricted Normalizing Flow for Residuals

The second stage models the residual distribution $\epsilon = Y - \hat{Y}$ using a Restricted Normalizing Flow (RNF).

• Odd Function Constraint: The core innovation is restricting the flow architecture to strictly odd functions ($f(-u) = -f(u)$). Based on theoretical guarantees (Kobayashi and Aotani, 2023), if the base distribution is symmetric (e.g., standard normal) and the transformation is odd, the expected value of the transformed variable is analytically zero ($E[\epsilon] = 0$). This mathematically guarantees that the predictive mean of the full model remains exactly $\hat{Y}$ without requiring sampling.
• Residual Odd Splines (ROSS): The authors implement this using Rational Quadratic Neural Spline Flows (RQ-NSF) constrained to be odd. Instead of mirroring bin parameters across positive and negative domains, the spline processes the magnitude of the input and restores the sign, ensuring $f(0)=0$ and exact odd symmetry.
• Linear Scaling Layer (LSL): A conditional scaling layer is prepended to adapt the input range to the spline's support. Crucially, this layer uses only scaling (no translation) to preserve the odd symmetry property.
• Architecture: The flow consists of $K$ blocks, each containing an LSL followed by a ROSS layer. A lightweight 1D Convolutional Network (SplineNet) maps the context embedding (derived from the frozen Stage 1 encoder) to the spline parameters, ensuring efficiency for long horizons.

Theoretical Guarantees

By construction, the composition of odd functions (LSL and ROSS) remains odd. When applied to a symmetric base distribution, the resulting residual density is symmetric about zero. Lemma 1 in the paper proves that this ensures the analytical mean and median of the predictive distribution are exactly the Stage 1 forecast, eliminating the need for Monte Carlo sampling to estimate the mean.

Key Contributions

1. Decoupled Framework: TORF isolates mean prediction from density estimation, allowing the use of any high-performing point predictor (Stage 1) without compromising its accuracy.
2. Odd-Constrained Residual Flow: The introduction of ROSS (Residual Odd Splines) enables flexible, non-Gaussian density estimation while provably preserving the exact analytical mean of the first stage, avoiding the sampling errors associated with standard Normalizing Flows.
3. Efficient Architecture: A lightweight CNN-based SplineNet efficiently maps time-series contexts to flow parameters, making the approach scalable to long horizons and high-dimensional multivariate settings.
4. State-of-the-Art Performance: The framework sets new benchmarks in both deterministic accuracy (NMAE) and probabilistic density estimation (CRPS) across diverse datasets.

Experimental Results

The authors evaluated TORF on 17 real-world datasets (8 short-term, 9 long-term) against 12 baselines, including point forecasters (PatchTST, iTransformer) and probabilistic models (K2VAE, TimeGrad, CSDI, GRU-NVP).

• Long-Horizon Forecasting: TORF achieved state-of-the-art results, winning on 9/9 datasets for NMAE and 7/9 datasets for CRPS. Notable improvements included a +35.4% gain in CRPS (ETTh2-L) and +25.1% in NMAE (ETTm2-L) over the strongest baseline (K2VAE).
• Short-Horizon Forecasting: TORF won on 5/8 datasets for NMAE and 6/8 datasets for CRPS.
• Ablation Studies:
  • The two-stage decomposition itself (MVE-2S, a Gaussian variant) already outperformed K2VAE on 11/12 distributional metrics, confirming that decoupling is the primary driver of mean accuracy gains.
  • The odd-constrained flow (TORF) further improved CRPS over the Gaussian baseline by modeling non-Gaussian residuals.
  • Removing the Linear Scaling Layer (LSL) significantly degraded performance, highlighting the need for adaptive input scaling.
  • Replacing the CNN-based SplineNet with an MLP caused memory outages (OOM) on long horizons, demonstrating the efficiency of the proposed architecture.
• Backbone Agnosticism: Experiments substituting SimpleTM with iTransformer as the Stage 1 model showed that TORF's flow hyperparameters transfer effectively, with probabilistic performance tracking the quality of the underlying point predictor.

Significance and Claims

The paper claims that TORF resolves the long-standing trade-off between distributional flexibility and mean accuracy in probabilistic time series forecasting. By strictly enforcing odd symmetry in the residual flow, the method guarantees that the predictive mean is analytically tractable and identical to the high-accuracy Stage 1 forecast, without the computational cost or estimation error of Monte Carlo sampling.

The authors position TORF as a new state-of-the-art framework that:

1. Achieves superior point prediction accuracy (NMAE) compared to existing probabilistic models, which often suffer from degraded means due to joint NLL training or sampling noise.
2. Provides strong density estimation (CRPS) by capturing complex, non-Gaussian residual structures that Gaussian baselines miss.
3. Offers a computationally efficient solution for long-horizon forecasting, avoiding the memory and time costs associated with sampling-based inference in diffusion or standard flow models.

The work suggests that for many time series tasks, the "faithfulness" of the mean (guaranteed by the two-stage odd flow) is a critical, previously underutilized property that leads to superior overall performance.