Technical Summary: Expectations vs. Realities: The Cost of MSE-Optimal Forecasting Under Conditional Uncertainty

1. Problem Statement

Multi-step time series forecasting (MSF) is traditionally evaluated using point-wise error metrics, most notably Mean Squared Error (MSE). This evaluation paradigm implicitly treats the conditional expectation, $E[Y_h | X]$, as the sufficient and desired prediction target. However, the authors argue that this assumption becomes fragile as the forecast horizon increases. In many real-world systems, uncertainty accumulates over time, causing the conditional distribution of future outcomes to become increasingly dispersed.

Under conditions of high conditional uncertainty, the conditional expectation (the MSE-optimal target) may diverge significantly from typical realized outcomes. Consequently, models optimized strictly for MSE often produce forecasts that are "under-dispersed"—they are accurate in a point-wise sense but fail to capture the marginal variability of the true future distribution. The paper posits that this is not merely a failure of model capacity or optimization, but a structural limitation inherent to deterministic forecasting under squared loss when conditional uncertainty is non-zero.

2. Methodology and Theoretical Framework

2.1 Theoretical Formalization

The authors formalize the problem using the Law of Total Variance:
$$\text{Var}(Y_h) = \text{Var}(E[Y_h | X]) + E[\text{Var}(Y_h | X)]$$
They define a Conditional Uncertainty Gap ($\text{Gap}_h$) as the fraction of total variability attributable to irreducible uncertainty given the past:
$$\text{Gap}_h = \frac{E[\text{Var}(Y_h | X)]}{\text{Var}(E[Y_h | X]) + E[\text{Var}(Y_h | X)]}$$
When $\text{Gap}_h > 0$, the conditional mean explains less than 100% of the variance of the realized future.

2.2 The Accuracy–Realism Trade-off (Theorem 1)

The core theoretical contribution is Theorem 1, which proves that for any deterministic predictor $f_h$:

1. If $\text{Gap}_h > 0$, no predictor can simultaneously minimize MSE and match the marginal distribution of realized futures ($L(Y_h)$).
2. The unique MSE-minimizer is the conditional expectation $f^*_h(X) = E[Y_h | X]$.
3. Because $\text{Var}(f^*_h(X)) < \text{Var}(Y_h)$ whenever $\text{Gap}_h > 0$, the marginal distribution of the MSE-optimal predictor strictly differs from the true marginal distribution.
4. This creates an unavoidable Pareto frontier between pointwise accuracy (MSE) and marginal realism (measured by Wasserstein distance $W_1$).

2.3 Empirical Characterization

To quantify this trade-off, the authors employ:

• Controlled Synthetic Systems: A stochastic Mackey–Glass system where the noise level ($\sigma_s$) and horizon ($h$) can be tuned to control the conditional uncertainty gap.
• Real-World Benchmarks: Nine datasets spanning energy, traffic, weather, air quality, and finance (e.g., METR-LA, Electricity, ExchangeRate).
• Strategies Evaluated:
  • Direct (MIMO): Predicts the entire horizon in one step.
  • Recursive (REC): Iteratively predicts one step ahead.
  • Inference Rules: For probabilistic models, they compare mean-based inference (predictive mean) vs. sample-based inference (single-sample rollout).
• Metrics: MSE for accuracy and $W_1$ (Wasserstein-1 distance) for marginal realism. They also report trajectory-level metrics (DTW, Vector $W_1$, etc.) as robustness checks.

3. Key Results

3.1 Geometry of the Trade-off

• Low Uncertainty Regime: When the conditional uncertainty gap is small, the Pareto frontier is degenerate; methods cluster near a single operating point where minimizing MSE also yields high realism.
• High Uncertainty Regime: As the gap increases (longer horizons or higher noise), the attainable set of performances expands into a pronounced Pareto front. This separates MSE-optimal predictors (which are under-dispersed) from methods that sacrifice point accuracy to match marginal variability.
• Correlation: The length of the Pareto front increases monotonically with the estimated conditional uncertainty gap.

3.2 The Cost of Strict MSE Selection

• Opportunity Cost: The authors define a "successful trade" as a scenario where a small relaxation in MSE yields a larger relative gain in realism.
• Empirical Findings: Across synthetic and real-world benchmarks, relaxing the MSE constraint by just 5% frequently unlocks disproportionate gains in marginal realism.
  • Median Improvement: 17.3% improvement in marginal realism ($W_1$) across all datasets.
  • Success Rate: In 84.1% of cases (global average), a small MSE relaxation resulted in a net-positive realism trade.
  • Dataset Specifics: Some datasets (e.g., Weather, ExchangeRate) showed median realism improvements exceeding 22% with high success rates.

3.3 Strategy and Inference as Control Knobs

The study demonstrates that different forecasting strategies systematically occupy different regions of the Pareto frontier:

• MSE-Optimal Extreme: Dominated by Direct (MIMO) strategies and Mean-based inference. These align with estimating the conditional expectation and produce smoothed, under-dispersed forecasts.
• Realism-Optimal Extreme: Dominated by Recursive (REC) strategies and Sample-based inference. These propagate variability through time, better matching the marginal distribution of realized futures at the cost of higher pointwise error.
• Probabilistic Models: Even probabilistic models do not eliminate the trade-off; they simply navigate it. Choosing the predictive mean places the model near the MSE extreme, while sampling places it near the realism extreme.

4. Contributions

The paper makes three primary contributions:

1. Theoretical Explanation: It proves that over-smoothing in long-horizon forecasting is a structural property of deterministic prediction under conditional uncertainty, not a model misspecification error. No deterministic predictor can minimize MSE and match the marginal distribution simultaneously if the uncertainty gap is non-zero.
2. Empirical Quantification: It characterizes the accuracy–realism Pareto frontier, showing that strict MSE-based model selection incurs a substantial and avoidable cost in marginal realism. Small relaxations in MSE often yield large gains in realism.
3. Practical Protocol: It reframes strategy selection (Direct vs. Recursive) and inference choice (Mean vs. Sample) as mechanisms for navigating this multi-objective space, rather than simply optimizing a single metric.

5. Significance and Recommendations

The paper argues that the incompatibility between pointwise accuracy and distributional realism is an intrinsic property of multi-step forecasting under conditional uncertainty, independent of the model class.

Practical Recommendation:
The authors propose a tolerance-band selection rule for model selection:

1. Identify the MSE-optimal model on validation data.
2. Retain all candidates whose MSE lies within a small, application-dependent tolerance $\epsilon$ (e.g., 5%) of the optimum.
3. Among these "near-ties," select the model that maximizes marginal realism (using metrics like $W_1$) or, for probabilistic models, uses calibration and sharpness diagnostics.

Significance:
This work challenges the standard practice of ranking models solely by MSE, particularly for long-horizon forecasting. It suggests that single-metric evaluation can be structurally misleading, systematically discarding forecasts that are more representative of realized futures. By treating long-horizon forecasting as a constrained multi-objective selection problem, practitioners can avoid the "cost" of MSE-only optimization and select models that better reflect the uncertainty inherent in the data-generating process.