Technical Summary: Quantifying the Risk–Return Tradeoff in Forecasting

Problem Statement

Traditional macroeconomic forecast evaluation relies heavily on average loss metrics (e.g., RMSE, MAE) and tests of statistical significance (e.g., Diebold–Mariano). While these tools determine if a model reduces forecast errors on average, they fail to capture the reliability and risk profile of those gains. Institutional users (central banks, asset managers) often possess asymmetric preferences: they tolerate episodes of outperformance but cannot afford catastrophic underperformance. Two models may exhibit identical average accuracy yet possess vastly different patterns of gains and losses over time, leading to different operational risks. The paper argues that average accuracy is not synonymous with forecast reliability and that current literature lacks a unified framework to quantify the tradeoff between forecast improvement and the volatility or severity of forecast failures.

Methodology

The paper imports risk-adjusted performance measures from finance to treat forecast loss differentials as a "return series."

1. The Return Analogy

For a model $M$ and benchmark $B$, the "return" at time $t$ is defined as the loss differential:
$$r_t = L_t^B - L_t^M$$
where $L$ is a loss function (e.g., squared error). A positive $r_t$ indicates the model outperforms the benchmark. The sequence $\{r_t\}$ is analyzed as a financial return series.

2. Risk-Adjusted Metrics

The framework adapts four key financial metrics to forecast evaluation:

• Forecast Sharpe Ratio: The ratio of average return to the standard deviation of returns. It measures the average improvement per unit of total volatility.
• Forecast Sortino Ratio: The ratio of average return to the semi-standard deviation (volatility of negative returns only). This penalizes underperformance while ignoring upside variability, aligning with the asymmetric preferences of forecast users.
• Forecast Omega Ratio: The ratio of the average upside to the average downside. It captures the full distribution of gains versus losses without assuming normality, making it suitable for non-Gaussian loss differentials.
• Maximum Drawdown: The largest cumulative loss from a historical peak to a trough. This identifies prolonged episodes of sustained forecast failure, which are particularly relevant during structural breaks (e.g., inflation surges).

3. The Edge Ratio

A novel metric introduced to measure a model's propensity to deliver unique predictive advantage relative to the entire set of available alternatives (the forecasting frontier).

• Definition: At each time $t$, the benchmark is the best-performing other model ($L^*_t = \min_{j \neq M} L_{j,t}$). The "edge" is $e_{M,t} = L^*_t - L_{M,t}$.
• Calculation: The Edge Ratio is the sum of edge wins ($e^+$) divided by the sum of edge regrets ($e^-$), scaled by $(M-1)$ to correct for pool size.
• Interpretation: A high Edge Ratio indicates the model frequently reaches the frontier and does so decisively, while minimizing the regret of not choosing the best alternative when it fails.

4. Meta-Analysis Extension

The framework is extended to a cross-sectional meta-analysis across targets, horizons, and samples. Instead of time-series returns, "returns" are defined as percentage improvements in performance metrics (e.g., RMSE) across a design space. This allows for the assessment of model robustness: whether a method wins consistently or succeeds only in specific pockets while failing catastrophically elsewhere.

5. Connection to Diebold-Mariano (DM)

The paper demonstrates that the Sharpe ratio coincides with the DM t-statistic (scaled) when forecast errors are serially uncorrelated. The DM statistic's denominator (long-run variance) penalizes persistent underperformance more heavily than transient failures, effectively acting as a risk-adjusted measure that accounts for autocorrelation.

Empirical Applications and Results

Application 1: Macro Forecasting (ML/AI vs. Professional Forecasters)

The study evaluates U.S. macroeconomic variables (GDP, unemployment, inflation, housing) across pre- and post-COVID periods, comparing econometric benchmarks, machine learning (ML) models, a foundation model (TabPFN), and the Survey of Professional Forecasters (SPF).

• GDP Growth: While flexible models (Kernel Ridge, Neural Networks) often achieve higher average MSE reductions than the SPF, the SPF exhibits superior risk-adjusted performance. The SPF's Sortino and Omega ratios are significantly higher, indicating exceptionally contained downside risk. The SPF rarely suffers catastrophic failures.
• Unemployment: Hybrid boosting models (LGB+) and foundation models (TabPFN) show strong risk-adjusted profiles at specific horizons. Notably, Random Forests achieve risk-adjusted performance comparable to sophisticated foundation models, while standard neural networks suffer from high downside volatility.
• Inflation: The SPF is difficult to beat on average. However, the Hemisphere Neural Network (HNN) matches the SPF in raw returns but dominates on risk-adjusted metrics. Crucially, HNN is the only model retaining positive average returns in the post-2021 inflation surge, demonstrating robustness to regime change.
• Housing Starts: ML models generally struggle to beat the autoregressive benchmark. The SPF matches the best ML models in average returns but with significantly lower downside risk (higher Sortino ratio).

Key Finding: It is often feasible to beat professional forecasters on average accuracy, but much harder to beat them on a risk-adjusted basis. The SPF's value lies in its consistency and lack of catastrophic failures, reflecting the value of contextual judgment.

Application 2: Meta-Analyses

• Density Forecasting (Goulet Coulombe et al., 2026): Re-evaluating point and density forecasts, HNN and BART emerge as the strongest point forecasters. HNN dominates decisively in density forecasting (log scores), offering larger, more stable gains with fewer catastrophic failures compared to competitors like DeepAR.
• M4 Competition: In the M4 monthly series, average performance rankings (MASE/OWA) broadly align with risk-adjusted rankings. Models that perform well on average also tend to be robust across the cross-section. However, the Edge Ratio reveals that top models (e.g., Model 118/ES-RNN) occasionally deliver decisive gains unavailable to competitors, rather than just marginal improvements.

Key Contributions

1. Framework Integration: The paper establishes a formal framework for evaluating forecasts using risk-adjusted metrics (Sharpe, Sortino, Omega, Drawdown), shifting the focus from "is the model better on average?" to "how reliably does it outperform?"
2. The Edge Ratio: Introduces a novel metric to quantify unique predictive advantage relative to the full model pool, distinguishing between models that are merely good and those that provide irreplaceable information.
3. Re-evaluation of Professional Judgment: Provides quantitative evidence that professional forecasters (SPF) are difficult to beat not necessarily because they have the lowest average error, but because they exhibit the most favorable risk profiles (low downside volatility, high Sortino ratios).
4. Meta-Analysis Tooling: Extends these metrics to cross-sectional design spaces, allowing practitioners to assess the robustness of a method across different targets and horizons, identifying models that are prone to "catastrophic failures" in specific pockets.

Significance and Claims

The paper claims that average forecast accuracy and forecast reliability are distinct concepts. By treating loss differentials as returns, the proposed framework reveals hidden fragilities in models that appear competitive under standard RMSE metrics.

• Operational Relevance: For institutions, a model that beats a benchmark by 15% on average but fails severely during crises offers a less attractive risk-return profile than one with moderate average gains and high stability.
• Model Selection: The framework suggests that while machine learning and foundation models can outperform traditional benchmarks, their adoption requires careful scrutiny of their downside risk. Tree-based models and specific architectures like HNN show promise in offering attractive risk profiles for specific targets.
• Future Direction: The paper suggests that if risk-adjusted metrics are operationally relevant, forecasting models should ideally be optimized directly for these objectives (e.g., maximizing the Sortino ratio) rather than relying on standard loss functions (like squared error) and hoping for favorable risk properties ex post.

The paper concludes that while average performance gains are necessary, they are insufficient for operational decision-making; the stability and reliability of those gains, captured by risk-adjusted metrics, are equally critical.