💰 quantitative finance

Heads, Not Backbones: Output Heads Dominate Architectures on Fat-Tailed Returns

This paper demonstrates that for forecasting fat-tailed financial returns at short horizons, the choice of output head (specifically a Gaussian mixture model) significantly outweighs the impact of the backbone architecture in capturing tail risk and improving distributional accuracy, although the backbone becomes more critical at longer time horizons.

Original authors: Sichao He, Yansong Zhang

Published 2026-06-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sichao He, Yansong Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to predict the weather for the next few days. You have two main tools to help you:

The Engine (The Backbone): The complex computer system that processes all the data (satellite images, barometric pressure, wind speed).
The Report (The Output Head): How the final prediction is presented to you. Is it just a single temperature number? Is it a bell curve saying "it will likely be 70°F"? Or is it a detailed map showing a 90% chance of rain, a 5% chance of a tornado, and a 5% chance of a heatwave?

This paper, titled "Heads, Not Backbones," asks a simple question about predicting financial markets (specifically the S&P 500 stock index): Does the complexity of the Engine matter more, or does the way we present the Report matter more?

The authors tested four different, high-tech "Engines" (modern AI models) and combined them with three different types of "Reports" (Output Heads). Here is what they found, using simple analogies.

1. The "Fat-Tailed" Problem

Financial markets are like a rollercoaster that mostly moves gently but occasionally has terrifying, extreme drops or spikes. In statistics, this is called a "fat-tailed" distribution.

Standard models assume the market behaves like a calm lake (a normal bell curve). They are great at predicting average days but terrible at predicting crashes or sudden booms.
The Goal: The authors wanted to see if they could build a better "Report" that acknowledges these extreme risks.

2. The Experiment: Swapping the Report

The researchers kept the "Engines" (the AI backbones) constant and swapped the "Reports" (the heads):

The Point Head (The Single Number): This is like a weather app that just says, "Tomorrow will be 72°F." It doesn't tell you the risk of a storm.
The Gaussian Head (The Bell Curve): This says, "Tomorrow will likely be 72°F, give or take 5 degrees." It assumes the ups and downs are symmetrical and smooth.
The Mixture Head (The Multi-Mode Map): This is the fancy one. It says, "Tomorrow will likely be 72°F, BUT there is a small chance of a sudden drop to 60°F and a small chance of a spike to 85°F." It uses a mix of different scenarios to capture the "fat tails" (the extreme risks).

3. The Big Discovery: The Report Wins

The results were surprising and clear: The "Report" (Head) mattered way more than the "Engine" (Backbone).

Changing the Engine: If you took the best AI engine and swapped it for a slightly different one, the prediction accuracy barely changed (less than 1.5% difference). It was like swapping a Toyota engine for a Honda engine in a car; the car still drives about the same.
Changing the Report: If you kept the same engine but switched from the "Single Number" report to the "Multi-Mode Map" report, the accuracy improved significantly (by about 3.7% to 6.4%).
The Analogy: It doesn't matter if you have a Ferrari engine (the best AI) if you are driving with a blindfold (the wrong report). If you have a standard engine but a perfect GPS (the right report), you will get to your destination much more accurately.

4. Why the "Multi-Mode Map" (Mixture) is Special

The "Multi-Mode Map" (Gaussian Mixture) was the clear winner, especially during crisis periods (like the 1970s stagflation or the 2008 financial crash).

The Single Number ignores the risk of a crash.
The Bell Curve assumes a crash is just a "big fluctuation" on a smooth curve.
The Multi-Mode Map specifically builds a separate "lane" for the crash scenario. It captures the "fat tails" that other models miss.

The paper notes that this extra value is most visible when things are chaotic. When the market is calm, the fancy map isn't much better than a simple guess. But when the market is in a panic, the fancy map is the only one that sees the danger coming.

5. Important Caveats (What the Paper Does Not Say)

The authors are very honest about the limits of their findings:

It's not a "Get Rich Quick" scheme: They tested a simple trading strategy using these predictions, and it lost money. Having a better prediction of risk doesn't automatically mean you can make more profit. You still need a smart strategy to use that information.
It depends on the time frame: The "Report" (Head) is the king for short-term predictions (next few days or months). However, for long-term predictions (6 months or a year), the "Engine" (Backbone) starts to matter more again.
It depends on the asset: This works great for stock returns (which are wild and fat-tailed). It does not work as well for things like Treasury bond yields or currency exchange rates, which behave more like a steady river than a rollercoaster. For those, a simple "Single Number" report is often just as good.

The Bottom Line

If you are trying to predict the wild, unpredictable swings of the stock market in the short term, don't obsess over which complex AI model you use. Instead, focus on how you interpret the data.

Using a model that can describe multiple possible futures (including the scary ones) is far more important than using the most sophisticated computer architecture. The "Head" (the output) dominates the "Backbone" (the architecture).

Technical Summary: Heads, Not Backbones

Problem Statement

In deep learning pipelines for forecasting fat-tailed financial returns (specifically short-horizon horizons), there is a prevailing assumption that architectural complexity (the "backbone") is the primary driver of performance. However, risk management, capital allocation, and regulatory stress testing require distributional forecasts (e.g., Value-at-Risk, Expected Shortfall) rather than point estimates.

Dominant deep forecasting architectures (TimesNet, DLinear, N-BEATS, iTransformer) typically ship with point heads trained on Huber or MSE loss. These losses incentivize the conditional median or mean but are not strictly proper scoring rules for fat-tailed distributions, failing to capture tail risk. The standard remedy is to replace the point head with a density head (e.g., Gaussian Mixture Model) trained via Negative Log-Likelihood (NLL).

This paper investigates a fundamental question: On fat-tailed financial returns, which matters more for predictive accuracy—the backbone architecture or the output head?

Methodology

Data and Protocol

Dataset: S&P 500 monthly log-returns (1871–2023, 1,832 observations) from the Shiller dataset. Empirical kurtosis is $\approx 6$ , significantly higher than the Gaussian value of 3.
Validation: Five anchored walk-forward folds with a training fraction of 0.50, test window of 0.07, and step of 0.10. Experiments are repeated across three random seeds.
Horizons: $h \in \{1, 3, 6, 12\}$ months.

Experimental Design

The authors construct a clean $4 \times 3$ factorial experiment (12 variants) to isolate the effects of the backbone and the head:

Backbones (4):
- TimesNet: FFT-based period discovery + 2D convolution.
- DLinear: Trend/seasonal decomposition + linear projection.
- N-BEATS: Residual stack of generic basis-expansion blocks.
- iTransformer: Per-time-step embedding + self-attention across time dimension.
- Note: PatchTST was excluded due to incompatibility with density heads (patching- $\sigma$ issue).
Output Heads (3): All share the same backbone hidden state ( $d_{hidden}=64$ $d_{hi dd e n} = 64$ ).
- Point Head: Linear projection to $\hat{y} \in \mathbb{R}^H$ , trained with Huber loss. (Evaluated via post-hoc Gaussian approximation for CRPS).
- Single-Gaussian Head: Linear projection to $(\mu, \sigma) \in \mathbb{R}^{2H}$ , trained with Gaussian NLL.
- Gaussian Mixture Head (GMM): Linear projection to $(\mu, \sigma, \pi) \in \mathbb{R}^{3HK}$ with $K=4$ components, trained with GMM NLL.

Metrics

Primary: Continuous Ranked Probability Score (CRPS) and CRPS-Skill-Score (relative to a TimesNet point baseline).
Secondary: Pinball loss (quantile loss) at $\tau \in \{0.05, 0.95\}$ , predictive-interval coverage, and Mean Squared Error (MSE).
Statistical Tests: Hansen–Lunde–Nason Model Confidence Set (MCS) on squared errors; Diebold–Mariano paired $t$ -tests on per-fold CRPS differentials.

Key Results

1. The Head Dominates the Backbone (Short Horizons)

On monthly S&P 500 returns at short horizons ( $h < 6$ ), the output head is the dominant factor:

Strict Gradient: Moving from Point $\to$ $\to$ Gaussian $\to$ $\to$ GMM yields a strict improvement in CRPS.
- Point $\to$ Gaussian: $\sim +1.3\%$ improvement.
- Gaussian $\to$ GMM: $\sim +2.4\%$ improvement.
- Total Point $\to$ GMM: $\sim +3.7\%$ improvement.
Backbone Impact: Switching backbones while holding the head constant changes CRPS by less than 1.5% on the point-head row. Even on density heads, the backbone spread (max 5.1% driven by N-BEATS) is generally smaller than the head gradient.
MCS Results: The Model Confidence Set on squared errors does not exclude any of the 12 variants at the 5% level. Neither backbone nor head distinguishes models on point-prediction accuracy (MSE). The head only separates models on distributional metrics (CRPS, Pinball, coverage).

2. The "h-Split" (Horizon Dependence)

The dominance of the head is horizon-dependent:

Short Horizons ( $h \le 3$ ): The head dominates. GMM heads on deep backbones outperform classical baselines.
Long Horizons ( $h \ge 6$ ): The backbone re-takes the lead. Classical recursive models (GARCH, ARIMA) with GMM heads collapse (e.g., GARCHgmm drops to $-50\%$ CRPS skill at $h=6$ ) because their volatility forecasts mean-revert to the unconditional mean. Deep backbones with multi-step direct training maintain performance, outperforming classical models by $\sim 55$ percentage points at $h=6$ .

3. Cross-Asset Generalization

The head gradient is not universal; it depends on the data generating process:

Return-like Processes: (Monthly/Daily S&P 500, Daily VIX). The GMM head is consistently best.
- VIX Case: A single Gaussian head fails catastrophically ( $-24.4\%$ ) on VIX due to extreme right-tail spikes. The GMM ( $K=4$ ) successfully captures these multimodal regimes.
Non-Return-like Processes: (Treasury yields, EUR/USD). These are mean-reverting around structural targets. Here, the GMM head adds parameter noise without structural benefit, often performing worse than the point head.

4. Risk Management Value

Calibration: GMM heads provide superior calibration (90% coverage $\approx 0.88\text{--}0.93$ ) compared to point heads (which over-cover at 0.96 due to post-hoc variance inflation).
Tail Risk: The incremental value of the mixture over a single Gaussian is largest in high-volatility regimes (e.g., $+13.9\%$ improvement during 1970s stagflation).
Regulatory Metrics: N-BEATS with a GMM head passes Kupiec's unconditional coverage test at 5%, 2.5%, and 1% VaR levels and achieves the lowest Expected Shortfall (ES) among tested models.
Limitation: At the extreme 1%-VaR threshold, the bounded $K=4$ GMM under-prices tail risk compared to the point head (due to Gaussian component tails), suggesting a need for GMM+EVT hybrids for extreme quantiles.

5. Trading Strategy Backtest

A naive mean-reversion strategy using these density forecasts loses money on all variants (annualized returns $-41\%$ to $-50\%$ ). This confirms that improved distributional forecasts are necessary for risk management but do not automatically translate to trading alpha without sophisticated position sizing and transaction cost modeling.

Significance and Claims

The paper concludes that for fat-tailed financial returns at short horizons, engineering effort is best spent on the output head, not the backbone.

Architectural Neutrality: For point prediction, deep backbones are statistically indistinguishable on this data. The "signal" distinguishing backbones is below the noise floor of point-accuracy metrics.
Loss Function Primacy: The gain comes from using a strictly proper scoring rule (GMM NLL) that penalizes tail miscalibration, which Huber/MSE losses ignore. The GMM head captures excess kurtosis and tail risk that unimodal Gaussians cannot.
Contextual Validity: The "head-dominance" claim holds specifically for return-like, fat-tailed processes. It does not apply to non-stationary, mean-reverting series like yields or FX, where the point head remains competitive.
Practical Implication: For risk management (VaR, ES, capital charges), switching from a standard point head to a GMM head is a high-value, low-cost intervention (a "2-line change") that yields genuine value during crisis periods, whereas swapping backbones yields negligible returns.

The paper explicitly states it does not propose a new backbone, a new trading strategy, or a universal solution for all time series. It establishes a boundary condition: on fat-tailed returns, the head dominates; on other processes, it does not.