💰 quantitative finance

A Controlled Comparison of Deep Learning Architectures for Multi-Horizon Financial Forecasting: Evidence from 918 Experiments

Through a rigorous controlled comparison of 918 experiments across nine deep learning architectures, this study identifies ModernTCN as the superior model for multi-horizon financial forecasting, demonstrating that architectural inductive bias significantly outweighs hyperparameter tuning and seed randomness while revealing that current MSE-trained models lack directional skill at hourly resolutions.

Original authors: Nabeel Ahmad Saidd

Published 2026-03-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Nabeel Ahmad Saidd

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a chef trying to decide which recipe makes the best soup. You have nine different recipes (Deep Learning Architectures), and you want to know which one works best for three different types of ingredients: spicy vegetables (Cryptocurrency), mild broth (Forex), and hearty meat (Stock Indices).

Most previous cooking competitions were rigged. Some chefs got to taste-test their soup 100 times to tweak the salt, while others got one shot. Some only tasted the soup at 5 PM, ignoring how it tasted at 8 PM. And no one checked if the results were just a fluke because the chef got lucky with the random stirring of the pot.

This paper is the first truly fair cooking competition for financial forecasting. The author, Nabeel Ahmad Saidd, ran 918 experiments to find out which "recipe" actually works best for predicting stock and crypto prices.

Here is the breakdown in simple terms:

1. The Contenders (The Nine Recipes)

The study tested nine different "AI chefs" (Deep Learning models) from four different schools of thought:

The Transformers: The new, fancy chefs who look at the whole pot at once to find patterns (like PatchTST and Autoformer).
The CNNs (Convolutional): The chefs who use big, wide spoons to scoop up patterns over a wide area (like ModernTCN).
The MLPs (Linear): The simple, no-nonsense chefs who believe a straight line is often the best path (like DLinear).
The RNNs (LSTM): The old-school chefs who remember the past one step at a time (like LSTM).

2. The Rules of the Game (The Protocol)

To make sure the results were fair, the author set up strict rules:

Same Ingredients: Every chef got the exact same data (Open, High, Low, Close, Volume prices).
Same Training Time: Every chef got the same amount of time to practice (Hyperparameter Optimization).
The "Three-Taste" Rule: Instead of just tasting the soup once, every chef made the soup three times with slightly different random starting points. If a chef's soup tasted different every time, they were unreliable. If it tasted the same, they were consistent.
Two Timeframes: They tested predicting the price 4 hours ahead (short-term) and 24 hours ahead (long-term).

3. The Results: Who Won?

The results were surprising and clear. The winners formed a distinct "Podium":

🥇 Gold Medal: ModernTCN
This model won 75% of the time. It's like a chef who uses a giant, high-tech ladle (large-kernel convolution) to scoop up patterns perfectly. It worked great on stocks, forex, and crypto. It was the most reliable "all-rounder."

🥈 Silver Medal: PatchTST
This model came in second. It's a Transformer that cuts the soup into "patches" (chunks) to analyze them. It was very close to the winner, especially on short-term predictions.

🥉 The Rest of the Pack
The other models fell into a middle group and a bottom group.

The Middle: Models like iTransformer and DLinear were decent but not great.
The Bottom: The old-school LSTM model came in last, with errors 7 to 33 times worse than the winner. It's like trying to win a race with a bicycle while everyone else is on a Ferrari.

4. The Big Surprises (Key Takeaways)

Surprise #1: Bigger isn't always better.
You might think the model with the most complex brain (most parameters) would win. But the simplest model (DLinear, with almost no "brain" power) beat the most complex ones.

Analogy: It's like realizing a simple, well-tuned hammer is better at driving a nail than a giant, over-engineered robot arm. The design matters more than the size.

Surprise #2: The "Seed" didn't matter.
In AI, you often start with a random "seed" (like shuffling a deck of cards). Some people worry that if you shuffle the cards differently, you get a different winner.

Finding: The study found that 99.9% of the difference in results was due to the recipe (the model architecture), and only 0.01% was due to the random shuffle.
Analogy: If you bake a cake, the recipe determines if it tastes good. Whether you mix the batter with your left or right hand (the seed) makes almost no difference. You don't need to bake the cake 100 times to know if the recipe is good; three times is enough.

Surprise #3: The "Direction" Problem.
This is the most important lesson for traders. While the models were great at predicting the price (e.g., "The price will be $100"), they were terrible at predicting the direction (e.g., "Will it go up or down?").

Finding: The models were right about the direction only 50% of the time.
Analogy: Imagine a weatherman who predicts the temperature will be 75°F. He is very accurate! But if you ask him, "Will it rain?" he is just flipping a coin.
Why? The models are trained to minimize the distance of the error (MSE). They naturally "play it safe" and predict the average, which kills their ability to spot sharp turns up or down.

5. What Should You Do? (Practical Advice)

If you are a portfolio manager or a trader using AI:

Use ModernTCN: It's the most reliable "workhorse" for almost any asset class.
Use PatchTST: If you prefer Transformer models, this is your best bet.
Don't use LSTM: It's outdated for this specific job.
Beware of "Directional" Claims: Just because a model predicts the price accurately doesn't mean it can tell you when to buy or sell. If you need to know the direction, you need to change how the model is trained, not just pick a better model.

The Bottom Line

This paper is a massive "quality control" check for the financial AI industry. It proves that how you design the model matters more than how much data you feed it or how many times you shuffle the deck. It gives us a clear leaderboard so we stop guessing and start using the tools that actually work.

1. Problem Statement

Multi-horizon price forecasting is critical for portfolio allocation, risk management, and algorithmic trading. However, the rapid proliferation of deep learning architectures (Transformers, CNNs, MLPs, RNNs) has outpaced rigorous empirical validation on financial data. Existing benchmarks suffer from five critical methodological flaws:

Uncontrolled Hyperparameter Budgets: Models are often compared with unequal tuning efforts.
Single-Seed Evaluation: Results rely on a single random initialization, ignoring stochastic variance.
Single-Horizon Analysis: Most studies evaluate only one forecast window, failing to capture how models degrade over time.
Absent Statistical Correction: Pairwise comparisons often lack corrections for multiple testing (e.g., family-wise error control).
Narrow Asset Coverage: Benchmarks rarely span diverse financial markets (crypto, forex, equities) with distinct microstructures.

The paper aims to resolve these gaps by providing a controlled, statistically validated comparison of nine state-of-the-art architectures across three asset classes and two forecasting horizons.

2. Methodology

The study employs a rigorous five-stage experimental protocol involving 918 total runs (270 hyperparameter optimization trials + 648 final training runs).

Experimental Design

Architectures (9 Models): Four families were evaluated:
- Transformers: Autoformer, iTransformer, PatchTST, TimeXer.
- MLP/Linear: DLinear, N-HiTS.
- CNN/TCN: ModernTCN, TimesNet.
- RNN: LSTM (as a baseline).
Data Universe: 12 financial instruments across three classes:
- Cryptocurrency: BTC, ETH, BNB, ADA (High volatility, 24/7).
- Forex: EUR/USD, USD/JPY, GBP/USD, AUD/USD (High liquidity, mean-reverting).
- Equity Indices: Dow Jones, S&P 500, NASDAQ 100, DAX (Trending, session-based).
Forecasting Setup:
- Horizons: Short-term ( $h=4$ hours) and Long-term ( $h=24$ hours).
- Strategy: Direct multi-step forecasting (predicting the entire horizon vector in one pass) to avoid recursive error accumulation.
- Features: Raw OHLCV (Open, High, Low, Close, Volume) only, with no technical indicators or external data.
- Splits: Strictly chronological 70/15/15 (Train/Val/Test).

Protocol Stages

Fixed-Seed HPO: Bayesian optimization (Optuna TPE) with a fixed seed (42) and a limited budget (5 trials) on representative assets to ensure fairness.
Configuration Freezing: Best hyperparameters found per asset class are frozen and applied to all assets in that class to prevent overfitting.
Multi-Seed Training: Final models are trained with three independent seeds (123, 456, 789) to quantify initialization variance.
Metric Aggregation: Performance is measured on the test set using RMSE, MAE, and Directional Accuracy (DA), aggregated as mean $\pm$ std dev across seeds.
Statistical Validation: Includes Friedman-Iman-Davenport omnibus tests, Holm-Wilcoxon pairwise tests, Diebold-Mariano tests, and two-factor variance decomposition.

3. Key Contributions

Protocol-Controlled Fairness: The first study to simultaneously address uncontrolled HPO, single-seed bias, and narrow asset coverage in financial forecasting.
Variance Decomposition: Demonstrates that architecture choice explains >99.9% of forecast variance, while random seed initialization accounts for <0.04%. This validates that three-seed replication is sufficient for robust conclusions.
Cross-Horizon Generalization: Analyzes how models degrade from $h=4$ to $h=24$ , showing that top-tier rankings remain stable despite absolute error amplification (2–2.5 $\times$ ).
Asset-Class Specific Guidance: Provides a "best-model matrix" revealing that while top models are generally robust, specific architectures (e.g., N-HiTS) have niche advantages on lower-capitalization crypto assets.
Open Benchmarking: Releases the full codebase, data, trained models, and evaluation outputs for independent replication.

4. Key Results

Performance Rankings (Global Leaderboard)

A clear three-tier hierarchy emerged across all 24 evaluation points (12 assets $\times$ 2 horizons):

Top Tier: ModernTCN (Mean Rank: 1.33, 75% win rate) and PatchTST (Mean Rank: 2.00).
- ModernTCN dominated, winning 18/24 points. It excelled particularly in Forex and Equity indices at the 24-hour horizon.
- PatchTST was a strong second, utilizing patch-based tokenization.
Middle Tier: iTransformer, TimeXer, DLinear, N-HiTS.
- N-HiTS showed niche dominance on lower-cap crypto (ETH, ADA).
- DLinear (linear model) performed surprisingly well, ranking 5th globally with minimal parameters (~1k).
Bottom Tier: TimesNet, Autoformer, LSTM.
- LSTM performed worst across all conditions, with errors 7–33 $\times$ higher than the best models.
- TimesNet and Autoformer struggled, likely due to their reliance on periodic assumptions (FFT/Auto-correlation) that do not hold well for hourly financial noise.

Complexity vs. Performance

Non-Monotonic Relationship: There is no correlation between parameter count and performance.
- DLinear (~~1k params) outperformed Autoformer (~~438k params) and LSTM (~172k params).
- This confirms that inductive bias (how parameters are structured) matters more than raw model capacity.

Directional Accuracy

Coin Flip Result: Across all 54 model-category-horizon combinations, the mean Directional Accuracy (DA) was 50.08%, indistinguishable from a random coin flip.
Implication: MSE-optimized architectures are excellent at predicting magnitude but fail to capture directional signals at hourly resolution. Trading applications would require explicit directional loss functions.

Statistical Significance

Variance Dominance: Architecture choice explains 99.90% of raw RMSE variance. Seed variance is negligible (0.01%).
Rank Stability: Spearman correlations between $h=4$ and $h=24$ rankings were high ( $\rho > 0.68$ ), confirming that top models generalize well to longer horizons.
Pairwise Tests: Holm-corrected Wilcoxon tests confirmed that the gap between the Top Tier (ModernTCN/PatchTST) and the rest is statistically significant ( $p < 0.05$ ).

5. Significance and Implications

Practical Deployment: Practitioners should prioritize ModernTCN for general multi-horizon forecasting due to its robustness across asset classes and horizons. PatchTST is a viable Transformer-based alternative. N-HiTS is recommended specifically for volatile, lower-cap crypto assets.
Methodological Shift: The study proves that extensive hyperparameter tuning on a single asset is less critical than selecting the correct architecture with the right inductive bias. Furthermore, it establishes that three-seed replication is sufficient to eliminate initialization noise in financial forecasting.
Limitations of Current DL: The failure of all models to predict direction (DA $\approx$ 50%) highlights a fundamental limitation of current MSE-trained deep learning in financial markets, suggesting that regression-to-the-mean is the primary signal learned, not directional trends.
Reproducibility: By releasing a fully deterministic, open-source framework, the authors set a new standard for reproducibility in financial time-series research, moving the field away from "cherry-picked" results toward rigorous, controlled benchmarking.

In conclusion, this paper provides the most comprehensive and statistically rigorous benchmark to date, identifying ModernTCN as the current state-of-the-art for multi-horizon financial forecasting while debunking the necessity of massive model capacity or complex attention mechanisms for this specific domain.