📈 economics

Dispersion-Matched Forecast Comparison: Separating Skill from Shrinkage in the Evaluation of Financial Prediction Models

This paper demonstrates that conventional financial forecast comparisons based on squared error are fundamentally flawed because they inadvertently rank models by their dispersion shrinkage rather than predictive skill, leading to spurious statistical significance, and proposes a "dispersion-matched" correction that rescales forecasts to the target's variance to isolate and accurately test genuine predictive ability.

Original authors: Shelly Suman Khuntia, Ajaya Kumar Parida, Satya Narayan Misra, Santisudha Panigrahi

Published 2026-10-12

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Shelly Suman Khuntia, Ajaya Kumar Parida, Satya Narayan Misra, Santisudha Panigrahi

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of finance, predicting the future price of a stock is the holy grail. Investors and banks pour billions into sophisticated computer systems, hoping to find a pattern in the chaos of daily market movements. These systems, often built using deep learning, are designed to digest vast amounts of information—past prices and economic reports—to guess whether a stock will rise or fall tomorrow. The standard way to judge if one of these systems is actually good is to compare its predictions against a list of other systems. If a new model makes fewer errors than the old ones, it is declared a winner, and its creators publish the results as a breakthrough. This process relies on a simple assumption: that a lower error score means the model has found a genuine signal in the data, a real ability to see what others cannot.

However, a new study challenges the very foundation of how these winners are chosen. The researchers, working with a team of experts in computer science and management, discovered that the standard scoring method can be fooled. They found that a computer model can achieve a perfect score not because it has learned to predict the market, but because it has learned to do almost nothing. When a market is difficult to predict, the safest mathematical move is to simply guess the average outcome every single time. If a model is too cautious, or if its design forces it to be too conservative, it will shrink its predictions until they are nearly flat. The study shows that the current ranking systems mistake this shrinking behavior for skill, creating a leaderboard where the "best" models are actually the ones that have given up on predicting anything at all.

The researchers set out to test a specific, modern architecture designed to forecast daily stock returns. This system, called MFGate-Net, was built to be a state-of-the-art predictor. It combined two different types of information: technical indicators derived from price charts and macroeconomic data like interest rates and market volatility, along with a mechanism to weigh these inputs dynamically. To see if it worked, the team trained it on seven years of historical data and then tested it on a fresh, unseen period covering the years 2022 through 2024. This test period was chosen carefully to include different market conditions, including a period of high inflation and a recovery phase, ensuring the model faced a realistic challenge.

When the team ran the standard comparison, the results looked like a triumph. The new model beat every other competitor, including older, simpler machine learning methods and other complex deep learning networks. It had the lowest error rate, and statistical tests confirmed that its advantage was not a fluke. By all conventional accounts, this should have been the headline: a new, superior way to predict the market. But the researchers did not stop there. They asked a simpler, more fundamental question: did this model actually beat the most basic possible prediction? In finance, the simplest prediction is to assume that tomorrow's price change will be exactly zero. This is known as the unconditional benchmark. It is the baseline of doing nothing.

When they compared their sophisticated model against this zero prediction, the story changed completely. The complex model did not beat the zero forecast. In fact, it was statistically indistinguishable from it. The model's predictions were so close to zero that they offered no real advantage over simply guessing that nothing would happen. Even more striking, every other model on the leaderboard, from the newest deep learning networks to older statistical methods, also failed to beat the zero forecast. They were all worse than doing nothing. The researchers realized that the ranking they had seen earlier was not measuring who was the best predictor, but rather who was the most cautious. The models that shrank their predictions the most, moving closest to zero, appeared to have the lowest error. The ranking was a measurement of how much the models had collapsed, not how much they had learned.

To prove this, the team derived a mathematical explanation for why this happens. They showed that when a market is nearly unpredictable, the error score of a model depends almost entirely on how much its predictions vary compared to the actual market. If a model predicts very little movement, its error score will look good, even if it is wrong. The study demonstrated that the apparent superiority of one model over another was simply a difference in how much they shrank their forecasts. The statistical tests used to declare a winner were actually measuring this shrinkage, and the more data they collected, the more confident they became in a false conclusion. The larger the dataset, the stronger the illusion of skill became, even though no skill existed.

The researchers then proposed a simple fix. Before comparing models, they adjusted every prediction so that its range of movement matched the range of the actual market. This process, which they call dispersion matching, removes the bias toward shrinking. Once they applied this correction, the dramatic differences between the models vanished. The models that had previously looked like clear winners were now statistically tied with the losers. The "overwhelming" significance that had suggested a breakthrough disappeared, replaced by a result showing that all the models were essentially performing the same as a random guess. In one specific case, a model that had been declared the best was found to be no better than a constant zero forecast, with a statistical probability of 0.97 that the difference was due to chance.

The study also looked at whether these models could make money. Even if a model cannot predict the price perfectly, it might still be useful if it can guide profitable trades. The researchers simulated trading strategies based on the models' predictions, accounting for the costs of buying and selling. The results were stark. No model generated a profit that could beat a risk-free investment, even when transaction costs were set to zero. The most successful model on the error charts turned out to be the worst performer in terms of money, because its cautious predictions led to a strategy that barely moved, accumulating small losses against the cost of capital. This confirmed that the statistical ranking was not just misleading, but economically irrelevant.

Perhaps the most unsettling finding concerned the tools used to explain how these models work. In the field of artificial intelligence, researchers often use visualizations to show which data points a model is paying attention to, hoping to prove that the system has learned meaningful patterns. The team tested this by training their model on data where the target had been randomly shuffled, destroying any real relationship between the inputs and the output. A model trained on such noise cannot possibly learn anything. Yet, when they analyzed this "broken" model, it produced explanations that looked almost identical to the ones from the real model. The visualizations showed the same features being highlighted and the same patterns of attention. This suggests that these interpretability tools are often reflecting the structure of the data itself rather than the specific knowledge the model has gained. They cannot distinguish between a model that has learned a secret and one that has learned nothing.

The researchers concluded that the entire field of financial prediction needs to change how it reports its results. They recommend that every study must include a comparison against the simple zero forecast to prove that a model has any skill at all. They also suggest reporting a measure of how much the model's predictions vary, to ensure that a low error score is not just the result of being overly cautious. Without these checks, the literature is filled with models that appear to be advancing the state of the art but are actually just ranking themselves by how well they can pretend to be nothing. The study does not claim that predicting the market is impossible, nor does it say that deep learning is useless. It simply shows that the current way of judging success is broken, and that until the rules are fixed, the winners on the leaderboard may not be the ones who can actually see the future.

Technical Summary: Dispersion-Matched Forecast Comparison

Problem Statement
The paper identifies a critical flaw in the standard protocol for evaluating financial prediction models, particularly in deep learning applications for daily equity return forecasting. Conventional practice involves ranking models by Mean Squared Error (MSE) and certifying superiority using the Diebold–Mariano (DM) test. The authors demonstrate that under a "no-skill" null hypothesis (where a target is effectively unpredictable), this protocol does not measure predictive skill. Instead, it ranks models based on shrinkage—the degree to which a model's forecast variance collapses toward the unconditional mean (approximately zero for daily log returns).

The core issue is that MSE confounds forecast dispersion with correlation. A model with no predictive signal minimizes training loss by collapsing toward a constant. Consequently, the model with the lowest MSE is simply the one that has shrunk its forecast variance the most. This leads to two counter-intuitive findings:

Spurious Significance: The DM statistic grows with the square root of the sample size ( $\sqrt{T}$ ) whenever two skill-less models have different shrinkage ratios. Thus, enlarging the evaluation window strengthens the apparent evidence of superiority rather than exposing the absence of skill.
Benchmark Failure: Models that appear superior to each other often fail to beat a trivial unconditional benchmark (e.g., predicting zero), yet standard literature frequently omits this comparison.

Methodology
The paper proposes a theoretical framework and a practical correction, validated on a testbed of nine forecasters (including MFGate-Net, LSTM, GRU, Transformer, CNN, RF, SVR) across ten US and Indian equity targets (2014–2024).

Theoretical Derivation:
- The authors derive a closed-form expression for MSE under the no-skill null: $MSE_i = \sigma^2(1 + \lambda_i^2) + b_i^2$ , where $\lambda_i$ is the ratio of forecast dispersion to target dispersion, and $b_i$ is bias.
- This proves that error ranking is strictly a ranking of shrinkage ( $\lambda_i$ ) when skill ( $\rho_i$ ) is zero.
- They show that the expected DM statistic diverges as $\sqrt{T}$ for models with unequal $\lambda_i$ , explaining why larger datasets yield more "significant" but spurious results.
The Correction: Dispersion Matching:
- To isolate skill from shrinkage, the authors propose dispersion matching. Before testing, every forecast is rescaled to match the target's dispersion: $\tilde{r}_i = \frac{\hat{r}_i - \mu_i}{\lambda_i \sigma} \sigma + \mu$ .
- This transformation leaves the correlation ( $\rho_i$ ) unchanged but forces $\lambda_i = 1$ .
- Post-matching, the MSE becomes $2\sigma^2(1 - \rho_i)$ , a function of skill alone. A DM test on these matched forecasts then tests for differential skill rather than differential shrinkage.
- The procedure requires no retraining and costs only two lines of code.
Aggregation and Diagnostics:
- The paper critiques the use of Fisher's method for combining p-values across targets due to cross-sectional dependence in equity panels. It advocates for the Cauchy combination test, which remains valid under arbitrary dependence.
- It recommends reporting two diagnostics alongside any comparison: an unconditional benchmark (to establish if skill exists) and the dispersion ratio ( $\lambda$ ).

Key Results

The "Zero" Forecast Wins: On the held-out test set (2022–2024), the trivial forecast $\hat{r}_{t+1} = 0$ achieved the lowest RMSE (0.01461) of all nine trained models. The best deep learning model (MFGate-Net) was statistically indistinguishable from zero ( $p=0.97$ ), while all other models were significantly worse than zero.
Shrinkage Drives Rankings: The dispersion ratio ( $\lambda$ ) ranged from 0.078 (MFGate-Net) to 1.619 (CNN). The ranking by RMSE was almost perfectly correlated with the ranking by shrinkage ( $r=0.88$ ), confirming that the leaderboard was measuring variance reduction, not accuracy.
Collapse of Significance: When applying the dispersion-matched comparison:
- Seven of eight pairwise comparisons between MFGate-Net and baselines collapsed from overwhelming significance ( $p < 10^{-14}$ ) to non-significance ( $p \in [0.78, 0.97]$ ).
- Win/loss records shifted from near-total dominance (e.g., 10/0) to near parity (e.g., 6/4).
- The only exception was the SVR baseline, which had a dispersion ratio near zero; rescaling it amplified numerical noise, highlighting a boundary condition where $\lambda < 0.01$ requires direct comparison to the benchmark rather than matching.
Economic Performance: No model supported a profitable trading strategy. The Deﬂated Sharpe Ratio was negative for all models, even at zero transaction costs. The model with the best RMSE (MFGate-Net) had the worst economic performance due to its near-constant position sizing.
Interpretability Limitations: A permutation control (training on shuffled labels) showed that attribution methods (SHAP) and gate activations produced similar structures for both real and noise-trained models. This indicates that interpretability outputs in this context reflect inductive bias and input distribution rather than learned signal.

Significance and Claims
The paper does not claim that daily equity returns are inherently unpredictable or that deep learning cannot eventually find signals. Instead, it claims that standard evaluation protocols are insufficient to distinguish a model that has learned a signal from one that has learned to predict almost nothing.

Protocol Correction: The standard protocol (MSE ranking + DM test) is shown to be a measurement of shrinkage, not skill. The proposed dispersion-matched comparison is a necessary correction to test for differential skill.
Reproducibility: The authors release code and data to demonstrate that the same architecture, when given a synthetic planted signal, recovers it with high correlation ( $r=0.97$ ), proving the pipeline is functional and the lack of skill is due to the absence of a signal in the financial data, not a broken implementation.
Practical Recommendation: The authors urge the community to report unconditional benchmarks and dispersion ratios. Without these, a leaderboard may simply rank models by how aggressively they shrink toward the mean, creating a false narrative of progress.

In summary, the paper argues that the "consistency" of deep learning results in financial forecasting may reflect a shared blind spot in evaluation methodology rather than accumulated predictive power. The proposed correction allows researchers to separate genuine skill from the statistical artifact of shrinkage.

Technical Summary: Dispersion-Matched Forecast Comparison

More like this