💰 quantitative finance

Overparametrized models with posterior drift

This paper demonstrates that posterior drift significantly degrades the out-of-sample forecasting accuracy of overparametrized machine learning models, particularly in financial markets where regime changes occur, ultimately advising caution when using large linear models for stock market predictions due to their sensitivity to parameter choices and sub-periods.

Original authors: Guillaume Coqueret, Martial Laguerre

Published 2026-05-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Guillaume Coqueret, Martial Laguerre

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Super-Student" Who Gets Confused by a New Teacher

Imagine you are training a brilliant student (a machine learning model) to predict the stock market. You give this student a massive textbook with thousands of pages of data (more variables than actual examples). This is what the paper calls an overparametrized model.

In the world of machine learning, there is a recent idea called "double descent." It suggests that if you give a student too much information (too many parameters), they might actually get better at guessing the future, rather than worse. This sounds like magic: more complexity equals better results.

However, this paper argues that this magic trick has a fatal flaw when the rules of the game change. The authors call this flaw posterior drift.

The Core Problem: The "New Teacher" Arrives

To understand posterior drift, imagine this scenario:

Training Phase: You teach your student using a specific teacher (the "training data"). This teacher explains that "When it rains, the stock market goes up." The student memorizes this rule perfectly, even if it's a bit of a stretch.
Testing Phase: You take the student out to the real world to make predictions. But, the world has changed. A new teacher (the "testing data") is now in charge. This new teacher says, "Actually, when it rains, the stock market goes down."

Posterior drift is the gap between what the student learned from the old teacher and what the new teacher is actually saying.

The paper finds that when this happens, the "super-student" (the complex model) doesn't just make a small mistake. Because the student was so focused on memorizing every tiny detail of the old teacher's specific style, they get completely confused when the new teacher arrives. The more complex the student is, the more they suffer when the rules change.

The Financial Experiment: Betting on the Market

The authors tested this using a strategy called market timing.

The Goal: Predict if the stock market will go up or down next month.
The Method: They used a complex model with hundreds of predictors (like a student with a huge textbook) to guess the "Equity Premium" (the extra return you get for taking a risk).
The Twist: They checked if the relationship between their predictors and the market stayed the same over time.

What they found:

The "Double Descent" Illusion: In some specific time periods (like 2005–2019), the complex model looked amazing, generating huge returns. It seemed like the "more parameters, the better" theory was true.
The Reality Check: When they looked at different time periods (like 1975–1989), the same complex model performed terribly. In fact, it sometimes lost money.
The Conclusion: The model's success wasn't because it was smart; it was because it got lucky that the "teacher" (the market rules) didn't change much during that specific 15-year window.

The "Bandwidth" Knob: How Much Detail Do We Need?

The paper introduces a concept called bandwidth, which you can think of as a "detail knob" on a camera.

Low Bandwidth (High Detail): The camera takes a super-sharp photo. It captures every tiny leaf on a tree. If the wind blows the leaves (a regime change), the photo looks nothing like the real scene. The paper found that with low bandwidth, the model's performance is a rollercoaster. One 15-year period might yield a 7% monthly return, while the next yields almost nothing.
High Bandwidth (Blurry Photo): The camera blurs the image slightly. It misses the tiny leaves but sees the general shape of the tree. The paper found that turning up the bandwidth (making the model simpler/less sensitive to tiny details) made the results much more consistent. However, the average return dropped significantly, often close to zero.

The Trade-off: You can have a model that is wildly successful in one era but fails in the next, OR you can have a boring, consistent model that barely beats the market. There is no "free lunch."

The "Signal-to-Noise" Analogy

Imagine you are trying to hear a friend's voice (the signal) in a crowded, noisy room (the noise).

Financial markets are very quiet. The "signal" that predicts the market is very weak.
Complex models are like people with super-hearing. They try to hear everything.
The Problem: When the room gets noisy or the friend changes their voice (posterior drift), the super-hearing person gets overwhelmed by the noise and starts hallucinating patterns that aren't there.
The Paper's Warning: Because the signal in finance is so weak, adding more complexity (more parameters) just amplifies the noise and the confusion caused by changing market rules.

The Final Verdict

The paper concludes with a strong warning for investors and analysts:

Don't trust the "Complexity is Virtuous" hype. Just because a model has thousands of parameters doesn't mean it will work in the future.
Beware of "Regime Changes." Financial markets change their behavior (like a teacher changing their mind). Complex models are very fragile when these changes happen.
Backtesting is tricky. If you test a complex model on a 15-year period, you might get great results. But if you test it on a different 15-year period, you might get terrible results. The performance is highly sensitive to when you test it.

In short: While sophisticated models can be useful, relying on massive, overcomplicated models for stock market predictions is risky. If the market's "rules" shift even a little, these models tend to break down, leaving investors with inconsistent and unpredictable returns. The authors recommend caution and suggest that simpler, more robust approaches might be safer for the average investor.

Technical Summary: Overparametrized Models with Posterior Drift

Problem Statement

This paper investigates the impact of posterior drift (also known as concept drift) on the out-of-sample forecasting accuracy of overparametrized linear models, specifically in the context of equity premium prediction. While recent literature (e.g., Kelly et al., 2024) suggests that increasing model complexity (the ratio of parameters to observations) can improve market timing performance through "benign overfitting" or "double descent," this paper argues that such benefits are contingent on the invariance of the data generating process (DGP) between training and testing.

In financial markets, the DGP is rarely static. The authors distinguish between two types of distributional shifts:

Covariate Shift: Changes in the marginal distribution of predictors ( $P_X$ ).
Posterior Drift: Changes in the conditional distribution of the target given the predictors ( $P_{y|X}$ ), meaning the relationship (loadings/betas) between predictors and returns evolves over time.

The central problem is that standard theoretical results for overparametrized models often assume $P_{y|X}$ remains constant. The authors posit that when posterior drift occurs—specifically when the regression coefficients estimated in-sample ( $\beta_{is}$ ) diverge from those in the out-of-sample period ( $\beta_{oos}$ )—the performance of complex linear models deteriorates significantly.

Methodology

Theoretical Framework

The authors extend the asymptotic analysis of Hastie et al. (2022) to a setting with misspecification and posterior drift.

Data Generating Process (DGP): The model assumes a linear relationship $y_i = x_i'\beta + w_i'\theta + e_i$ , where $x_i$ are observed predictors and $w_i$ are unobserved latent features. The model is misspecified because the agent only observes $x_i$ .
Drift Mechanism: The authors explicitly model a change in the true parameters between the training phase ( $\beta_{is}, \theta_{is}$ ) and the testing phase ( $\beta_{oos}, \theta_{oos}$ ).
Regime: The analysis operates in the proportional regime where the number of observations $n$ , observed features $p$ , and unobserved features $q$ all tend to infinity such that $p/n \to c\phi$ and $(p+q)/n \to c$ .
Estimators: The study focuses on Ridge estimators (with regularization parameter $z$ ) and the ridgeless limit ( $z \to 0$ ).

Empirical Protocol

The empirical section replicates and extends the study by Kelly et al. (2024) using:

Data: 14 macroeconomic indicators from Goyal et al. (2024) and CRSP value-weighted returns.
Feature Expansion: Utilization of Random Fourier Features (RFFs) to artificially increase the dimensionality of the predictor space, creating an overparametrized setting ( $p=600, n=12$ , complexity $c=50$ ).
Validation: The study tests performance across multiple 15-year sub-periods and varying bandwidth parameters ( $\gamma$ ) of the RFF kernel.
Change Point Detection: The authors employ the changeforest algorithm (Londschien et al., 2023) to empirically verify the existence of structural breaks in the time-series of estimated betas.

Key Contributions

Theoretical Extension to Posterior Drift: The paper derives the expected return and variance of a market-timing strategy under posterior drift. It generalizes the results of Hastie et al. (2022) and Kelly et al. (2024) by introducing a "drift risk" term.
- Proposition 4: Shows that the asymptotic expected return is proportional to the scalar product $\langle \beta_{is}, \beta_{oos} \rangle$ . If the in-sample and out-of-sample loadings are orthogonal, the expected return collapses to zero.
- Proposition 8: Establishes that performance deteriorates if and only if the change in loadings ( $\beta_{oos} - \beta_{is}$ ) is negatively correlated with the in-sample loadings ( $\beta_{is}$ ).
Quantification of Risk-Adjusted Returns: The authors derive an analytical formula for the Sharpe ratio (Equation 12). They demonstrate that while complexity can increase raw returns in stable environments, the presence of posterior drift increases volatility and reduces the Sharpe ratio, particularly when the signal-to-noise ratio is weak (a common feature of financial data).
Empirical Evidence of Instability: The paper provides robust empirical evidence that the "virtue of complexity" is highly sensitive to the specific time period and the choice of bandwidth parameters.
- Sub-period Sensitivity: Performance varies drastically across 15-year windows. For instance, a strategy yielding +7.24% monthly returns in 2005–2019 dropped to +0.52% in 1975–1989 under the same bandwidth.
- Bandwidth Sensitivity: The optimal bandwidth is time-varying. While a bandwidth of $\gamma \approx 2$ is often cited as optimal, the authors show that this choice yields inconsistent results across different economic regimes.

Key Results

Performance Degradation: Theoretical and simulation results confirm that as the misalignment between $\beta_{is}$ and $\beta_{oos}$ increases, the out-of-sample expected return decreases. In the presence of strong posterior drift, the performance of overparametrized models can be worse than simpler models or even zero.
Volatility and Leverage: The volatility of the strategy is linearly increasing in the leverage induced by the model. In the presence of drift, the leverage required to capture weak signals leads to prohibitively high volatility, resulting in poor risk-adjusted returns (Sharpe ratios often below 0.3, compared to ~0.44 for the market index).
Counterfactual Analysis: By generating counterfactual returns where the DGP remains invariant (no posterior drift), the authors show that the potential value of the model is substantial. However, the actual realized returns are severely curtailed by the changing economic environment, confirming that the loss in performance is directly attributable to posterior drift.
Regularization Trade-off: Increasing the regularization parameter (bandwidth) reduces the uncertainty and variance of the strategy but simultaneously drives the average return toward zero, making the strategy less appealing from a return perspective.

Significance and Claims

The paper claims to offer a necessary cautionary perspective on the application of large, overparametrized linear models in finance. While the authors acknowledge that sophistication can bring value, they argue that the benefits reported in recent literature (e.g., Kelly et al., 2024) may be overstated because they rely on assumptions of distributional invariance that do not hold in practice.

The authors conclude that:

Stability is Paramount: Analysts relying on overparametrized models must pay particular attention to the stability and sensitivity of performance in backtests, as results are highly dependent on the specific sub-periods chosen.
Drift is a Critical Factor: The changing relationship between macroeconomic predictors and equity premiums (posterior drift) fundamentally undermines the efficacy of complex models in equity premium prediction.
Prudent Application: The findings recommend caution when resorting to large linear models for stock market predictions, suggesting that the "double descent" phenomenon may not translate to financial returns when regime changes are frequent and unpredictable.

The paper does not propose a new model to solve this issue but rather provides a theoretical and empirical framework to understand the limitations of current high-complexity approaches in non-stationary financial environments.