💰 quantitative finance

Cross-Sectional Heterogeneity in LSTM Networks for Financial Time Series

This paper proposes an enhanced LSTM architecture that integrates macro-financial covariates and learnable sector embeddings to effectively capture cross-sectional heterogeneity in S&P 500 returns, demonstrating superior predictive performance and interpretability over standard benchmarks through a strategy driven by short-term reversal and industry momentum factors.

Original authors: Julius Döbelt

Published 2026-08-07

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Julius Döbelt

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to predict the weather, but instead of looking at clouds, you are looking at the stock market. This is the world of financial time series prediction, a field where scientists try to guess if a stock's price will go up or down tomorrow based on its history. It's a notoriously tricky game. Think of the stock market as a giant, noisy ocean. The "signal" is the actual wind direction telling you where the waves are going, but the "noise" is the splashing, the seagulls screaming, and the random ripples that make it hard to see the real pattern. Because the market is so efficient—meaning everyone knows the news instantly—finding a reliable pattern is like trying to hear a whisper in a hurricane.

To tackle this, researchers use machine learning, specifically a type of computer brain called an LSTM (Long Short-Term Memory). You can think of an LSTM as a super-organized librarian who reads the history of a stock's price day by day, remembering the important parts and forgetting the noise, to make a guess about the future. For a long time, these librarians were trained to treat every single stock as if it were the same person, just wearing a different hat. They looked at the history of Apple and the history of a utility company exactly the same way. But in the real world, a tech company and a power company react to news very differently. This paper asks a simple question: What if we taught our librarian to recognize that stocks belong to different "neighborhoods" (sectors) and that these neighborhoods have their own unique personalities?

The Great Stock Market Sorting Hat

In this study, the author, Julius Döbelt, decides to upgrade the standard LSTM librarian. The old version was good, but it was a bit clumsy because it treated a high-flying technology stock exactly the same as a steady utility stock. Döbelt introduces a new trick: Sector Embeddings.

Imagine you are at a massive party with hundreds of guests. The old librarian would look at every guest and try to guess who will dance next based only on how they moved in the last hour. But Döbelt's new librarian gets a special "sorting hat." Before the librarian even looks at the dancing, the hat instantly tags each guest with their group: "Tech," "Healthcare," "Energy," and so on. The librarian learns that "Tech" guests tend to dance in a specific rhythm, while "Energy" guests have a totally different style. By giving the computer these "sector tags" as a special input, the model can finally understand that a stock's past performance doesn't just depend on its own history, but also on the vibe of its entire neighborhood.

The Experiment: A 30-Year Time Travel

To test this idea, Döbelt built a time machine. He took data from the S&P 500 (the 500 biggest companies in the US) from 1995 to 2024. He didn't just look at one year; he ran a simulation where he trained the model on three years of history and then tested it on the very next year, repeating this process 27 times. This is like training a student for three years and then testing them on a new school year, over and over again, to see if they can really learn or if they just memorized the answers.

He compared his new "Sector LSTM" against three other competitors:

The Basic LSTM: The old librarian who ignores neighborhoods.
The Random Forest: A different kind of computer brain that makes decisions by asking many small questions (like a committee of experts).
The Buy-and-Hold Strategy: The investor who just buys the whole market and waits.

The Results: The Neighborhood Trick Wins (Mostly)

The results were clear and exciting. The new Sector LSTM consistently outperformed the others. When the model used the sector tags, it made more correct predictions about which stocks would beat the average and which would lose.

The Winner: The Sector LSTM had the best balance of making money and taking low risks. It achieved a Sharpe ratio (a score that measures how much profit you get for the risk you take) of 0.69, which was significantly better than the basic LSTM (0.39) and the Random Forest (0.53).
The Loser: Interestingly, adding extra "macro-financial" data—like oil prices, gold futures, and interest rates—didn't help. In fact, the model that tried to use all that extra economic news performed the worst. It seems that for this specific daily prediction task, knowing the neighborhood (sector) was far more important than knowing the global weather report.

The "Black Box" Cracked Open

One of the biggest complaints about AI is that it's a "black box"—we know it works, but we don't know why. Döbelt decided to peek inside the box. He found that the model had secretly learned two famous rules of the stock market without anyone telling it:

Industry Momentum: The model learned that if a whole neighborhood (like Technology) has been doing well over the last few months, it tends to keep doing well for a while. The "sector tags" acted like a momentum booster, pushing the model to bet on the winning neighborhoods.
Short-Term Reversal: The model also learned that if a single stock had a huge jump yesterday, it's likely to take a small step back today. It's like a rubber band snapping back.

However, the paper also found a catch. This "neighborhood momentum" trick works great in stable times, like in 2007, where the model correctly bet on the winners. But in times of chaos, like right after the 2000 dot-com crash or the 2020 pandemic, the trick backfired. When the market panicked, the "losers" of the previous year suddenly became the "winners," and the model's reliance on neighborhood trends caused it to bet on the wrong horses. This suggests that while the model is smart, it isn't magic; it still struggles when the market rules suddenly change.

The Bottom Line

This paper suggests that teaching AI to recognize that "a stock is more than just a number; it's part of a team" is a powerful way to improve predictions. By adding simple "sector tags," the model became better at navigating the noisy stock market. But the author also warns us that the days of easy, massive profits are likely over. The model's success is modest, and it depends heavily on the market behaving normally. It's a solid step forward, proving that even in a noisy ocean, knowing which boat you're on makes a big difference.

Technical Summary: Cross-Sectional Heterogeneity in LSTM Networks for Financial Time Series

Problem Statement
Predicting financial asset returns is a central yet unresolved challenge in empirical finance, characterized by a low signal-to-noise ratio and the semi-strong form of market efficiency. While Long Short-Term Memory (LSTM) networks have demonstrated promise in capturing temporal dependencies, standard architectures often fail to account for cross-sectional heterogeneity. Traditional approaches, such as the single-layer LSTM applied to pooled cross-sectional data by Fischer and Krauss (2018), implicitly assume that all assets are drawn from a common return-generating process. This assumption contradicts established evidence that industry membership and firm characteristics induce systematic differences in return dynamics. Furthermore, the "black-box" nature of deep learning models limits their interpretability, making it difficult to validate whether learned representations align with known economic factors like momentum or reversal.

Methodology
The paper proposes a novel architectural extension to the basic LSTM model designed to improve predictive accuracy and interpretability while mitigating overfitting.

Data and Validation Protocol: The study utilizes a survivorship-bias-free universe of S&P 500 constituents from January 1995 to December 2024. The evaluation follows a rolling-window protocol with 27 non-overlapping study periods. Each period consists of a 3-year training set and a 1-year out-of-sample test set. To prevent data leakage, the input sequence is limited to 60 lagged days (compared to 240 in prior work), ensuring no overlap between the training and test sequences.
Forecasting Task: The problem is framed as a binary classification task: predicting whether a stock's return on day $t+1$ will be above or below the cross-sectional median return of the S&P 500. This transforms the regression problem into a directional signal suitable for a long-short portfolio strategy.
Architectural Extensions:
1. Learnable Sector Embeddings: The core innovation is the integration of learnable TRBC (The Refinitiv Business Classification) sector embeddings. Instead of treating all stocks identically, the model maps each stock to a sector-specific vector ( $e_c$ ) in a continuous latent space. This vector is concatenated with the LSTM's final hidden state before the classification layer. This allows the model to learn a low-dimensional representation of sector-level structures, effectively incorporating a cross-sectional intercept that shifts classification probabilities based on industry group.
2. Macro-Financial Covariates: The model is extended to include sequences of macro-financial variables (oil returns, VIX, gold futures, and Treasury yield spreads) alongside stock returns to capture broader economic signals.
3. Regularization: To address overfitting in the noisy financial environment, the model employs label smoothing (softening targets to 0.05/0.95), gradient clipping, weight decay, early stopping, and both input and recurrent dropout.
Benchmarks: Performance is evaluated against a basic LSTM, a Random Forest model (using cumulative returns as features), and a traditional market buy-and-hold strategy.

Key Results

Performance Metrics: The Sector LSTM consistently outperforms all benchmarks. It achieves the highest daily returns and classification accuracy. In terms of risk-adjusted performance, the Sector LSTM yields a significantly higher annualized Sharpe ratio (0.69) compared to the Base LSTM (0.39) and the Covariate LSTM (0.27). While the Random Forest performs competitively on the Sharpe ratio, the Sector LSTM significantly outperforms it in predictive accuracy (p < 0.01).
Ineffectiveness of Covariates: The inclusion of macro-financial covariates did not improve directional prediction; the Covariate LSTM performed the worst among the deep learning variants, suggesting that the additional economic signals did not provide exploitable content for this specific task.
Long-Term Validation: Extending the evaluation through 2024 confirms the deterioration of LSTM-based statistical arbitrage profits documented by Fischer and Krauss (2018). High returns and Sharpe ratios were concentrated prior to 2008 (with a brief exception in 2020). Post-2010, no year produced strategy returns approaching those of the late 1990s, supporting the literature on diminishing returns to quantitative trading.
Interpretability and Signal Decomposition: The paper successfully decomposes the predictive signal into two endogenous components:
1. Industry Momentum: The learned sector embeddings function as an implicit industry momentum factor. The model assigns positive biases to sectors that were recent winners during the training window. This signal is regime-dependent: it drives outperformance in stable years (e.g., 2007) but leads to significant losses during momentum crashes (e.g., 2003, 2021–2024) when sector rotations occur.
2. Short-Term Reversal: The temporal component of the LSTM (the hidden state) captures the well-documented short-horizon reversal effect, where recent winners tend to underperform and recent losers tend to outperform in the subsequent week.

Significance and Claims
The paper claims three primary contributions to the literature:

Architectural Innovation: It introduces the first application of continuous sector embeddings within a recurrent network for cross-sectional return prediction. This explicitly models cross-sectional heterogeneity, allowing the model to adapt to varying industry dynamics rather than treating assets as exchangeable.
Long-Term Validation: By extending the sample through 2024, the paper provides robust evidence that the decline in LSTM profitability after 2010 is a durable shift in market efficiency rather than a transient phenomenon.
Explainability: The study demonstrates that deep learning models can recover known asset-pricing regularities (industry momentum and short-term reversal) endogenously. By visualizing the latent space and analyzing the weights, the paper shows that the "black box" can be opened to reveal economically meaningful representations, specifically that sector embeddings act as a learnable industry momentum factor.

The author concludes that while incorporating cross-sectional heterogeneity via learnable embeddings is a parsimonious and effective way to improve LSTM-based prediction, the incremental signal is modest, highly conditional on market regimes, and has weakened in the last decade.

The Great Stock Market Sorting Hat

The Experiment: A 30-Year Time Travel

The Results: The Neighborhood Trick Wins (Mostly)

The "Black Box" Cracked Open

The Bottom Line

More like this