Technical Summary: From Index to Equity: Pre-Training Transformers for Stock Return Prediction

Problem Statement

Predicting stock prices remains a formidable challenge due to the chaotic, non-stationary nature of financial markets, which are influenced by a complex interplay of economic, political, and social factors. Traditional forecasting models often struggle with the inherent volatility and distribution shifts between training and testing periods. While recent applications of Transformer architectures in financial forecasting have shown promise, they typically require massive datasets to learn contextual relationships effectively. This research addresses the specific challenge of improving prediction accuracy for individual stocks on the Toronto Stock Exchange (TSX) by leveraging transfer learning strategies. The core hypothesis is that pre-training on aggregated market data (the TSX Index) can help models learn generalized patterns before fine-tuning on the more volatile and noisy data of individual equities.

Methodology

Data and Feature Engineering

The study utilized a dataset of 1,853 TSX tickers spanning from March 17, 1980, to July 4, 2023, comprising over 4.9 million daily records. Data was sourced via the Yahoo Finance API.

• Target Variables: The study addressed two tasks:
  1. Classification: Predicting the direction of intra-day returns (positive vs. negative).
  2. Regression: Predicting the specific value of intra-day returns.
• Features: The input features included historical lag data and a suite of technical indicators widely used in the financial industry, such as Stochastic %K, Price Rate of Change (ROC), Relative Strength Index (RSI), Accumulation Distribution Oscillator (AccDO), Moving Average Convergence-Divergence (MACD), and various Exponential Moving Averages (EMA).
• Input Structure: For deep learning models (Transformers and LSTMs), inputs were structured as sequences of 10 days with 15 features per day. For XGBoost, these were flattened into a 150-feature vector.
• Preprocessing: Data underwent min-max scaling based on training set statistics. Outliers in technical indicators (caused by division-by-zero errors) were replaced with neutral values (50 for %K, 0 for AccDO), and extreme intra-day returns (>2) were clipped to mitigate scaling distortion. The first year of data for each stock was excluded to reduce unpredictable initial behavior.

Model Architecture and Training Strategy

The research compared several architectures:

1. Transformer: A model consisting of four encoder blocks followed by three fully connected dense layers (128, 64, and 32 nodes).
2. LSTM: A benchmark model replacing Transformer encoder blocks with four LSTM layers.
3. XGBoost: A gradient boosting model trained with specific hyperparameters (100 estimators, learning rate 0.03, max depth 9).
4. Ensemble: An average of predictions from the fine-tuned deep learning and XGBoost models.

The Pre-Training Approach:
The central methodological contribution was a two-stage training process:

1. Pre-Training: Models were first trained on the TSX Index data to predict the direction of intra-day returns. This stage utilized aggregated market data to learn generalized market dynamics.
2. Fine-Tuning: The pre-trained weights were transferred to models trained on individual stock data. For the regression task, the classification models were further fine-tuned by replacing the output layer to predict continuous return values.

Experimental Setup

• Data Split: Training data covered 2010–2022; validation data was from 2022; and the hold-out test set covered 2023 up to December 1, 2023.
• Evaluation Metrics:
  • Classification: Accuracy, Precision, F1 Score, and Binary Cross-Entropy Loss.
  • Regression: Mean Squared Error (MSE).
  • Portfolio Performance: Average daily returns from a strategy buying the top 5 and shorting the bottom 5 predicted stocks daily.
• Application: A Streamlit-based application was developed to visualize real-time predictions and technical indicators, demonstrating the practical deployment of the models.

Key Results

Classification Performance

Pre-training on the TSX Index significantly improved the Transformer's ability to generalize to individual stocks.

• Loss Reduction: Pre-training reduced the binary cross-entropy loss for individual stock prediction from 0.6942 (random initialization) to 0.6404.
• Accuracy vs. F1: While the baseline Transformer achieved the highest test accuracy (0.6667), it failed to predict the minority class (positive returns) effectively, resulting in a precision and F1 score of 0.0000. The XGBoost model achieved the strongest overall classification performance with the highest F1 scores (Test F1: 0.5675) and precision.
• Comparison: The fine-tuned Transformer outperformed the fine-tuned LSTM in classification metrics but was surpassed by XGBoost.

Regression Performance

In the regression task, deep learning models with pre-training demonstrated superior accuracy in minimizing error.

• MSE: The fine-tuned Transformer achieved the lowest test MSE (5.682e-9), slightly outperforming the fine-tuned LSTM (5.683e-9). Both deep learning models significantly outperformed the XGBoost (9.923e-9) and Ensemble (6.147e-9) models in terms of regression accuracy.
• Portfolio Returns: Despite the superior regression accuracy of the Transformer, the Ensemble model generated the highest average daily portfolio return (0.0069), followed by XGBoost (0.0059). The Transformer and LSTM models yielded lower returns (0.0003 and 0.0002, respectively).
• Analysis: The authors note that models with lower MSE (Transformers/LSTMs) tended to select stocks with predicted returns closer to zero, whereas the XGBoost and Ensemble models selected stocks with greater return variation, leading to higher realized profits in this specific backtesting scenario.

Significance and Claims

The paper modestly claims that while the absolute returns achieved were positive but relatively modest, the study provides evidence for the efficacy of transfer learning in financial forecasting.

• Pre-Training Value: The primary finding is that pre-training on aggregated market index data helps models learn generalized patterns, smoothing the chaotic fluctuations inherent in individual stock data. This approach successfully reduced loss and improved regression accuracy compared to random initialization.
• Architecture Trade-offs: The research highlights a distinction between minimizing prediction error (MSE) and maximizing trading profitability. While Transformers excelled at regression accuracy, ensemble methods and tree-based models (XGBoost) proved more effective at capturing the variance necessary for profitable trading strategies in this specific context.
• Practical Application: The development of a functional application demonstrates the feasibility of integrating these models into real-time trading support systems.

The authors conclude that while the inherent difficulty of stock market data limits performance, the proposed methodology offers a promising direction for improving individual stock predictions. Future work is proposed to increase model capacity, incorporate broader global indicators, and integrate risk management strategies to filter out low-predictability stocks.