Sentiment-Aware Stock Price Prediction with Transformer and LLM-Generated Formulaic Alpha

This paper proposes a novel framework that leverages large language models to automatically generate interpretable, sentiment-aware formulaic alphas, which serve as enhanced features for Transformer-based models to significantly improve stock price prediction accuracy and transparency.

The Gist

Imagine you are trying to predict the weather for next week.

The Old Way (Traditional Finance):
In the past, financial experts (quantitative analysts) tried to predict stock prices by manually writing complex mathematical recipes. They would look at the data and say, "If the price went up yesterday and the volume was high, maybe it will go up today." They called these recipes "Alpha Formulas."

• The Problem: This was like trying to write a new weather forecast formula by hand every single day. It was slow, exhausting, and once everyone figured out your formula, it stopped working (a phenomenon called "alpha decay"). Also, these experts often missed the "human element"—like how a rumor about a competitor might make people panic and sell their stocks.

The New Way (This Paper's Solution):
This paper introduces a team-up between a Super-Intelligent Robot (LLM) and a Pattern-Recognizing Brain (Transformer) to predict stock prices.

Here is how it works, step-by-step:

1. The "News Detective" (Sentiment Analysis)

First, the system doesn't just look at the stock chart. It reads thousands of news articles.

• The Twist: It doesn't just read about the company you are interested in (e.g., Toyota). It also reads about Toyota's friends and enemies (e.g., Honda, Ford, Tesla, and even Apple).
• Why? If there is bad news about a major supplier or a rival, Toyota's stock might drop even if Toyota itself did nothing wrong. The system calculates a "mood score" (sentiment) for all these related companies.

2. The "Creative Chef" (The LLM)

This is the star of the show. The researchers gave a Large Language Model (like a very smart AI) a list of ingredients:

• Ingredients: Stock prices, technical math (like moving averages), and the "mood scores" of the related companies.
• The Task: "Chef AI, please invent 5 new, creative recipes (Alpha Formulas) that mix these ingredients to predict tomorrow's price."

Instead of just giving a number, the AI writes out the recipe in plain English math, like:

"Take Toyota's momentum, add a dash of how happy people are about Tesla, and subtract the fear about Ford."

The AI acts like a creative chef who can taste the market and invent new flavor combinations that humans might never think of.

3. The "Pattern Master" (The Transformer)

Once the AI Chef creates these new recipes, they aren't used to trade directly. Instead, they are fed into a Transformer model.

• Think of the Transformer as a master detective who is really good at spotting long-term patterns in a messy crime scene.
• It takes the "recipes" created by the AI Chef and uses them as clues to predict exactly what the stock price will be tomorrow.

The Results: Why is this cool?

The researchers tested this on five big companies (Apple, HSBC, Pepsi, Toyota, Tencent).

• Better Accuracy: The team-up of the "Creative Chef" (LLM) and the "Pattern Master" (Transformer) predicted prices much better than older methods (like standard computer models or human-written formulas).
• No More "Black Box": Usually, AI gives you an answer but won't tell you why. Here, because the LLM writes the formulas in readable math, we can actually see the logic. We can say, "Ah, the model predicted the price went up because it noticed positive news about a competitor!" This makes the prediction trustworthy.
• Beating the Competition: When they compared their AI-generated formulas against:
  1. Standard Computer Tools: The AI Chef was smarter.
  2. Human Experts: The AI Chef was much faster and more creative, avoiding the "old recipe" problem where human formulas stop working over time.

The Big Picture Analogy

Imagine you are trying to guess the winner of a horse race.

• Old Method: You look at the horse's past speed and weight.
• This Paper's Method: You ask a super-smart AI to read the newspaper, check the weather, see what the jockey's rival is saying, and then invent a brand-new way to calculate the odds. Then, you give that new calculation to a super-observant detective who spots the winner.

In short: This paper shows that by letting an AI invent its own "trading recipes" based on news and market mood, and then letting a deep-learning model use those recipes, we can predict stock prices more accurately and understand why the prediction was made. It's like giving a financial analyst a superpower to instantly invent new ways to see the future.

Technical Summary

Here is a detailed technical summary of the paper "Sentiment-Aware Stock Price Prediction with Transformer and LLM-Generated Formulaic Alpha."

1. Problem Statement

The paper addresses two primary challenges in quantitative finance and stock price prediction:

• Alpha Decay: Traditional "formulaic alphas" (mathematical expressions identifying market patterns) lose predictive power over time as markets adapt and more traders exploit them. Manually crafting new alphas is slow, expensive, and limited by human creativity.
• Feature Engineering Limitations: Standard deep learning models often struggle with noisy financial data when relying solely on raw features (Open, High, Low, Close, Volume) or static technical indicators. They often fail to capture complex, non-linear dependencies involving market sentiment and cross-company relationships.

The authors propose a framework to automate the generation of adaptive, interpretable alpha factors using Large Language Models (LLMs) to mitigate alpha decay and enhance prediction accuracy.

2. Methodology

The proposed framework integrates LLM-driven feature generation with Transformer-based sequence modeling. The process consists of four main stages:

A. Data Collection and Preprocessing

• Target Stocks: Five diverse companies (Apple, HSBC, Pepsi, Toyota, Tencent) covering technology, finance, consumer goods, automotive, and internet services.
• Data Sources: Historical OHLCV data (2016–2024) and daily news articles.
• Related Company Extraction: Using Named Entity Recognition (NER) on news articles, the authors identify "related companies" (competitors, suppliers, partners) based on co-occurrence frequency.
• Sentiment Analysis: Daily sentiment polarity scores are calculated for both target and related companies using VADER (a lexicon-based sentiment tool).
• Technical Indicators: Standard indicators (SMA, EMA, RSI, MACD, Bollinger Bands, OBV, Momentum) are computed using pandas-ta.

B. LLM-Generated Formulaic Alphas

• Model: The DeepSeek-R1-Distill-Llama-70b model is used via Groq.
• Prompting Strategy: A structured prompt instructs the LLM to generate 5 distinct formulaic alphas per company. The prompt requires the combination of:
  • Stock features (OHLCV).
  • Technical indicators.
  • Sentiment scores of the target and related companies.
• Output: The LLM produces human-readable mathematical formulas (e.g., combining momentum with sentiment divergence between a target and a competitor). These formulas are converted into numerical features.
• Normalization: Generated alphas are Min-Max scaled using training set parameters to prevent data leakage.

C. Prediction Model (Transformer)

• Architecture: A vanilla Encoder-Decoder Transformer is employed.
  • Encoder: Processes the 5 LLM-generated alphas over a 5-day sliding window. It utilizes 1D Convolutional Embedding, Temporal Embedding (day/week/month), and Positional Embedding.
  • Decoder: Takes the previous day's closing prices (5-day window) and attends to the encoder's output to predict the next day's closing price.
• Training: Optimized with Adam (learning rate $5 \times 10^{-5}$) over 50 epochs using Mean Squared Error (MSE) loss.

D. Baseline Comparisons

The study compares the Transformer against:

• Deep Learning: LSTM, TCN, Informer.
• Machine Learning: SVR, Random Forest, XGBoost.
• Alternative Alpha Sources:
  • Featuretools: Automated feature engineering via mathematical transformations.
  • Human-Defined: Alphas from the famous "101 Formulaic Alphas" paper (static, no sentiment).

3. Key Contributions

1. Automated Alpha Generation: Demonstrates that LLMs can effectively generate diverse, adaptive, and interpretable formulaic alphas that incorporate complex cross-entity sentiment relationships, addressing the issue of alpha decay.
2. Sentiment-Aware Cross-Company Analysis: Introduces a method to dynamically integrate sentiment from related companies (extracted via NER) into the prediction model, showing that external entity sentiment significantly impacts target stock prices.
3. Interpretability: Unlike "black-box" deep learning features, the LLM-generated alphas provide natural language reasoning and explicit mathematical formulas, enhancing transparency for financial decision-making.
4. Robustness Validation: Extensive statistical testing (Diebold-Mariano test, variance decomposition) confirms that performance improvements are due to model architecture and feature quality rather than random training variance.

4. Experimental Results

• Predictive Accuracy (MSE): The Transformer model with LLM-generated alphas achieved the lowest Mean Squared Error (MSE) across all five stocks, significantly outperforming LSTM, TCN, SVR, and tree-based models.
  • Example: For Toyota, the Transformer MSE was 0.0001, compared to 0.0015 for LSTM and 0.0048 for SVR.
• Statistical Significance: The Diebold-Mariano (DM) test confirmed that the Transformer's superiority over baselines (LSTM, TCN, SVR, RF, XGBoost) is statistically significant ($p < 0.01$). The difference between Transformer and Informer was not significant, suggesting attention mechanisms are the key driver.
• Impact of Sentiment: Including sentiment data (target + related companies) consistently reduced MSE and standard deviation compared to models without sentiment, proving the value of cross-entity sentiment integration.
• Comparison with Other Alpha Sources:
  • LLM vs. Featuretools: LLM-generated alphas yielded lower MSE than Featuretools-generated features.
  • LLM vs. Human-Defined: The performance gap was massive. Models using static "101 Formulaic Alphas" (Human-Defined) had MSEs roughly 100x higher (e.g., 0.19 vs 0.0003 for Apple) than those using LLM-generated alphas, highlighting the severe alpha decay of static formulas.
• Information Coefficient (IC): Interestingly, while LLM alphas showed weak correlation with returns (low IC), they exhibited strong predictive power for price levels (low MSE), suggesting LLMs are better suited for direct price forecasting than return ranking in this context.

5. Significance and Conclusion

This paper establishes a novel paradigm where LLMs act as automated quantitative researchers, continuously generating fresh, sentiment-aware trading signals to feed into deep learning models.

• Practical Value: The framework offers a scalable solution to alpha decay, reducing the reliance on manual feature engineering.
• Interpretability: The ability to explain why a signal was generated (via the LLM's reasoning process) bridges the gap between complex AI models and the need for transparency in finance.
• Future Directions: The authors suggest expanding the dataset to more stocks, testing different time horizons, and exploring other LLMs or prompt engineering strategies to further refine alpha generation.

In summary, the study proves that combining LLM-generated, sentiment-rich features with Transformer architectures creates a robust, state-of-the-art system for stock price prediction that outperforms traditional and static quantitative approaches.