🤖 machine learning

CLaC@FinMMEval 2026 Task 3: Sentiment-Augmented Deep Reinforcement Learning for Active Trading -- An Alpha-Reward Approach

This paper presents a sentiment-augmented deep reinforcement learning system for the CLaC@FinMMEval 2026 Task 3, demonstrating that a DDPG agent utilizing LLaMA 3.2 news sentiment and alpha-based rewards significantly outperforms buy-and-hold strategies on TSLA and BTC, while also highlighting the challenges of generalizing policies from bull to bear market regimes.

Original authors: Andrei Neagu, Eeham Khan, Leila Kosseim

Published 2026-07-20

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Andrei Neagu, Eeham Khan, Leila Kosseim

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to play a very tricky video game: the stock market. In this game, the goal isn't just to win points; it's to beat the score of a player who never stops playing, never changes their strategy, and just holds onto their character the whole time (a strategy called "buy-and-hold"). The game is chaotic, filled with noisy sound effects, sudden plot twists, and a constant stream of news headlines that might change the rules. To win, you need a smart agent that can look at the screen, read the news, feel the rhythm of the game, and decide whether to move forward, stand still, or move backward. This is the world of Deep Reinforcement Learning (DRL). Think of DRL as a method where an AI learns by trial and error, getting a "high-five" (reward) when it makes a good move and a "frown" (penalty) when it messes up. The paper you're about to read explores how to build the ultimate trading robot for two very different characters: a tech company called Tesla (TSLA) and a digital coin called Bitcoin (BTC).

The researchers from Concordia University set out to build a trading system that doesn't just guess; it learns. They treated the stock market like a complex video game level where the player (the AI) has to make a daily decision: go Long (bet the price goes up), go Flat (sit on the sidelines), or go Short (bet the price goes down). To help the AI make these choices, they fed it three types of information: Technical Indicators (like speedometers and trend lines drawn from past prices), Calendar Cycles (knowing if it's Monday or the end of the month, because markets sometimes act differently on those days), and Sentiment Scores (a "mood ring" for the news, calculated by a super-smart AI that reads articles to see if people are happy or scared about the asset).

The team trained four different types of AI "brains" to play this game: Policy Gradient (PG), Proximal Policy Optimization (PPO), Deep Q-Learning (DQL), and Deep Deterministic Policy Gradient (DDPG). But here's the clever part: instead of just rewarding the AI for making money, they gave it a special "Alpha Reward." Imagine if you were racing a friend; instead of just rewarding you for running fast, you only get points if you run faster than your friend. This forced the AI to focus on beating the "buy-and-hold" baseline rather than just getting lucky because the whole market was going up. They also made the training sessions start at random times so the AI wouldn't just memorize the script of one specific game but would learn to handle any situation.

When they tested these robots on real data from 2025, the results were a mix of triumph and a harsh lesson about the future. For Tesla, the DDPG robot was the star, earning a massive 54.96% return, while the DQL robot came in a close second with 52.62%. Both crushed the "buy-and-hold" strategy, which only made 16.45%. The DDPG robot was particularly good at avoiding trouble, keeping its losses (drawdown) much lower than the others.

However, the Bitcoin test was a much tougher level. The market turned into a bear market, where prices crashed, and the "buy-and-hold" strategy lost a huge 34.27%. In this storm, the DDPG robot was the only one that managed to stay afloat, ending with a tiny but positive gain of 1.58%. The other robots, including the DQL which had looked amazing during training, struggled to adapt to the sudden shift from a rising market to a falling one.

The paper suggests that while these AI agents can learn to trade effectively, they are still sensitive to the "weather" of the market. The DDPG algorithm proved to be the most robust, handling both the sunny days of Tesla and the stormy days of Bitcoin better than the others. However, the researchers found a tricky gap: the robot that looked best during training (DQL) didn't always win in the real test, especially when the market regime changed from a bull market to a bear market. This tells us that while AI can learn to navigate the financial seas, it still needs to be careful not to get too used to calm waters, because the next wave might be a tsunami. Ultimately, the study shows that combining news sentiment with smart learning algorithms can help beat the market, but the "best" robot depends heavily on what kind of market it is facing.

Technical Summary: CLaC @ FinMMEval 2026 Task 3

Problem Definition

This paper addresses Task 3 of the CLEF 2026 FinMMEval Lab, a challenge requiring the development of an automated trading agent capable of making daily decisions (Long, Flat, or Short) for two distinct assets: Bitcoin (BTC) and Tesla (TSLA). The agent must synthesize multi-modal data, including historical market pricing, technical indicators, and daily news articles, to maximize performance metrics such as Cumulative Return (CR), Sharpe Ratio (SR), and Maximum Drawdown (MDD). The core difficulty lies in navigating complex, non-linear market dynamics and adapting to regime shifts (e.g., transitioning from bull to bear markets) while avoiding overfitting to historical training data.

Methodology

1. Markov Decision Process (MDP) Formulation

The trading problem is framed as a discrete-action MDP $(\mathcal{S}, \mathcal{A}, \mathcal{R}, \mathcal{P}, \gamma)$ :

State Space ( $\mathcal{S}$ ): A 15-dimensional vector comprising:
- Price Dynamics: Log returns.
- Agent State: Previous position ( $p_{t-1} \in \{-1, 0, 1\}$ ) to account for transaction costs.
- Calendar Encodings: Cyclical sine/cosine representations of Day-of-Week, Day-of-Month, and Month-of-Year to capture seasonality.
- Technical Indicators: Normalized EMA (10, 50), RSI (14), MACD, Bollinger Bands, and Log Volume Change.
- Sentiment: A daily sentiment score derived from news articles.
Action Space ( $\mathcal{A}$ ): Discrete set $\{-1 \text{ (short)}, 0 \text{ (flat)}, 1 \text{ (long)}\}$ .
Reward Function ( $\mathcal{R}$ ): Utilizes an Alpha Reward formulation (see Contributions) rather than raw portfolio return.
Transition Dynamics: Assumed to be exogenous (price-taker assumption), where agent actions do not influence market prices.

2. Data Processing and Feature Engineering

Data Sources: Historical price/volume data from Yahoo Finance (2010–2022 for TSLA; 2014–2022 for BTC) and news articles from the Financial News Multi-Source dataset.
Sentiment Analysis: Daily news sentiment is scored using LLaMA 3.2 1B Instruct in a zero-shot setting. Articles are truncated to the first 300 words (headline and lede) to focus on core signals. The model performs a single forward pass, extracting logits for "positive," "negative," and "neutral" tokens. The sentiment score is calculated as $P(\text{positive}) - P(\text{negative})$ , bounded in $[-1, 1]$ .
Temporal Alignment: To prevent look-ahead bias, articles published after market close are shifted to the next trading day for TSLA, while BTC (24/7) is anchored to UTC calendar dates.
Dataset Splits: Chronological splits were used: Training (pre-2023), Validation (2023–2024, a bull market period), and Testing (2025–2026, containing a bear market for BTC).

3. Deep Reinforcement Learning (DRL) Algorithms

Four algorithms were implemented and compared:

Policy Gradient (PG): LSTM-based actor with Monte-Carlo returns and entropy bonus.
Proximal Policy Optimization (PPO): LSTM-based actor-critic with clipped surrogate objective and GAE.
Deep Q-Learning (DQL): Feed-forward network with Double DQN updates and experience replay.
Deep Deterministic Policy Gradient (DDPG): Actor-critic with continuous action space discretized via thresholds ( $\pm 0.33$ ) and Ornstein-Uhlenbeck noise for exploration.

4. Training Strategy

Alpha Reward: The reward is defined as the excess return over the market: $r_t = \log(V_t/V_{t-1}) - \log(c_t/c_{t-1})$ . This incentivizes outperforming a buy-and-hold baseline rather than absolute profit.
Random Episode Starts: Training episodes begin at random days to prevent memorization of fixed sequences and improve generalization.
Hyperparameter Optimization: Conducted via Ray Tune with 180 trials per algorithm-asset pair.
Model Selection: Models were selected based on the Validation Sharpe Ratio, with early stopping applied.

Key Contributions

Multi-Modal State Representation: The paper demonstrates that combining technical indicators, cyclical calendar encodings, and LLM-derived sentiment scores creates an effective state signal for DRL agents.
Alpha Reward Formulation: The authors introduce and analyze an alpha reward that trains agents to maximize excess return over the market. They establish three theoretical properties:
- It equals the log-alpha of the strategy when summed over an episode.
- Under price-taker assumptions, it shares the same optimal policies as raw log-return rewards.
- It acts as a variance-reducing control variate, significantly reducing gradient estimator variance in high-volatility environments.
Comprehensive Algorithmic Comparison: A rigorous evaluation of PG, PPO, DQL, and DDPG with hyperparameter optimization reveals that DDPG offers the most robust performance across both assets, while highlighting the sensitivity of validation-based model selection to market regime shifts.

Results

Performance on TSLA (Test Period)

Baseline (Buy-and-Hold): 16.45% CR, SR 0.61.
DDPG: Achieved 54.96% CR and SR 1.44, with the lowest Maximum Drawdown (MDD) of 19.20%.
DQL: Achieved 52.62% CR and SR 1.38, but with a higher MDD (35.76%).
PG and PPO: Underperformed the baseline or significantly trailed the top models.

Performance on BTC (Test Period)

Baseline (Buy-and-Hold): -34.27% CR, SR -0.87 (severe bear market).
DDPG: The only model to achieve a positive return (1.58%) with an SR of 0.23 and MDD of 35.30%. It successfully mitigated market losses by shifting to flat/short positions during downturns.
PG: Reduced losses to -14.53% but remained negative.
DQL and PPO: Underperformed significantly, with DQL achieving -23.22% and PPO -33.80%.

Generalization Gap

A significant validation-to-test generalization gap was observed, particularly for BTC. The validation period (2023–2024) was a sustained bull market, leading models like DQL (which had the highest validation SR) to overfit to upward momentum. Consequently, DQL failed to generalize to the bear market test conditions, whereas DDPG's continuous action space and actor-critic architecture provided better adaptability to regime shifts.

Significance and Claims

The paper claims that the integration of LLM-based sentiment analysis with Deep Reinforcement Learning is a viable approach for algorithmic trading, provided that:

Reward Shaping: Using an alpha reward aligns the training objective with the competitive goal of beating the market, reducing overfitting to absolute price movements.
Algorithm Selection: DDPG demonstrates superior robustness in handling regime shifts compared to recurrent on-policy methods (PG, PPO) and purely off-policy value methods (DQL), particularly in volatile, bearish environments.
Limitations of Validation: The study highlights a critical limitation in financial ML: model selection based on a single validation window (especially one coinciding with a bull market) can lead to poor generalization in subsequent bear markets. The authors note that while DQL was selected for the live competition based on validation metrics, DDPG proved superior in the actual test period, underscoring the difficulty of adapting policies trained in bull markets to bear market conditions.

The authors conclude that while their system achieved substantial outperformance over the buy-and-hold baseline (especially on TSLA and in preserving capital on BTC), the observed generalization gap suggests a need for future work involving walk-forward cross-validation and hierarchical RL to better capture regime-dependent strategies.