When Valid Signals Fail: Regime Boundaries Between LLM Features and RL Trading Policies

1. Problem Statement

The paper addresses a critical gap in applying Large Language Models (LLMs) to financial decision-making via Reinforcement Learning (RL). While modular pipelines often use LLMs as sentiment classifiers, they suffer from objective mismatch: LLMs are typically optimized for NLP metrics (e.g., accuracy, BLEU) rather than downstream financial utility (e.g., portfolio returns).

Furthermore, there is an unproven assumption that valid intermediate representations (features with high predictive correlation to returns) automatically translate into robust downstream policy performance. The authors investigate whether LLM-derived features improve RL trading agents, particularly under distribution shifts caused by macroeconomic shocks, and whether the "feature validity" gap leads to policy failure.

2. Methodology

System Architecture

The authors built a modular, stateless pipeline separating the LLM from the trading agent:

• Data Ingestion: A deterministic Go-based pipeline aggregates daily news, RSS feeds, and SEC filings (Form 4, 8-K) for specific tickers. It enforces strict daily information boundaries to prevent look-ahead bias.
• LLM Feature Extractor: A frozen LLM (Qwen3-235B) acts as a zero-shot feature extractor. It processes daily text bundles and outputs a fixed-length, 9-dimensional numerical vector:
  • LLM Features (4 dims): Sentiment, Impact, Conflicting Signals, News Novelty.
  • Macro/Regime Features (5 dims): VIX, 10Y Treasury, Credit Spread, plus two LLM-inferred regime flags.
• RL Agent: A Proximal Policy Optimization (PPO) agent (using FinRL) consumes the composite state vector (OHLCV, technical indicators, LLM features, macro features) to make trading decisions (Buy/Hold/Sell) on a 21-ticker universe.

Prompt Optimization Loop

Instead of standard prompt engineering, the authors introduced an automated mutation–evaluation–selection loop:

• Objective: The LLM extraction prompt is treated as a discrete hyperparameter.
• Metric: Optimization targets the Information Coefficient (IC) (Spearman rank correlation between predicted and realized returns), not NLP losses.
• Process: A meta-optimizer (Claude API) suggests structural mutations (e.g., adding few-shot examples, redefining impact logic). Features are extracted, and IC gates are evaluated.
• Result: The "Few-Shot" mutation (mut2) was selected as the optimal prompt, significantly improving IC from -0.024 (baseline) to +0.104.

Experimental Design

• Training: 2023–2024 data.
• Validation (H1 2025): A volatile period characterized by a "Tariff Shock" (macroeconomic regime shift).
• Held-Out Test (H2 2025): A calmer, stable market regime.
• Ablation Study: Four configurations were tested:
  1. Baseline (OHLCV + Tech only).
  2. LLM-only (News features).
  3. Macro-only (Economic indicators).
  4. LLM + Macro (Full model).

3. Key Contributions

1. Prompt-as-Hyperparameter Optimization:
   The paper demonstrates that treating the extraction prompt as a tunable hyperparameter optimized against financial IC (rather than text quality) yields genuinely predictive features. The optimized prompt achieved an IC IR of +0.104, proving that LLMs can extract continuous numerical signals relevant to trading.

2. The Feature-to-Policy Gap:
   The study reveals a critical disconnect: Valid features do not guarantee policy improvement. While the LLM features had high predictive power in isolation, they failed to improve the RL agent's performance during a macroeconomic shock. In fact, adding LLM features during the volatile H1 period degraded performance compared to a price-only baseline.

3. Regime-Dependent Robustness:
   The utility of LLM features is strictly regime-dependent.

   • High Volatility (H1): LLM features act as noise, causing the agent to underperform.
   • Low Volatility (H2): LLM features recover value, helping the agent outperform the baseline.
   • Macro Variables: Macroeconomic indicators (VIX, Treasury yields) proved to be the most robust driver of policy stability, acting as a "regime brake" that prevents over-trading during shocks.

4. Results

• Feature Validity: The optimized prompt produced features with significant predictive power. Conflicting Signals (IC=0.233) and Impact (IC=0.177) showed the strongest correlation with 5-day forward returns. The signal decayed after 10 days, consistent with news latency constraints.
• H1 2025 (Shock Regime):
  • Baseline Sharpe: 0.010
  • LLM+Macro Sharpe: -0.267 (Worse than baseline).
  • LLM-Only Sharpe: -0.411 (Worst performance).
  • Conclusion: During macro shocks, idiosyncratic news signals add noise, and the RL agent fails to distinguish signal from regime-driven volatility.
• H2 2025 (Calm Regime):
  • Baseline Sharpe: 0.809
  • LLM+Macro Sharpe: 1.038 (Significant improvement).
  • Macro-Only Sharpe: 1.099 (Best performance).
  • Conclusion: In stable markets, LLM features successfully capture alpha, but macro variables remain the primary driver of robustness.
• Feature Importance: A gradient-boosted tree analysis showed that despite having zero cross-sectional IC (as they are constant across tickers), macro features accounted for 58% of the model's decision weight, highlighting their role in time-serial risk management.

5. Significance and Implications

• Beyond NLP Metrics: The paper argues that optimizing LLMs for NLP tasks is insufficient for finance. Direct optimization against downstream financial metrics (like IC) is necessary to extract usable signals.
• Distribution Shift Awareness: The findings parallel challenges in transfer learning. A model trained on valid features may fail catastrophically if the data distribution shifts (e.g., from idiosyncratic news-driven markets to macro-driven markets).
• The "Regime Brake" Concept: Macro indicators are essential not for picking specific stocks, but for conditioning the agent's risk exposure. They allow the agent to "stand down" when the environment becomes unpredictable.
• Evaluation Protocol: The authors propose that any pipeline feeding LLM representations into downstream learners must undergo multi-regime out-of-sample testing, specifically including stress periods, to ensure policy robustness.

In summary, the paper provides a rigorous framework for integrating LLMs into RL trading but cautions that feature validity is necessary but not sufficient for policy success. The success of such systems depends heavily on the stability of the market regime and the inclusion of macroeconomic state variables to manage distribution shifts.