Technical Summary: GIFT (Guided Interface Design for Financial Trading)

Problem Definition

Financial portfolio trading is formulated as a sequential decision-making problem where an agent rebalances assets to balance return, risk, and transaction costs. While Deep Reinforcement Learning (DRL), particularly Proximal Policy Optimization (PPO), is a dominant paradigm for this task, its effectiveness is critically dependent on the state-reward interface provided to the agent.

Current approaches face two primary limitations:

1. Under-specified States: Raw OHLCV (Open, High, Low, Close, Volume) observations often leave complex financial structures (e.g., momentum, volatility, downside risk, liquidity, mean reversion) implicit. The policy network must simultaneously learn to recover these structures from noisy sequences and learn the trading policy, complicating training.
2. Weak Reward Signals: Short-horizon portfolio returns are noisy and influenced by market shocks and transaction costs. They often fail to provide immediate positive feedback for actions that reduce long-term risk (e.g., drawdown control), leading to weak guidance for long-term return-risk objectives.

While Large Language Models (LLMs) offer a mechanism to inject domain knowledge into state and reward design, unguided, open-ended LLM generation is risky in financial contexts. Financial time series are non-stationary and noisy; unconstrained LLMs may encode spurious correlations, follow short-term fluctuations, or generate designs that encourage excessive trading.

Methodology: GIFT Framework

The authors propose GIFT (Guided Interface Design for Financial Trading), an LLM-guided framework designed to construct and refine the state-reward interface for PPO-based financial reinforcement learning. Crucially, GIFT does not use the LLM as a direct trading agent; instead, the LLM acts as a constrained interface designer, while PPO remains responsible for learning the trading policy.

GIFT operates in an offline design phase before evaluation and consists of three core components:

1. Factor-guided State Enhancement (FSE)

FSE constrains the LLM to generate executable state features based on a predefined library of financial-factor primitives ($\mathcal{B}$).

• Mechanism: Instead of arbitrary feature construction, the LLM selects, transforms, and composes a compact set of factor channels from primitives covering momentum, volatility, downside risk, liquidity, mean reversion, and trend strength.
• Output: The LLM generates functions that augment the raw OHLCV encoding with these financial factors, creating an enhanced state representation $\tilde{s}_t$ for the PPO agent.

2. Risk-rule-guided Reward Shaping (RRS)

RRS addresses the noise and misalignment of short-horizon returns by augmenting the base portfolio reward with financial risk-control signals.

• Mechanism: The LLM generates an intrinsic reward term based on the enhanced state and selects a small subset of risk rules from a library ($\mathcal{R}$). These rules include diversification control, turnover penalties, drawdown control, volatility scaling, and momentum alignment.
• Output: A shaped reward function $R_t$ that combines the base return, an LLM-generated intrinsic reward, and selected risk-rule penalties.

3. Diagnostic-guided Refinement (DGR)

DGR closes the loop between generated interfaces and PPO training outcomes to iteratively improve the design.

• Mechanism: For each candidate interface, GIFT trains PPO on a historical design window and computes diagnostics. These diagnostics measure:
  • State Informativeness: Using channel Information Coefficient (IC) and critic attribution.
  • Reward Stability: Analyzing reward trends, autocorrelation, and signal-to-noise ratio.
  • Portfolio Behavior: Evaluating drawdown, return, and risk-adjusted performance.
• Feedback: The diagnostic summary is compressed into a feedback memory ($H_m$) and fed back to the LLM to guide the next round of refinement (e.g., removing low-IC factors or strengthening risk rules).

Evaluation Protocol

After $M$ refinement rounds, GIFT selects the best executable interface $(F^\star, R^\star)$ based on rollout performance on the historical design window. This interface is frozen before evaluation. During the test phase, a newly initialized PPO agent uses this fixed interface to adapt and evaluate on the second half of the test window. No LLM queries, prompt updates, or interface modifications occur during evaluation, ensuring no data leakage.

Key Contributions

1. Novel Perspective: The paper introduces the first LLM-guided state-reward interface design perspective for financial RL, positioning the LLM as a constrained knowledge-guided designer rather than a direct trading agent.
2. GIFT Framework: A PPO-based framework integrating Factor-guided State Enhancement, Risk-rule-guided Reward Shaping, and Diagnostic-guided Refinement to generate and revise executable interfaces.
3. Empirical Validation: Comprehensive rolling-window experiments across diverse market regimes (bull, bear, mixed, V-shaped) and portfolio sectors demonstrate that GIFT improves learning-signal quality and out-of-sample risk-adjusted performance compared to baselines.

Experimental Results

The authors evaluated GIFT on S&P 500 constituents across six rolling test windows (2020–2023) and six portfolio panels (Technology, Healthcare, Energy, Industrials, Light Mix, Heavy Mix).

• Performance vs. Pure PPO: GIFT-enhanced PPO outperformed the Pure PPO baseline (which uses raw OHLCV and short-horizon returns) in 160 out of 180 metric-level comparisons. In 35 of 36 panel-window cells, GIFT achieved wins on at least 4 of 5 metrics (Return, Sharpe, Sortino, MDD, Calmar).
• Risk-Adjusted Gains: Improvements were particularly pronounced in stressed market windows (e.g., rate hikes, geopolitical shocks), where GIFT reduced drawdowns and improved Sharpe/Sortino/Calmar ratios while maintaining returns.
• Training Signal Quality: Diagnostic experiments showed that GIFT-generated interfaces provided PPO with clearer training feedback, characterized by higher reward trend slopes, stronger autocorrelation, and better signal-to-noise ratios compared to free-form LLM generation or fixed designs.
• Ablation Studies: Removing the Diagnostic-guided Refinement (DGR) component caused the largest performance degradation, highlighting the importance of iterative feedback. Removing FSE or RRS also weakened performance, confirming the necessity of both state enhancement and reward shaping.
• Robustness: The framework demonstrated transferability to other RL backbones (A2C) and robustness across different random seeds and search budgets.

Significance and Claims

The paper claims that GIFT supports a more controlled role for LLMs in financial reinforcement learning: designing the learning interface while leaving trading decisions to the RL policy.

The authors assert that the performance gains stem primarily from guided offline interface design rather than fixed feature engineering or unconstrained LLM generation. By constraining the LLM with financial factor and risk-rule libraries and refining designs via PPO diagnostics, GIFT creates state-reward interfaces that are more informative and stable across non-stationary market regimes.

The work emphasizes that this approach avoids the pitfalls of open-ended LLM trading (such as hallucinated rationales or look-ahead bias) by restricting the LLM to the structural design layer. The authors note limitations, including reliance on historical daily data and the exclusion of real-world frictions like liquidity constraints and execution delays, suggesting these as areas for future study.