Learning Stock Trading Policies via Barycenter-Based Adversarial Inverse Reinforcement Learning

Technical Summary: BRaG – Barycenter-Based Risk-Aware Adversarial Inverse Reinforcement Learning for Stock Trading

Problem Statement
Designing effective stock trading strategies via Reinforcement Learning (RL) faces significant hurdles, including delayed and noisy reward signals, poor exploration in high-dimensional environments, and the difficulty of enforcing explicit risk constraints. Traditional supervised learning struggles with label noise and translating predictions into profitable actions, while standard Deep RL (DRL) methods often suffer from unstable learning dynamics due to sparse rewards and rely on manually specified reward functions that may not generalize across assets or market regimes. Furthermore, existing Inverse Reinforcement Learning (IRL) approaches often depend on single-expert demonstrations, leading to overfitting and a lack of robustness in heterogeneous market conditions.

Methodology: The BRaG Framework
The authors propose BRaG (Barycenter-based Risk-aware Adversarial GAIL), a unified framework that integrates multi-expert imitation learning with explicit risk control. The methodology proceeds in three distinct stages:

1. Performance-Weighted Wasserstein Barycenter Aggregation:
   Instead of relying on a single expert or uniformly merging demonstrations, BRaG aggregates multiple heterogeneous expert strategies (e.g., momentum, mean-reversion, trend-following) into a single "pseudo-expert" distribution.

   • Weighting: Experts are assigned weights ($\alpha_k$) proportional to their risk-adjusted performance (Sharpe ratio), clipping negative performers to zero.
   • Barycenter Construction: The framework computes a performance-weighted sliced Wasserstein barycenter of the expert state-action trajectories. This approach preserves the geometric structure of expert behaviors and provides a robust consensus representation that reduces sensitivity to individual expert bias.
   • Feasibility Check: Since the sliced Wasserstein back-projection does not guarantee a unique inverse map, reconstructed samples are filtered through a feasibility check against available cash and holdings to ensure they represent valid trades.

2. Adversarial Imitation Learning (GAIL) Pretraining:
   The aggregated barycenter distribution serves as the target for Generative Adversarial Imitation Learning (GAIL).

   • A discriminator ($D_\phi$) is trained to distinguish between the barycenter pseudo-expert samples and the agent's generated trajectories.
   • The agent policy ($\pi_\theta$) is optimized to maximize the discriminator's output, effectively recovering an implicit reward signal ($r_\phi = -\log D_\phi$) without explicit reward specification.
   • This stage initializes the policy in a region of the policy space aligned with expert behavior, mitigating the unstable exploration typical of training from scratch in noisy financial environments.

3. Risk-Constrained Refinement via Control Barrier Functions (CBF):
   To ensure risk-aware decision-making, BRaG incorporates Control Barrier Functions to enforce maximum drawdown limits.

   • A barrier function $B(V_t)$ is defined based on the portfolio value and a maximum allowable drawdown ($\delta_{max}$).
   • Hard Constraint: At execution time, if a proposed action violates the safety condition (predicted drawdown exceeds the limit), the action is projected onto the nearest safe set.
   • Soft Regularization: A penalty term is added to the PPO objective during training to discourage repeated barrier violations.
   • Fine-tuning: After GAIL pretraining, the policy is refined using Proximal Policy Optimization (PPO) with true market rewards (portfolio returns), while the CBF mechanism remains active to constrain exploration.

Key Contributions
The paper identifies five primary contributions:

1. BRaG Framework: A novel barycenter-based adversarial IRL framework that aggregates heterogeneous expert strategies to construct a stable pseudo-expert representation for imitation learning.
2. Performance-Weighted Barycenter: The introduction of a performance-weighted Wasserstein barycenter to combine expert trajectories, reducing sensitivity to individual bias and enhancing robustness across diverse market conditions.
3. Adversarial Pretraining: A demonstration that adversarial imitation learning serves as an effective pretraining mechanism to address poor exploration in financial RL.
4. Explicit Risk Control: The integration of Control Barrier Functions to enforce drawdown constraints at action execution and regularize policy learning.
5. Empirical Validation: Extensive evaluation across four global equity markets (US DJI, UK FTSE 100, Indian Sensex, and Taiwanese TAIEX) showing consistent outperformance over classical rules and state-of-the-art DRL baselines.

Experimental Results
The framework was evaluated on daily stock data from 2010 to 2025, with training on 2010–2023 and testing on 2024–2025.

• Performance: BRaG achieved the highest cumulative and annualized returns across all four markets. For instance, on the DJI dataset, BRaG achieved a cumulative return of 53.977, outperforming the next best DRL baseline (DREB at 42.522) and the best technical strategy (MA Crossover at 25.625).
• Risk-Adjusted Metrics: BRaG consistently attained the highest Sharpe ratios (e.g., 1.472 on DJI vs. 1.405 for the next best) and maintained competitive or lower maximum drawdowns compared to high-performing DRL agents.
• Stability: The win ratio analysis indicated that BRaG achieved a higher proportion of profitable trades (e.g., 58.05% on DJI), suggesting performance driven by consistent decision-making rather than reliance on extreme outcomes.
• Ablation Studies:
  • Expert Aggregation: Using the full barycenter with Sharpe-based weighting outperformed using a single best expert, uniform sampling, or simple Sharpe-weighted sampling without barycenter construction.
  • Component Analysis: Removing GAIL pretraining caused the sharpest drop in performance, highlighting its role in initialization. Removing the CBF resulted in slightly higher raw returns but significantly degraded the Sharpe ratio and increased maximum drawdown, confirming the CBF's role in risk stability.

Significance and Claims
The paper claims that BRaG addresses the practical challenges of multi-expert imitation learning in financial trading by principled integration of expert aggregation, stable policy initialization, adaptive refinement, and risk-aware optimization. The authors assert that the framework successfully captures general trading principles rather than dataset-specific effects, as evidenced by consistent performance across markets with varying volatility and structural characteristics.

The work is modest in its claims, acknowledging that the current implementation relies on predefined rule-based experts and a simplified risk model. The authors suggest future work could incorporate richer expert sources, more accurate market dynamics, adaptive risk constraints, and extensions to higher-frequency trading. The core significance lies in demonstrating that combining diverse expert behaviors via a geometric barycenter, initializing via adversarial learning, and constraining actions via barrier functions yields a more robust and profitable trading agent than existing standalone approaches.