Decomposable Reward Modeling and Realistic Environment Design for Reinforcement Learning-Based Forex Trading

1. Problem Statement

Applying Reinforcement Learning (RL) to Foreign Exchange (Forex) trading faces three critical structural gaps that limit real-world applicability:

• Low Environment Fidelity: Many existing simulators ignore critical market mechanics such as transaction costs (spread, slippage, commission), rollover financing, margin constraints, and liquidation rules. They often suffer from "lookahead leakage," where agents inadvertently use future data to make current decisions.
• Reward Design Opacity: Most studies rely on a single scalar reward (e.g., Profit & Loss or Sharpe ratio). This "monolithic" approach makes it difficult to diagnose which specific signals drive learning, hindering the ability to distinguish between improved trade timing and excessive trading activity.
• Inadequate Action Spaces: Existing formulations often use coarse discrete actions (Buy/Sell/Hold) or unconstrained continuous policies. These fail to capture practical trading primitives like pyramiding (scaling into winning positions), martingale scaling (averaging down), partial reduction, or explicit reversals, and often fail to enforce hard margin constraints.

2. Methodology

The authors propose a modular, open-source framework designed around three core pillars: Economic Fidelity, Temporal Causality, and Auditability.

A. High-Fidelity Execution Environment

• Anti-Lookahead Semantics: The environment enforces a strict causal timeline:
  1. Observe: $t_{close}$ (Close price of current bar).
  2. Decide: Agent selects action.
  3. Execute: Order fills at $t+1_{open}$ (Open price of next bar).
  4. Mark-to-Market: Positions are valued at $t+1_{close}$.
• Realistic Frictions: The simulator incorporates spread, commission, slippage, rollover financing (including triple-swap on Wednesdays), and maintenance margin rules.
• Risk Controls: Includes forced liquidation if equity drops below a threshold (25%) and strict leverage limits (30x).

B. Decomposable Reward Architecture

Instead of a single reward, the system uses an 11-component reward function with fixed, pre-specified weights. Components include:

1. Profit: Post-cost equity return.
2. Holding Incentive: Bonus for profitable holding.
3. Volatility Penalty: Discourages unstable equity trajectories.
4. Drawdown Penalty: Penalizes downside risk.
5. Transaction Burden: Normalized penalty for costs.
6. Overtrading Penalty: Suppresses unnecessary frequency.
7. Pyramiding Penalty: Constrain aggressive scaling on winners.
8. Martingale Penalty: Strongly penalize loss-amplifying scaling.
9. Margin Utilization Penalty: Discourages extreme leverage.
10. Liquidation Penalty: Catastrophic penalty for forced liquidation.
11. Constraint Violation Penalty: Penalizes illegal action proposals.

• Mechanism: Each component is logged per step, allowing for systematic ablation studies to determine the contribution of each term.

C. Structured Action Space with Legal Masking

• 10-Action Discrete Interface: Includes HOLD, OPEN_LONG/SHORT, PYRAMID_LONG/SHORT, MARTINGALE_LONG/SHORT, REDUCE, CLOSE, and REVERSE.
• Legal-Action Masking: A dynamic mask is computed at every step based on current position, margin availability, and scaling depth. Invalid actions are masked out (probability set to zero) during both exploration ($\epsilon$-greedy) and target value computation (in DQN/DDQN).
• Comparison: A simplified 3-action adapter (HOLD, TARGET_LONG, TARGET_SHORT) is used for baseline comparison.

D. Agent and Training

• Algorithms: Deep Q-Network (DQN) and Double DQN (DDQN).
• Architecture: Feed-forward MLP with hidden layers [512, 512, 256].
• Training Protocol: 1 million steps, 40k replay buffer, Huber loss, and strict chronological data processing (no shuffling) to prevent data leakage.

3. Key Contributions

1. Decomposable Reward System: Introduces an auditable 11-component reward architecture that enables systematic ablation and component-wise attribution analysis, moving beyond opaque scalar rewards.
2. Legality-Aware Action Control: Formulates a 10-action discrete space governed by margin-aware constraints, applying valid-action masking consistently in both data collection and value-target computation.
3. High-Fidelity Execution Contract: Designs an environment with explicit anti-lookahead timing, comprehensive transaction cost modeling, and realistic rollover financing (including triple-swap mechanics).
4. Reproducible Experimental Protocol: Provides a configuration-driven pipeline (YAML-based) that allows for the generation of experimental variants without code changes, ensuring deterministic reproducibility.

4. Key Results

Experiments were conducted on the EURUSD hourly dataset (training split only) to analyze learning dynamics rather than generalization.

• Reward Ablation (RQ1):
  • Adding penalty terms does not produce monotonic improvements.
  • Intermediate reward configurations (e.g., adding only transaction costs) often performed worse than the baseline.
  • The full reward configuration (r7) achieved the best results: Sharpe Ratio of 0.765 and Cumulative Return of 57.09%, with a Max Drawdown of 2.31%.
• Action Space Comparison (RQ2):
  • The Extended 10-action interface yielded higher cumulative returns (57.09%) but increased turnover and reduced the Sharpe ratio compared to the conservative 3-action adapter (Sharpe 2.43, Return 33.21%).
  • This indicates a return–activity trade-off: richer primitives allow for better opportunity capture but introduce higher path variability.
• Scaling Strategies (RQ3):
  • All scaling-enabled variants outperformed the "no scaling" baseline in terms of drawdown reduction.
  • The Combined configuration (Pyramid + Martingale) achieved the strongest endpoint return, while the Martingale-only variant showed the lowest drawdown among scaling variants.
  • The asymmetric penalty structure (Martingale penalized more heavily than Pyramiding) successfully shaped behavior, preventing the agent from collapsing into pure loss-amplifying strategies.
• Benchmarking:
  • The Double DQN agent significantly outperformed rule-based baselines (Buy-and-Hold, Momentum, Mean-Reversion) and a Random Policy, which all resulted in negative returns or high drawdowns.

5. Significance and Implications

• Methodological Rigor: The paper shifts the focus from "black-box" performance benchmarking to systematic diagnosis of RL trading agents. It proves that reward engineering is an empirical calibration task, not a simple additive process.
• Bridging the Sim-to-Real Gap: By enforcing strict anti-lookahead timing and realistic friction models, the framework reduces the risk of strategies that work in simulation but fail in live markets due to overlooked costs or timing errors.
• Action Space Design: Demonstrates that granular, legally-masked action spaces allow agents to learn complex risk management behaviors (like pyramiding) that coarse action spaces cannot express.
• Reproducibility: The framework sets a new standard for reproducibility in financial RL by providing a complete, configuration-driven pipeline with deterministic seeding and artifact management.

Conclusion: The study concludes that while the framework does not claim universal superiority over all algorithms, it provides a systems-level methodological contribution. It successfully couples realistic execution, transparent incentives, and reproducible experimentation, offering a robust baseline for future research in leveraged Forex trading. Future work should focus on multi-seed uncertainty quantification, cross-pair portfolio allocation, and deployment-grade execution modeling.