Asset-Universe Design and Action-Space Granularity in Cointegration-Based Deep Reinforcement Learning for Cryptocurrency Statistical Arbitrage

Technical Summary: Asset-Universe Design and Action-Space Granularity in Cointegration-Based DRL for Cryptocurrency Statistical Arbitrage

Problem Statement
Cryptocurrency markets offer high volatility and continuous trading, creating opportunities for statistical arbitrage based on cointegration. However, the performance of Deep Reinforcement Learning (DRL) frameworks in this domain is often evaluated solely on algorithmic choice (e.g., DQN vs. PPO), while treating the construction of the asset universe and the granularity of the action space as fixed, secondary preprocessing steps. This study posits that these two design choices are critical determinants of performance. Specifically, it investigates whether the stability and tradability of synthetic arbitrage portfolios depend on the economic coherence of the selected asset universe and whether allowing agents to scale exposure (granular actions) improves risk control compared to binary full-exposure decisions.

Methodology
The study employs a cointegration-based DRL framework using 30-minute USDT-denominated Binance spot data. The empirical design is structured around four distinct asset universes to test the impact of universe size and economic coherence:

1. Baseline 14-Asset Universe: A replication of a standard 14-asset set.
2. Broad 20-Asset Universe: An expanded set to test scalability and noise.
3. Payment-Coin 6-Asset Universe: A focused subset of payment-oriented assets (e.g., BTC, XRP, DOGE).
4. Layer-1 Platform 9-Asset Universe: A sector-specific subset of blockchain infrastructure assets.

Experimental Framework:

• Portfolio Construction: For each trading scenario, a 288-observation (6-day) formation window is used to estimate cointegration vectors and construct two synthetic portfolios (A and B). These weights are fixed for the subsequent 48-observation (1-day) trading horizon.
• DRL Agents: Three agents are evaluated: Deep Q-Network (DQN), Proximal Policy Optimization (PPO), and Advantage Actor–Critic (A2C).
• Action Spaces:
  • Binary: Agents must choose full long/short exposure ($a_t \in \{-1, 1\}$).
  • Granular: Agents can choose full, half, or flat exposure ($a_t \in \{-1, -0.5, 0, 0.5, 1\}$).
• Reward Function: Rewards are based on the realized relative log-returns between portfolios A and B, penalized by transaction costs proportional to the change in exposure ($\Delta a_t$).
• Evaluation: Strategies are compared against a non-DRL cointegration benchmark (COIN) using out-of-sample testing (Dec 2024 – May 2025). Robustness is assessed across random seeds (42, 101, 202).

Key Contributions

1. Asset-Universe as a Design Variable: The study treats asset-universe selection not as a neutral preprocessing step but as a core empirical variable. It demonstrates that the economic coherence of the asset set significantly influences the learnability of cointegration relationships.
2. Action-Space Granularity Analysis: It directly compares binary versus granular action spaces within the same cointegration framework, testing whether partial exposure capabilities improve risk-adjusted performance or merely reduce raw returns.
3. Seed-Level Robustness: The paper moves beyond single-run reporting by evaluating key models across multiple random seeds to distinguish stable model behavior from stochastic training artifacts.

Empirical Results

• Impact of Asset Universe: Performance is highly sensitive to universe design.
  • The Payment-Coin 6 and Baseline 14 universes yielded the strongest results. In the Payment-Coin universe, DQN with binary actions achieved a 32.26% total return (seed 42). In the Baseline 14 universe, DQN binary returned 30.24%.
  • The Broad 20 universe performed poorly for DRL agents (all negative returns), with the non-DRL COIN benchmark (3.25%) outperforming all DRL strategies.
  • The Layer-1 Platform 9 universe resulted in negative returns across all strategies, including the benchmark.
  • Conclusion: Expanding the asset universe does not automatically improve performance; a smaller, economically coherent universe often provides more stable cointegration signals.
• Impact of Action-Space Granularity:
  • Binary actions produced higher raw returns in the strongest settings (e.g., DQN in Payment-Coin and Baseline 14), suggesting full directional exposure is optimal when signals are strong.
  • Granular actions did not universally increase returns but significantly improved risk control in weaker environments. They allowed agents to reduce drawdowns and turnover in scenarios where signals were noisy (e.g., Layer-1 and Broad 20 universes).
• Robustness:
  • Results are sensitive to stochastic training variation. While Payment-Coin DQN binary remained positive on average across seeds (mean 16.50%), it exhibited high return dispersion.
  • The strong performance of Baseline 14 DQN binary in the seed-42 run did not hold across other seeds (mean return became negative), indicating that single-run success does not guarantee stability.

Significance and Claims
The paper claims that the success of DRL-based cryptocurrency statistical arbitrage is not solely a function of algorithmic superiority but emerges from the interaction between the cointegration structure, asset-universe design, action-space specification, and training stability.

• For Researchers: The study argues that DRL trading evaluations must report asset-universe design, action-space specifications, and seed sensitivity rather than focusing exclusively on the best-performing algorithm in a fixed setting.
• For Practitioners: It suggests that asset selection is a primary modeling decision that determines whether an arbitrage relationship is learnable. Furthermore, granular action spaces should be viewed primarily as risk-management mechanisms to handle weak signals or high transaction costs, rather than as universal return enhancers.
• Overall Conclusion: The design of the trading environment (universe and action space) matters as much as the DRL algorithm itself. DRL can add value in favorable settings (e.g., Payment-Coin universe with DQN), but it does not universally outperform classical cointegration benchmarks, particularly when underlying asset relationships are noisy or unstable.