Technical Summary: MTG-Causal-RL

Problem Statement

Causal Reinforcement Learning (RL) currently lacks benchmarks that simultaneously address sequential decision-making, hidden information, large masked action spaces, and explicit causal structure. Existing environments typically offer either low-level control with clear physical interventions (e.g., robotics) or strategic games lacking an explicit causal interface (e.g., standard board games). This gap hinders research into causal credit assignment, structural transfer across defined archetypes, and policy auditability under complex, partially observable conditions.

Methodology

The MTG-Causal-RL Benchmark

The authors introduce MTG-Causal-RL, a Gymnasium-based benchmark built on Magic: The Gathering (MTG). The environment is designed to be strategically rich yet computationally tractable, featuring:

• Observation Space: A 3,077-dimensional partial observation vector covering visible hand, battlefield, graveyard, life totals, phase, and mana state.
• Action Space: A 478-action masked discrete space organized into 16 categories (e.g., casting spells, attacking, blocking, mana payment). Typically, only 2–15 actions are legal per state.
• Archetypes: Five competitive Standard 2025 deck archetypes (Mono-Red Aggro, Azorius Control, Dimir Midrange, Domain Ramp, Boros Convoke) serve as the training and evaluation pool.
• Structural Causal Model (SCM): A hand-designed SCM defines the causal relationships between strategic variables. It is not learned from data but derived from MTG strategic theory to prioritize interpretability. The model consists of four layers:
  1. Resources: Mana, land drops, mana creatures.
  2. Board State: Board pressure, threat density.
  3. Strategic Position: Card advantage, tempo, life buffer, removal availability.
  4. Outcome: Win probability ($WinProb$), modeled as a sigmoid function of its six parents (the strategic factors).
• Intervention Interface: The environment exposes causal variables, SCM-predicted intervention effects (calculated analytically via $do(\cdot)$ propagation), and per-factor credit traces. Actions map to interventions (e.g., playing a land corresponds to $do(LandDrop=1)$), propagating effects through the SCM to update $WinProb$.

Proposed Agent: CGFA-PPO

As a secondary contribution, the authors propose Causal Graph-Factored Advantage PPO (CGFA-PPO). This agent extends standard masked PPO with three causal components:

1. Per-Factor Critics: A set of $K$ critic heads ($V_k$) corresponding to the SCM parents of $WinProb$. These are trained against per-factor rewards defined as the change in factor values ($\phi_k(s_{t+1}) - \phi_k(s_t)$).
2. Learnable Mixture Weights: A softmax layer ($w_k$) that blends the scalar advantage with the factor-aligned advantages. These weights are initialized from the SCM's logistic regression weights to provide a structural prior.
3. State-Conditional Residual Gate: A gate $g(s) \in (0,1)$ that dynamically decides the reliance on the scalar advantage versus the factor-aligned advantage for each state.
4. Intervention-Calibration Loss: A Pearson correlation loss ($L_{cal}$) that aligns the learned per-factor advantages ($A_k$) with the SCM-predicted intervention effects ($\epsilon_k$).

Experimental Protocol

The evaluation employs a rigorous statistical protocol to ensure reproducibility and valid causal claims:

• Paired Seeds: Comparisons use paired seeds across agents and decks.
• Statistical Tests: Win rates use Wilson score intervals and paired-bootstrap confidence intervals. Pairwise comparisons utilize Wilcoxon signed-rank tests and Welch t-tests.
• Correction: Holm-Bonferroni correction is applied within pre-registered comparison families to control for multiple comparisons.
• Metrics: Beyond headline win rates, the benchmark tracks leave-one-out cross-archetype transfer gaps, per-factor calibration trajectories, and credit assignment shares.

Key Results

Headline Performance

• Baseline Comparison: Both standard Masked PPO and CGFA-PPO significantly outperform the uniform random baseline across all five archetypes.
• Deck Dependency: Neither agent uniformly dominates the other. CGFA-PPO outperforms PPO on Azorius Control (+5.0 pp) and Mono-Red Aggro (+2.8 pp). Conversely, PPO outperforms CGFA-PPO on Boros Convoke (-3.9 pp), Dimir Midrange (-2.2 pp), and Domain Ramp (-8.3 pp).
• Heuristic Anchors: Hand-coded heuristics perform strongly on proactive decks (e.g., Mono-Red) but poorly on reactive or complex decks (e.g., Azorius Control, Domain Ramp), indicating the benchmark contains regimes favoring both scripting and learning.

Ablation and Diagnostics

• Causal Augmentation: On the Mono-Red diagnostic slice, the full CGFA-PPO (70.6%) performs comparably to the scalar-only control (69.4%) and the no-gate variant (70.6%). The "no-calibration" variant actually achieved 73.3%, suggesting that calibration and gating are mechanisms for alignment and interpretability rather than immediate performance boosters in this setting.
• Transfer: Leave-one-out transfer tests revealed that held-out opponent mixes were often easier than the training pool (negative generalization gaps for both PPO and CGFA-PPO), demonstrating the benchmark's ability to detect evaluation shifts rather than proving solved transfer.
• Calibration: Training trajectories showed that factor correlations moved positive, credit share concentrated heavily on the "life buffer" factor, and the residual gate remained active, confirming the agent utilizes the causal interface in an auditable manner.

Significance and Claims

The paper claims that MTG-Causal-RL provides a shared testbed for three open questions that current benchmarks cannot pose together:

1. Causal Credit Assignment: Evaluating agents under masked action spaces where actions must be credited to specific causal factors.
2. Structural Transfer: Testing generalization across cleanly defined strategic archetypes.
3. Policy Auditability: Using SCM-grounded metrics (calibration curves, intervention effects) to audit why a policy makes decisions, not just what the win rate is.

The authors emphasize that the primary contribution is the benchmark environment and causal interface, which allows for the separation of capacity, factor critics, gating, and calibration. The results suggest that while causal agents can be competitive, the "causal" label alone does not guarantee superior scalar win rates; the value lies in the ability to diagnose structural behaviors and audit policy updates against explicit causal models. The benchmark is released openly to support causal RL, world-model, and LLM-agent research.