Technical Summary: EMAgnet

Problem Statement

Recent advances in solving two-player zero-sum imperfect-information games (IIGs) have demonstrated that regularized policy gradient methods, such as Proximal Policy Optimization (PPO), can match or exceed specialized game-theoretic algorithms when used in self-play. A critical design decision in this framework is the choice of the regularization target. The uniform distribution has emerged as a standard target, implemented via an entropy bonus. However, the uniform target is strategically agnostic; it regularizes the policy equally toward all actions, regardless of their viability.

In games with large strategy spaces containing many strictly dominated strategies (suboptimal options), the uniform target wastes the regularization budget on irrelevant strategies. This limitation becomes increasingly consequential as game complexity grows and the fraction of strategically relevant strategies shrinks. Furthermore, existing approaches using annealed uniform regularization face a dilemma: strong early regularization forces the policy to explore dominated strategies, while weak late-stage regularization fails to provide sufficient pressure to encourage strategic mixing, often leading to convergence on pure strategies rather than Nash equilibria.

Methodology

The authors introduce EMAgnet, a method that extends the concept of a "moving magnet" regularization target from tabular settings to deep reinforcement learning. Instead of regularizing toward a fixed uniform distribution, EMAgnet regularizes toward an Exponential Moving Average (EMA) of the policy's own network parameters.

Core Mechanism

The method builds upon PPO in symmetric self-play. The standard PPO objective is augmented with a Kullback-Leibler (KL) divergence term that pulls the current policy parameters ($\theta$) toward a "magnet" policy ($\theta_{mag}$):

$$L_{PPO-EMAg}(\theta) = L_{PPO}(\theta) + \lambda_{KL} \mathbb{E}_{z \sim \mathcal{T}} \left[ D_{KL}(\pi_{\theta_{mag}}(\cdot | z) \parallel \pi_{\theta}(\cdot | z)) \right]$$

The magnet parameters are updated after each PPO step using an arithmetic average in parameter space:
$$\theta_{mag} \leftarrow (1 - \tau) \theta_{mag} + \tau \theta$$
where $\tau \in (0, 1]$ is the step size. The magnet is initialized with the same random weights as the policy.

Adaptive Properties

The key innovation is the adaptive nature of the target:

1. Forgetting Bad Strategies: As the policy learns to avoid strictly dominated strategies, the EMA magnet gradually stops regularizing toward them because the policy's weights no longer support those actions.
2. Remembering Good Strategies: The EMA naturally accumulates a smoothed mixture over the strategies encountered during self-play cycling. This encourages the policy to maintain coverage over strategically relevant options.
3. Constant Strength: Unlike annealed uniform regularization, the KL regularization strength ($\lambda_{KL}$) remains constant, but the target evolves. This avoids the "dilemma" where strong regularization forces exploration of bad strategies or weak regularization fails to prevent collapse to pure strategies.

Key Contributions

1. Algorithmic Extension: The paper extends the tabular moving magnet concept (previously explored by Sokota et al.) to deep RL by implementing a parameter-space EMA of the policy weights as the regularization target.
2. Adaptive Regularization: The method provides an adaptive regularization target that evolves with the agent's improving strategy, distinguishing it from fixed targets like the uniform distribution.
3. Empirical Evaluation: The authors evaluate PPO-EMAg against PPO self-play with uniform-magnet regularization (using both linear and power-law annealing schedules) across three categories of games:
   • Standard benchmarks (Biased RPS, Goofspiel-4, Kuhn Poker).
   • Modified benchmarks with a single strictly dominated "forfeit" action (FF variants).
   • Modified benchmarks where the vast majority of the strategy space is dominated via gridworld navigation tasks (Control variants).

Results

The evaluation demonstrates that PPO-EMAg achieves lower exploitability than uniform baselines in environments containing strictly dominated strategies, while remaining competitive in standard benchmarks.

• Standard Games: In Biased RPS, Goofspiel-4, and Kuhn Poker, PPO-EMAg is competitive with baselines, with the EMA magnet policy often achieving the lowest final exploitability.
• Forfeit (FF) Games: In games augmented with a strictly dominated forfeit action, PPO-EMAg significantly outperforms uniform baselines. The uniform magnet continues to regularize toward the forfeit action throughout training, whereas the EMA magnet adapts to reduce regularization toward it as the policy learns to avoid it.
• Control Games: In environments where most strategies are strictly dominated (due to navigation failures), PPO-EMAg shows superior performance and significantly faster convergence.
  • In Control Biased RPS, uniform linear annealing fails to learn for approximately 7 million steps before dropping late in training, while PPO-EMAg converges by 2 million steps.
  • In Control Goofspiel, uniform linear annealing barely improves over 6 million steps, whereas PPO-EMAg reaches low exploitability much earlier.
• Trajectory Analysis: Visualizations of policy trajectories in the strategy simplex reveal that uniform annealing causes policies to get stuck on dominated strategies early on or collapse to pure strategies later. PPO-EMAg avoids this by gradually reducing mass on dominated actions while maintaining mixing pressure, allowing the policy to cycle near the Nash equilibrium.

Significance and Claims

The paper claims that EMAgnet offers a simple yet effective mechanism for adapting regularization targets to an agent's evolving strategy in large games. By regularizing toward a parameter-space EMA of the policy's own weights, the method avoids the pitfalls of uniform regularization in games with large numbers of strictly dominated strategies.

The authors posit that as games grow in complexity, the fraction of the strategy space that is strategically relevant tends to shrink. In such scenarios, allocating regularization budget to suboptimal options incurs an increasing cost. EMAgnet addresses this by "forgetting" bad strategies while "remembering" good ones, requiring only a single EMA update per training step beyond the standard PPO loop. The paper concludes that while PPO-EMAg is competitive on standard benchmarks, its primary advantage lies in games with high densities of dominated strategies, where it enables faster convergence and lower final exploitability compared to annealed uniform regularization.