🤖 machine learning

High entropy leads to symmetry-equivariant policies in Dec-POMDPs

This paper demonstrates that high entropy regularization in Dec-POMDPs theoretically guarantees convergence to a unique, symmetry-equivariant policy and empirically shows that increasing entropy coefficients significantly improves cross-play compatibility between independently trained agents, enabling new state-of-the-art results in environments like Hanabi.

Original authors: Johannes Forkel, Constantin Ruhdorfer, Michael Beukman, Andreas Bulling, Jakob Foerster

Published 2026-06-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Johannes Forkel, Constantin Ruhdorfer, Michael Beukman, Andreas Bulling, Jakob Foerster

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a team of robots to play a complex cooperative game, like a high-stakes game of "Hanabi" (a card game where you can't see your own cards) or a chaotic kitchen simulation called "Overcooked." The goal is for them to work together perfectly.

The problem is that when you train these robots to play against themselves (Self-Play), they often develop weird, secret "handshakes" or conventions that only they understand. For example, Robot A might decide that "red means left" and Robot B agrees. But if you take a Robot A trained with a different random seed and pair it with a Robot B from a different training run, they might have decided "red means right." When they try to play together, they crash and burn because their secret languages don't match. This is called a coordination failure.

This paper proposes a surprisingly simple solution: Make the robots more "confused" during training.

Here is the breakdown of their findings using everyday analogies:

1. The Problem: The "Secret Handshake" Trap

Think of the game environment as a room with symmetrical furniture. There are two identical chairs.

Standard Training: The robots learn that "I always sit in the left chair." They do this because it works perfectly when they play against their exact clone.
The Issue: If you swap them with a different pair of robots, one might sit in the left chair and the other in the right. They collide. They broke the symmetry of the room to find a solution, but that solution only works for them, not for anyone else.

2. The Solution: The "Entropy" Spice

The authors introduce a concept called Entropy Regularization. In plain English, this is a penalty added to the robots' learning process that forces them to be less certain about their choices. It's like telling the robots: "Don't just pick the one move that feels best; keep your options open and be a little bit random."

The paper proves a fascinating mathematical fact: If you add enough of this "confusion" (high entropy), the robots stop developing secret handshakes.

Instead of picking "Left" or "Right" exclusively, they learn a strategy that treats "Left" and "Right" exactly the same. They become Symmetry-Equivariant.

The Analogy: Imagine a group of dancers. Instead of everyone agreeing to "dance on the left side of the stage" (which fails if you swap dancers), they learn a dance move that looks the same no matter who is dancing or where they start. They become perfectly compatible with any partner, even strangers they've never met.

3. The "Greedification" Trick

There is a catch. If you make the robots too confused, they become so indecisive that they play terribly, even with their own team. They might just pick random moves.

The paper suggests a two-step recipe:

Train with High Confusion: Train the robots with a very high "entropy" coefficient. This forces them to learn a fair, symmetrical strategy that works with anyone.
Greedy After Training: Once training is done, take those robots and tell them: "Okay, now stop being confused. Look at the symmetrical strategy you learned, and just pick the single best move from it."

The Result: The robots keep the "fairness" of the symmetrical strategy (so they can play with strangers) but regain the "confidence" to play perfectly (so they get high scores).

4. What They Found in the Experiments

The researchers tested this on famous AI benchmarks:

Hanabi: They achieved a new "State-of-the-Art" (SOTA) score. By using a standard algorithm (IPPO) with a higher-than-usual entropy setting, they created robots that could play with each other almost perfectly, even if they were trained on different computers with different random seeds. They beat specialized algorithms designed specifically for this problem.
Overcooked: They found that even with extremely high entropy settings (much higher than anyone usually tries), the robots could still learn to coordinate effectively after being "greedified."
The Limit: They also showed that in some very specific, tricky scenarios, this method can't find the perfect solution. Sometimes, being too "fair" and symmetrical prevents the robots from exploiting a specific, highly efficient trick that requires breaking symmetry. However, for most real-world scenarios, the method works wonders.

The Main Takeaway

The paper argues that AI researchers have been too afraid to turn up the "entropy" knob. They usually keep it low to get high scores quickly. But the authors show that turning the knob way up forces the AI to learn a universal, fair language that allows different agents to coordinate instantly without prior agreement.

In short: To make AI agents that can work with anyone (humans or other AIs), don't just teach them to win; teach them to be a little bit undecided during training, and then let them make firm decisions only at the very end.

Technical Summary: High Entropy Regularization Leads to Symmetry-Equivariant Policies in Dec-POMDPs

Problem Statement

In cooperative Multi-Agent Reinforcement Learning (MARL) within Decentralized Partially Observable Markov Decision Processes (Dec-POMDPs), a central challenge is Zero-Shot Coordination (ZSC). Standard training methods, such as Self-Play (SP), often lead agents to converge on specific coordination conventions that break the inherent symmetries of the environment. While these policies achieve high returns when paired with copies of themselves (SP), they fail catastrophically when paired with independently trained policies (Cross-Play or XP), such as those trained with different random seeds or algorithms. This incompatibility arises because different training runs may select different, mutually exclusive symmetry-breaking conventions (e.g., one agent always chooses action A while the other chooses B, versus the reverse).

Existing specialized algorithms (e.g., Other-Play, Off-Belief Learning) attempt to enforce symmetry-equivariance, but they often require prior knowledge of the environment's symmetries or complex algorithmic modifications. This paper investigates whether standard policy gradient methods, specifically Independent PPO (IPPO), can naturally learn symmetry-equivariant policies through a specific hyperparameter adjustment: high entropy regularization.

Methodology

Theoretical Framework

The authors analyze the dynamics of tabular softmax policy gradient ascent with entropy regularization. They define the entropy-regularized policy gradient flow for a Dec-POMDP.

Key Distinction: The paper distinguishes between the gradient of the entropy-regularized objective ( $\nabla_\theta J^\alpha_{SP}$ ) and the entropy-regularized gradient of the standard objective ( $\nabla^\alpha_\theta J_{SP}$ ). The latter, which corresponds to the standard PPO entropy bonus, is the focus of the theoretical analysis.
Symmetry Definitions: A Dec-POMDP symmetry is defined as a tuple of bijections over states, actions, and observations that preserve the transition dynamics, rewards, and initial state distribution. A policy is symmetry-equivariant if it remains invariant under the action of these symmetries.
Greedification: The authors propose a post-training "greedification" step, where the learned stochastic policy is converted into a deterministic policy by selecting the action with the highest probability (or uniform distribution over tied actions).

Theoretical Contribution

The paper proves Theorem 3.1, which establishes that for any Dec-POMDP with a tabular softmax parametrization, there exists a finite entropy threshold $\alpha'$ . For any entropy coefficient $\alpha > \alpha'$ :

The vector field of the entropy-regularized policy gradient has a unique zero point $\theta_\alpha$ .
This unique fixed point corresponds to a symmetry-equivariant joint policy.
The policy gradient flow converges to this unique point from any initialization.
Consequently, policies trained with different initializations (seeds) become fully compatible, meaning their Cross-Play (XP) returns equal their Self-Play (SP) returns.

The authors explicitly note that this result does not hold for the gradient of the maximum entropy RL objective ( $\nabla_\theta J^\alpha_{SP}$ ), constructing a counterexample where high entropy leads to multiple incompatible maxima.

Empirical Approach

The authors evaluate these theoretical predictions using standard Independent PPO (IPPO) in three environments:

Hanabi: A cooperative card game with imperfect information.
Overcooked: A cooperative cooking simulation.
Yokai: A recently developed environment for tracking beliefs over space and time.

Experimental Protocol:

Hyperparameter Sweep: They train IPPO agents across a wide range of entropy coefficients ( $\alpha$ ), significantly higher than typical values (e.g., up to 0.50 in Overcooked, compared to the standard 0.01–0.10).
Evaluation: They measure both Self-Play (SP) and Cross-Play (XP) returns. Crucially, they evaluate both the raw stochastic policies ("non-greedy") and the greedified deterministic policies ("greedy").
Architecture: Experiments utilize standard architectures including Feed-Forward (FF), LSTM, and a Public-Private LSTM (used for Off-Belief Learning compatibility).

Key Results

1. Convergence to Symmetry-Equivariant Policies

The experiments confirm the theoretical prediction: as the entropy coefficient $\alpha$ increases, the gap between SP and XP narrows.

Hanabi: With an entropy coefficient of $\alpha = 0.05$ and an LSTM architecture, the authors achieve an inter-seed XP return of 24.45 (out of a maximum of 25). This surpasses the previous State-of-the-Art (SOTA) of 24.30 achieved by specialized algorithms like Off-Team Learning (OT-OBL).
Overcooked: Increasing $\alpha$ to extremely high values (0.50) continues to improve greedy XP returns, with SP and XP becoming nearly identical.
Yokai: The entropy threshold appears to be very high; above a certain point, the policies become too random to learn effectively, suggesting that in some environments, the cost of high entropy regularization may outweigh the benefits for coordination.

2. The Role of Greedification

A critical finding is that while high entropy regularization forces the stochastic policy to be symmetry-equivariant, the resulting expected return of the stochastic policy may be low (approaching uniform random behavior). However, the greedified version of these policies retains the symmetry-equivariant property while recovering high returns.

In Hanabi and Overcooked, the greedy XP of high-entropy policies is significantly higher than non-greedy XP and often exceeds the performance of low-entropy policies.
This suggests a training recipe: train with high entropy to ensure symmetry-equivariance, then greedify the policy for deployment.

3. Architecture Sensitivity

The required entropy threshold depends on the policy architecture.

LSTM vs. FF: In Hanabi, Feed-Forward (FF) networks require significantly higher entropy coefficients to achieve compatibility compared to LSTM networks. The authors interpret this as FF networks lacking memory, forcing them to rely on highly specialized, fragile conventions to communicate, which high entropy helps to dissolve.

4. Limitations

The paper acknowledges that high entropy regularization is not a universal panacea.

Suboptimal Equivariance: In certain environments (e.g., a modified "cat/dog" game and the "Yokai" environment), high entropy ensures convergence to a symmetry-equivariant policy, but not necessarily the optimal one. In these cases, the optimal symmetry-equivariant policy requires deterministic actions that high entropy regularization suppresses, leading to suboptimal performance even after greedification.
Bias in Advantage Estimates: The use of Generalized Advantage Estimation (GAE) with low $\lambda_{GAE}$ (high bias) can reintroduce symmetry-breaking, even with high entropy.

Significance and Claims

The paper claims that standard policy gradient methods are "much more powerful" for Zero-Shot Coordination than previously assumed, provided that:

Entropy coefficients are swept over a much larger range than typically practiced (often an order of magnitude higher).
Evaluation includes Cross-Play (XP) metrics, not just Self-Play (SP), to detect symmetry-breaking.
Greedification is applied post-training to recover performance while maintaining compatibility.

The authors argue that this approach offers a simpler alternative to specialized ZSC algorithms, as it requires no prior knowledge of the environment's symmetries and uses standard MARL infrastructure. While they admit that in some specific Dec-POMDPs the optimal symmetry-equivariant policy may remain unlearnable via this method, the empirical results in Hanabi and Overcooked demonstrate that high entropy regularization can achieve SOTA performance in inter-seed coordination using vanilla IPPO.

The paper concludes by urging the community to reconsider the standard hyperparameter ranges for entropy in Dec-POMDPs, suggesting that the "sweet spot" for coordination often lies at much higher entropy values than those used for maximizing single-seed SP returns.