🤖 machine learning

Self-Play Reinforcement Learning under Imperfect Information in Big 2

This paper introduces a self-play reinforcement learning framework for the imperfect-information card game Big 2, demonstrating that PPO outperforms value-based methods under controlled conditions and highlighting the benefits of entropy regularization and current-policy self-play for training robust multi-agent agents.

Original authors: Aalok Patwa

Published 2026-05-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Aalok Patwa

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are sitting at a table with three friends playing a card game called Big 2. You can see your own hand of 13 cards, and you can see the cards everyone else has played so far. But you cannot see the cards hidden in your friends' hands. This is what researchers call "imperfect information." You have to guess what they are holding based on what they play, what they pass on, and how many cards are left.

The goal of this paper is to teach a computer program (an AI agent) how to play this game really well using Reinforcement Learning (RL). Think of RL as a student learning by trial and error: the AI plays thousands of games, gets "grades" (rewards) for winning or losing, and slowly figures out the best strategies.

Here is a breakdown of what the paper found, using simple analogies:

1. The Challenge: A Game of Hidden Secrets

Big 2 is tricky because:

You don't know everything: You have to guess your opponents' hands.
The rules change every turn: Sometimes you can play one card, sometimes a pair, sometimes a complex "straight." The list of legal moves changes constantly.
Short-term vs. Long-term: Sometimes playing your strongest card right now feels good, but it might leave you helpless later. The AI has to learn to sacrifice a small win today to win the whole game tomorrow.

2. The Experiment: Who Learned Best?

The researchers set up a "training camp" where they taught four different types of AI students using the exact same rules, the same amount of computer power, and the same amount of time. They wanted to see which learning style worked best.

The Students:
1. PPO (The "Policy" Student): This student learns by trying different strategies and adjusting its "gut feeling" about what move to make. It's like a coach telling you, "Try playing a pair here, it feels right."
2. Q-Learning / SARSA / Monte Carlo (The "Value" Students): These students try to memorize a score for every possible situation. They ask, "If I play this card, what is the exact point value I will get in the end?"

The Result: The PPO student won the training camp. It learned faster and became a better player than the "Value" students.

Why? In a game with hidden information and many players, trying to calculate the exact future score for every single move is like trying to predict the weather for next year with perfect accuracy—it's too noisy and takes too long. PPO's approach of "adjusting the strategy based on how things went" was more efficient.

3. The Secret Sauce: Entropy (Don't Be Too Predictable)

The researchers noticed something interesting about the winning PPO student. At first, it was very confident, picking the same "best" move 90% of the time. But in a game where your opponents can't see your cards, being 100% predictable is dangerous. If you always play the same way, smart opponents will figure you out.

The Analogy: Imagine playing Rock-Paper-Scissors. If you always throw "Rock" because it feels like the best move, your opponent will quickly start throwing "Paper" and beat you. You need to mix it up a little bit.
The Fix: The researchers added a little bit of "entropy" (randomness) to the AI's training. They told it, "Don't be 100% sure; keep a little bit of surprise in your moves."
The Result: The AI that kept a moderate amount of randomness performed the best. It became harder to read and more adaptable. However, too much randomness made it play stupidly, so balance was key.

4. The Training Partner: Who Should You Play Against?

The AI learned by playing against itself (Self-Play). The researchers tested three different ways to set up these practice matches:

Fixed Opponent: The AI played against a computer that always played the same "Smart" strategy.
Checkpoint Opponent: The AI played against older versions of itself saved from earlier in the training.
Current-Policy Opponent: The AI played against the current version of itself, which was getting better every single day.

The Result: Playing against the Current-Policy (the version of itself that was improving right now) was the best method.

The Analogy: Imagine learning to swim.
- If you only practice against a slow, fixed robot, you get good at beating that robot, but you might not learn how to handle a fast swimmer.
- If you practice against your "past self," you might be fighting battles you've already won.
- If you practice against your current self, the difficulty level rises exactly as fast as your skills improve. It's like a personal trainer who adjusts the weight on the bar exactly to your current strength. This kept the AI constantly challenged and learning the most relevant lessons.

Summary

This paper shows that for complex card games like Big 2:

Strategy-based learning (PPO) works better than trying to memorize exact scores.
Keeping a little bit of randomness in your moves makes you a tougher opponent.
Practicing against a version of yourself that is improving right alongside you is the fastest way to learn.

The authors conclude that Big 2 is a great "test kitchen" for teaching computers how to make smart decisions when they don't have all the facts, a skill that could eventually help in many real-world situations where information is hidden.

Technical Summary: Self-Play Reinforcement Learning under Imperfect Information in Big 2

Problem Statement

The paper addresses the challenges of training reinforcement learning (RL) agents in imperfect-information multiplayer games. Specifically, it investigates the game of Big 2, a four-player card-shedding game characterized by:

Hidden Information: Players observe only their own hands and public play history; opponents' hands must be inferred from actions, passes, and remaining card counts.
Sparse and Delayed Rewards: The primary reward (winning) occurs only at the terminal state, with intermediate rewards being zero.
Non-Stationary Opponents: In self-play settings, the opponent distribution changes as agents improve.
Variable Action Spaces: Legal actions are hand-specific combinations (singles, pairs, straights, etc.) that change drastically between turns, making fixed action abstraction difficult.

The authors aim to determine whether policy-gradient or value-based objectives learn more effectively in this setting under a controlled, limited-compute budget, and how specific training design choices (entropy regularization, opponent curriculum) influence stability and performance.

Methodology

1. Environment and Representation

The authors developed a simulator where the observation state ( $o_i$ ) strictly reflects available information:

Input: The acting player's private hand (padded), the active trick, all seen cards, remaining card counts for opponents, pass counts, and per-opponent played-card histories.
Action Space: The simulator enumerates all legal combinations in the player's hand and filters them based on the active trick, producing a variable-length legal action set $A(o_i)$ . This avoids invalid-action exploration.
Neural Architecture:
- State Encoder: Uses a shared card embedding table. The player's hand is processed via self-attention and pooled. Public information (trick, seen cards, opponent counts) is embedded and concatenated to form a state embedding.
- Action Scoring: Legal actions are encoded as 80-dimensional feature vectors. A dot product between the state embedding and the action embedding scores each legal candidate.
- Distinction: Unlike prior work that used fixed 1695-action heads masked to legal moves, this model scores only legal actions, allowing held cards to interact before pooling.

2. Learning Algorithms

The study compares four algorithms under identical architectures and training budgets (5,000 batches, 64 episodes/batch):

PPO (Proximal Policy Optimization): A policy-gradient baseline using clipped surrogate objectives and a separate value head.
Monte Carlo Q Approximation: A value-based method using the full discounted return ( $y_t = \sum \gamma^{k-t} r_k$ ) as the target.
SARSA: An on-policy value-based method using one-step targets ( $y_t = r_t + \gamma Q(s_{t+1}, a_{t+1})$ ).
Q-Learning: An off-policy value-based method using max targets ( $y_t = r_t + \gamma \max_a Q(s_{t+1}, a)$ ).

All value-based methods use $\epsilon$ -greedy exploration during training and greedy selection during evaluation.

3. Training Curricula and Evaluation

Self-Play: The default setting uses current-policy self-play, where the current policy controls all four seats.
Baselines: Evaluation is performed against three heuristic opponents:
- Random: Uniformly selects legal actions.
- Greedy: Plays the weakest legal non-pass combination.
- Smart: A rule-based policy that scores actions based on strategic features (card shedding, preserving combinations, avoiding early use of '2's).
Metrics: Win rate and average game score (sum of remaining cards for losers, negative for losers) against each opponent class.

Key Contributions and Findings

1. Algorithmic Performance

Under a limited compute budget, PPO outperforms all value-based methods (Monte Carlo Q, SARSA, Q-learning) across all opponent classes.

PPO achieved an 85.4% win rate against Random, 58.2% against Greedy, and 37.1% against Smart.
Value-based methods, while learning useful policies, failed to match PPO's final performance. The authors attribute this to the difficulty of value estimation in Big 2's high-variance, imperfect-information state space, where strategically important states are visited rarely, and terminal returns must be assigned credit across long sequences.

2. Entropy Regularization

The authors found that PPO without entropy regularization becomes overly deterministic, sampling top actions with >90% probability even in ambiguous strategic situations.

Finding: Moderate entropy regularization ( $\beta_{ent} = 0.05$ ) significantly improved performance across all opponent classes (e.g., increasing win rate against Smart from 37.1% to 43.5%).
Implication: Stochastic policies are beneficial in imperfect-information games to maintain unpredictability and avoid over-committing to suboptimal action preferences. However, excessive entropy degrades performance.

3. Opponent Curriculum

The study compared current-policy self-play against checkpoint self-play (training against saved past policies) and fixed-opponent training (training only against the Smart heuristic).

Finding: Current-policy self-play yielded the best results for both PPO and Monte Carlo Q.
Reasoning: Training against a fixed deterministic opponent (Smart-only) narrows the state distribution and reduces exploration. Checkpoint self-play introduces diversity but dilutes the learning signal by exposing the agent to "stale" mistakes it has already learned to beat. Current-policy self-play acts as an adaptive curriculum, keeping the opponent difficulty aligned with the agent's current skill level.

Significance and Claims

The paper claims to provide the first controlled empirical study comparing RL objectives and training design choices specifically for Big 2. Its significance lies in:

Establishing a Baseline: It offers an accessible baseline for future research on search, abstraction, and opponent modeling in Big 2, isolating factors often confounded in larger systems.
Demonstrating Efficiency: It shows that deep RL can learn robust strategies in a four-player imperfect-information game without engineered opponent models, tree search, or heuristic action pruning beyond legal filtering.
Identifying Critical Factors: It highlights that in limited-budget settings, algorithm choice (PPO > Value-based), policy stochasticity (entropy regularization), and curriculum design (current-policy self-play) are decisive factors for success.

The authors remain modest, noting that their study does not compare against Counterfactual Regret Minimization (CFR), search-augmented agents, or larger training budgets, and that future work is needed to determine if value-based methods can close the gap with extended training or if cross-play reveals new strategic weaknesses.