🤖 machine learning

Superhuman AI for Generals.io Using Self-Play Reinforcement Learning

This paper presents a superhuman AI agent for the real-time strategy game Generals.io that achieves the #1 ranking on the public leaderboard by leveraging a JAX-native simulator with a 10,000x speedup to enable efficient end-to-end training of a vision transformer policy via self-play reinforcement learning.

Original authors: Matej Straka, Viliam Lisý, Martin Schmid

Published 2026-06-23

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Matej Straka, Viliam Lisý, Martin Schmid

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a digital battlefield called Generals.io. It's a real-time strategy game where two commanders (or teams) command armies on a grid. They can't see the whole map at once; they only see the land they own and the immediate area around their troops. The rest is hidden in a "fog of war." The goal is simple: capture the enemy's headquarters (the General) while protecting your own.

This paper describes how a team of researchers built an AI that is so good at this game, it beat the very best human players in the world. Here is the story of how they did it, broken down into simple concepts.

1. The "Super-Speed" Simulator

Before they could train the AI, they needed a way to practice. In the past, training AI on this game was like trying to learn to drive by watching a car move one inch per hour. It was too slow.

The researchers built a new "simulator" (a digital version of the game) using a special tool called JAX. Think of this as upgrading from a bicycle to a supersonic jet.

The Old Way: The previous version could run about 3,500 game steps per second on a standard computer.
The New Way: Their new version runs 50 million steps per second on a single graphics card.
The Result: This is a 10,000x speedup. It means the AI can play millions of games in the time it takes a human to play one. This speed removed the biggest bottleneck: the AI could finally learn fast enough to get really good.

2. The "Self-Play" Training Camp

Instead of teaching the AI by showing it videos of human experts (which is like teaching a chess player by only showing them grandmaster games), they let the AI play against itself.

The Method: The AI plays millions of games against copies of itself.
The Reward: The game is very simple: you either win (+1 point) or you lose (-1 point). There are no "participation trophies" or points for killing a few enemy soldiers. You only get a reward if you capture the enemy General.
The Challenge: Because the reward is so rare (you only get it at the very end of a long game), it's hard for the AI to figure out what it did right. It's like trying to learn to bake a cake by only tasting the final product, with no feedback on whether you added too much sugar or not enough flour.

3. The Secret Sauce: What Actually Worked

The researchers tested many different "recipes" to see what made the AI superhuman. They found that some things people thought were necessary were actually useless, while a few simple tricks made all the difference.

No "Cheat Sheets" Needed: They didn't need to give the AI complex rules or hand-tuned rewards for doing "good" things (like capturing a castle). Just the simple "Win or Lose" signal was enough, thanks to the super-fast simulator.
The "Average" Student (EMA): In many AI systems, you use the very last version of the AI that finished training. The researchers found that taking an Exponential Moving Average (EMA) of the AI's brain over time worked better.
- Analogy: Imagine a student taking a test every day. The "last iterate" is just their score on the final day. The "EMA" is like taking the average of their performance over the whole month. The researchers found that the "average student" was actually more consistent and smarter than the student on their best (or worst) single day.
Filtering the "Good" Games (Top-Advantage Filtering): The AI played millions of games, but most were boring or random. The researchers realized they didn't need to learn from every game. They only kept the top 25% of games where the AI had a clear advantage and learned from those.
- Analogy: If you are trying to learn to swim, you don't need to watch every time you flail in the water. You only need to study the times you swam smoothly and efficiently. This made training much faster and more efficient.
Starting Small: They didn't start the AI on a huge map. They started with the Generals very close together so the AI could learn how to win quickly. Then, they slowly moved the Generals further apart, letting the AI learn long-term strategy step-by-step.

4. The Results: Beating the Humans

After training for four days on powerful computers, the AI (nicknamed "Average Joe") went to the public leaderboard.

Ranking: It reached #1 out of over 5,000 human players.
The Gap: It beat the second-place human by a huge margin. In fact, the gap between the AI and the second-place human was the same size as the gap between the second-place human and the 25th-place human.
Head-to-Head: When they played directly against the top two human players, the AI won 199 games and lost only 70.
Against the Old Bots: It also crushed the previous best AI and the best "rule-based" bot (which uses math formulas instead of learning) with a perfect 100% win rate.

5. What the AI "Saw"

The researchers looked inside the AI's brain to see if it was thinking like a human.

The "Fog" Detector: They found one specific part of the AI's brain that seemed to track where the enemy General was hiding.
The Analogy: Imagine you are playing a game of hide-and-seek in the dark. At first, you think the person could be anywhere. As you hear a noise or see a shadow, you narrow it down to a few spots. Finally, you spot them. The AI's brain did exactly this: it started with a guess, narrowed it down as it gathered clues, and finally "locked on" to the enemy General's location, even though it couldn't see them directly.

Summary

The paper proves that you don't need complex, hand-crafted rules or massive amounts of human data to create a superhuman AI for strategy games. If you have a fast enough simulator and a simple "Win or Lose" reward, a standard learning algorithm can figure out the rest on its own. The key ingredients were speed, patience (averaging the results), and focusing only on the most informative games.

Technical Summary: Superhuman AI for Generals.io Using Self-Play Reinforcement Learning

Problem Statement
The paper addresses the challenge of creating a superhuman AI agent for Generals.io, a real-time strategy (RTS) game characterized by strong imperfect information (fog of war), simultaneous moves, and a need for both long-horizon planning and short-term tactics. Unlike complex RTS games like StarCraft II or Dota 2, which require massive computational resources and often rely on dense, hand-tuned reward shaping or complex outer-loop population management (e.g., league systems), Generals.io presents a "lightweight" but strategically rich environment. The core research question is whether a simplified training pipeline—specifically a single, well-tuned inner loop using policy gradients with sparse win/loss rewards—can achieve superhuman performance without the need for behavior cloning, reward shaping, or complex outer-loop mechanisms.

Methodology
The authors propose a training pipeline centered on Proximal Policy Optimization (PPO) with self-play, supported by a high-performance simulation environment.

JAX-Native Simulator: A primary enabler of this work is a custom-built simulator written in JAX. By expressing the entire pipeline (observations, rewards, fog-of-war masking, combat resolution) as pure functions over a flat PyTree, the authors achieved end-to-end JIT compilation. This allows the simulation to run on a single NVIDIA H200 GPU at speeds reaching 50.7 million frames per second (with 32,768 parallel environments), representing a roughly 10,000× speedup over prior CPU-based baselines.
Agent Architecture: The agent utilizes a Vision Transformer policy.
- Input: The observation tensor ( $38 \times H \times W$ ) is split into $3 \times 3$ spatial patches. Additionally, two non-spatial tokens encode global temporal statistics (sliding windows of the opponent's total army and land counts) to help infer hidden information behind the fog of war.
- Output: A policy head produces a distribution over 9 actions per cell (pass, or move all/half units in 4 cardinal directions).
Training Objective: The agent is trained end-to-end using PPO with a sparse terminal reward ( $+1$ for win, $-1$ for loss). The objective includes an entropy bonus to prevent premature strategy collapse but explicitly avoids potential-based reward shaping or behavior cloning.
Key Training Techniques:
- Spawn-Distance Curriculum: Training begins with generals placed close together (4 cells apart) to ensure frequent game endings and learnable signals, gradually increasing the distance to the full game scale (17+ cells).
- Parameter Exponential Moving Average (EMA): Instead of deploying the final iterate of the policy parameters, the agent uses an EMA ( $\bar{\theta}_t = \tau \bar{\theta}_{t-1} + (1-\tau)\theta_t$ ) during inference, which consistently outperforms the last iterate.
- Top-Advantage Filtering: The training loop filters collected transitions, retaining only the top 25% of samples ranked by the critic's predicted advantage. This improves both wall-clock time and sample efficiency.

Key Contributions

JAX Environment: The release of a vectorized, JAX-native Generals.io environment supporting 1v1, 2v2, and free-for-all modes with a shared interface, enabling high-throughput multi-agent research.
Superhuman Agent: A single transformer policy trained purely via self-play with sparse rewards that reaches #1 on the public 1v1 leaderboard (over 5,000 human players).
Ablation Study: A systematic isolation of training components demonstrating that:
- Reward Shaping is Unnecessary: At high throughput, sparse win/loss rewards suffice, whereas reward shaping can destabilize late-stage training.
- EMA is Critical: The EMA of parameters is a consistently stronger deployment policy than the last iterate.
- Filtering Efficiency: Top-advantage filtering significantly boosts efficiency.
- Robustness: The method is robust to variations in learning rate and entropy coefficient schedules, suggesting that specific annealing schedules are less critical than the overall range.

Results
The agent, named "Average Joe," was trained for four days on 4× NVIDIA H200 GPUs.

Leaderboard Performance: It achieved a rating of 118 OpenSkill points, leading the second-ranked human (101 points) by a margin equal to the gap between the top human and the 25th-ranked human.
Head-to-Head Records:
- Against the top-ranked human (shimatetsu): 27–12 (69.2% win rate).
- Against the second-ranked human (Mithraaaa): 172–58 (74.8% win rate).
- Against the strongest prior non-learning bot (Human.exe) and the previous state-of-the-art learning bot (zero v3): 100% win rate (20–0 against each).
Statistical Significance: The combined record of 199–70 against top humans is statistically significant ( $p < 0.001$ ) against a fair coin.

Significance and Claims
The paper claims to demonstrate that for a large-scale imperfect-information RTS game, a generic policy-gradient loop is sufficient to reach superhuman performance without the need for:

Complex outer-loop population management (e.g., leagues, double-oracle).
Behavior cloning from human data.
Hand-tuned, dense reward shaping.

The authors highlight that once the data bottleneck is removed via a fast simulator, the "recipe" for success simplifies to a well-tuned inner loop with specific stabilizers (EMA) and efficiency filters (top-advantage). The work serves as a reference point for the scalability of policy-gradient methods in multi-agent systems, suggesting that the complexity of outer loops may be less essential than previously thought for certain game classes. The authors also note that the specific attention head in their transformer appears to learn a coherent posterior belief over the opponent's general location, offering a glimpse into the agent's internal reasoning regarding imperfect information.