Scaling Strategy, Not Compute: A Stand-Alone, Open-Source StarCraft II Benchmark for Accessible Reinforcement Learning Research

To address the complexity gap between StarCraft II's full game and its mini-games, this paper introduces the Two-Bridge Map Suite, an open-source, lightweight benchmark that isolates tactical navigation and combat skills to enable accessible reinforcement learning research under realistic compute budgets.

The Gist

Imagine you want to teach a robot how to play chess. You have two options:

1. The "Grandmaster" Option: You throw the robot into a massive, 3D chess tournament where it has to manage an entire army, build a base, gather gold, and fight on a map the size of a football field. The problem? It takes a supercomputer the size of a warehouse and millions of dollars in electricity to train the robot. Only a few rich labs can afford this.
2. The "Drill Sergeant" Option: You put the robot in a tiny room with just two pawns. It learns to move them back and forth instantly. The problem? It's too easy. The robot masters it in five minutes, and you learn nothing about how to handle a real battle.

The Problem:
For a long time, researchers studying Artificial Intelligence (AI) in strategy games (like StarCraft II) were stuck in this gap. They either couldn't afford the "Grandmaster" version, or the "Drill Sergeant" version was too boring and didn't teach the AI anything useful about real strategy.

The Solution: "Two-Bridge"
The authors of this paper built a middle-ground training ground called the Two-Bridge Map Suite. Think of it as a "driving test" for AI.

Instead of a full-blown war, imagine a specific, controlled scenario:

• The Map: A canyon with a steep cliff in the middle. The only way to cross is via two narrow bridges.
• The Mission: You have a squad of soldiers (your AI). On the other side, there are two things:
  1. A Beacon (a flag you need to capture).
  2. Enemy Soldiers (who want to shoot you).
• The Rules: You can't build bases or gather gold. You just have to decide: Do I rush the bridge to fight the enemies, or do I sneak around to grab the flag?

Why is this cool?
It strips away the messy, expensive parts of the game (like building economies) and focuses purely on tactics:

• Navigation: Can the AI figure out the shortest path?
• Combat: Can the AI manage its soldiers to win a fight?
• Decision Making: If the enemy is strong, should the AI run away and grab the flag instead?

The "Camera" Trick
One of the paper's clever ideas is how the AI "sees" the world.

• In the full game, the AI has to manually move a camera around the map to see what's happening. This is like asking a driver to steer the car and constantly twist their neck to look out the back window. It's distracting!
• In Two-Bridge, the camera is locked to the soldiers. Wherever the soldiers go, the camera follows automatically. This lets the AI focus entirely on fighting and moving, rather than wasting brainpower on "looking."

What Happened When They Tested It?
The researchers trained standard AI algorithms on this map using a normal gaming laptop (no supercomputers needed).

• The Results: The AI learned to fight and move, but it also developed some funny "bad habits."
  • Sometimes, if the AI lost a few soldiers, it would just run away in a straight line and hide in a corner until the game ended.
  • Sometimes, it would get so obsessed with fighting that it ignored the flag, even when the flag was right there.
• The Takeaway: These "bad habits" are actually great! They show us exactly where current AI is weak. It proves the map is a good test: it's hard enough to make the AI struggle, but simple enough for regular researchers to study why it failed.

The Big Picture
This paper is like opening a public park for AI researchers. Before, only the people with private jets (supercomputers) could play in the big stadium. Now, anyone with a bicycle (a standard laptop) can come to the Two-Bridge park, practice their strategy, and learn how to make smarter AI without needing a massive budget.

It's not about winning the game; it's about building a better, more accessible gym where everyone can train to get stronger.

Technical Summary

Here is a detailed technical summary of the paper "Scaling Strategy, Not Compute: A Stand-Alone, Open-Source StarCraft II Benchmark for Accessible Reinforcement Learning Research."

1. Problem Statement

The Reinforcement Learning (RL) research community faces a significant "complexity gap" in StarCraft II (SC2) environments:

• Full-Game SC2: Offers high strategic depth and long decision horizons but requires extreme computational resources (e.g., AlphaStar used 26 trillion steps), proprietary components, and complex training pipelines, making it inaccessible to most researchers.
• Mini-Games & SMAC: These are lightweight and reproducible but isolate narrow skills (e.g., pure combat or navigation) with short decision horizons. Agents often saturate performance quickly, limiting the study of complex strategic trade-offs.

There is a lack of an intermediate benchmark that balances strategic complexity with computational tractability, allowing researchers to experiment with modern RL algorithms under realistic compute budgets without relying on massive replay datasets or distributed training.

2. Methodology: The Two-Bridge Map Suite

The authors introduce the Two-Bridge Map Suite, a custom-built SC2 environment designed to isolate core tactical skills while removing economic and fog-of-war mechanics.

Core Design Principles

• Environment Simplification: The map disables resource collection, base building, and fog-of-war. This creates a fully observable environment focused purely on long-range navigation and micro-combat.
• Map Layout: The terrain features a vertical cliff bisecting the map, forcing all movement through two narrow bridges (choke points). This creates a "breaching scenario" ideal for studying tactical engagement.
• Entities:
  • Units: Homogeneous groups of Marines (no special abilities or upgrades). Friendly units are controlled by a centralized RL agent; enemy units are controlled by a reactive, centralized built-in AI.
  • Beacon: A static, non-controllable objective that spawns in a specific region.
• Objectives (Multi-Objective RL): Each episode presents two mutually exclusive goals:
  1. Navigation: Capture a beacon.
  2. Combat: Eliminate all enemy units.
  • Note: Agents must learn to arbitrate between these objectives based on the specific map configuration.

Benchmark Variants

The suite consists of 9 map configurations formed by the Cartesian product of two axes:

1. Unit Count Balance:
   • V1 (Easy): Friendly advantage (5 vs 3).
   • V2 (Medium): Balanced (5 vs 5).
   • V3 (Hard): Enemy advantage (5 vs 8).
2. Objective Proximity:
   • Base: Beacon and enemies are equidistant.
   • Combat-Proximal: Enemies are closer than the beacon.
   • Navigate-Proximal: Beacon is closer than enemies.

Technical Implementation

• Interface: Built as a lightweight, Gym-compatible wrapper on top of PySC2.
• Observation Space: A combined vector-spatial representation:
  • Vector: 55-dimensional features (unit positions, health, cooldowns, beacon coordinates, distances).
  • Spatial: 17-channel feature screen and 7-channel minimap (64x64 resolution).
• Action Space: Discrete, unit-level commands with Action Masking.
  • Actions include: No-op, Move (8 directions), Attack (target specific enemy).
  • Masking: Explicitly disables infeasible actions (e.g., cannot attack if no enemies are alive, cannot move if all units are dead) to guide exploration.
• Reward Function: Dense shaping rewards for progress (distance to beacon/enemies, health changes) combined with sparse terminal rewards (Win/Loss/Timeout).
• Camera Control: In Experiment 3, a camera-locked setting is enforced where the screen always centers on friendly units, removing the need for the agent to learn camera movement actions.

3. Key Contributions

1. Intermediate Benchmark: The first open-source SC2 benchmark specifically engineered to sit between mini-games and full-game, offering non-trivial decision horizons without extreme compute costs.
2. Accessibility: A plug-and-play Gym interface allowing standard RL algorithms (e.g., PPO) to be trained on consumer-grade GPUs (approx. 5–6 days per run) without distributed training or imitation learning.
3. Strategic Diagnostics: A structured suite of 9 variants designed to probe specific strategic behaviors, such as how agents arbitrate between numerical advantages and objective proximity.
4. Open Source Release: Full release of maps, wrappers, and reference scripts to encourage broad adoption.

4. Results and Qualitative Analysis

The authors evaluated the benchmark using Maskable PPO (Stable-Baselines3) across the 9 variants.

• Emergent Behaviors:
  • V1 (Friendly Advantage): Agents consistently adopted a "combat-dominant" strategy, ignoring the beacon to engage enemies, relying on numerical superiority rather than tactical positioning.
  • V2 (Balanced): Agents exhibited a pattern of initial engagement followed by deterministic retreat. Navigation success was incidental, occurring only if the beacon lay on the retreat path.
  • V3 (Enemy Advantage): Performance degraded significantly. Agents either fragmented forces (leading to loss) or avoided combat entirely, often getting stuck in local exploration loops.
• Impact of Camera Locking (Exp 3):
  • Enforcing a camera-locked view revealed a "camera-bound failure mode." Agents would move units out of the camera's field of view, fail to engage remaining enemies outside the screen, and eventually timeout. This highlighted a limitation in the agent's ability to maintain global situational awareness when visual input is restricted to a moving viewport.
• General Findings:
  • Agents learned coherent maneuvering and engagement behaviors without full-game costs.
  • However, standard RL baselines struggled with complex strategic trade-offs (e.g., choosing navigation over combat when enemies were numerically superior but distant), indicating the benchmark remains a challenging, unsolved problem for current methods.

5. Significance and Future Work

• Significance: The Two-Bridge Map Suite democratizes SC2-based RL research. It allows a broader community to study strategic decision-making, multi-objective optimization, and tactical coordination without the prohibitive costs of full-game training. It serves as a "middle ground" testbed for developing more sample-efficient and generalizable RL algorithms.
• Limitations: Currently limited to Windows due to PySC2 stability issues; CPU-bound execution restricts GPU-cluster scaling.
• Future Directions: The framework supports extensions such as decentralized multi-agent control, curriculum learning, reintroducing partial observability (fog-of-war), and hierarchical action representations.

In conclusion, the paper successfully demonstrates that strategy can be scaled independently of compute by carefully designing intermediate environments that isolate specific tactical challenges, providing a vital new tool for the RL community.