Technical Summary: RTSGameBench

Problem Statement

Modern Vision-Language Models (VLMs) have demonstrated success in various tasks but struggle with strategic reasoning, defined as the ability to anticipate and influence other agents' actions under uncertainty in competitive and cooperative settings. While Real-Time Strategy (RTS) games are a natural testbed for this capability due to their demands for long-horizon planning, partial observability, and multi-agent coordination, existing benchmarks suffer from three critical limitations:

1. Limited Evaluation Scope: Most benchmarks (e.g., those based on StarCraft II) focus primarily on 1v1 scenarios, neglecting cooperative and multi-agent dynamics essential for strategic reasoning.
2. Lack of Systematic Diagnosis: Full-game evaluations often conflate distinct strategic competencies, making it difficult to attribute success or failure to specific capabilities (e.g., resource management vs. opponent modeling).
3. Fixed Scenario Coverage: Existing benchmarks rely on static, pre-designed scenarios, preventing the evaluation of generalization and limiting the ability to expand diagnostic coverage on demand.

Furthermore, operating in large-scale RTS environments poses a technical challenge for VLMs: the sheer number of units and long game durations require scalable coordination and sustained coherence, which standard per-unit control interfaces cannot efficiently manage.

Methodology

The authors propose RTSGameBench, a comprehensive benchmark and evaluation platform built on Beyond All Reason (BAR), a large-scale open-source RTS game. BAR offers significantly expanded scales compared to StarCraft II, supporting up to 100 players, 554 unique units, and a per-player unit cap of 2,000, thereby enlarging the strategic space and requiring longer-horizon planning.

The methodology integrates three core components:

1. Full Game Evaluation

The benchmark evaluates agents across diverse matchup structures to expose distinct strategic demands:

• Duel (1v1): Tests individual decision-making.
• Symmetric Team (2v2, 3v3): Introduces allied coordination.
• Asymmetric Team (3v4): Demands coordination under numerical disadvantage.
• Free-for-All (1v1v1v1): Requires multi-polar threat prioritization.

2. Diagnostic Mini-Games

To enable granular assessment, the authors introduce five mini-games, each targeting a specific strategic competency identified in prior RTS AI taxonomies:

• Resource Management (TCP): Time-constrained production under enemy raids.
• Spatial & Temporal Reasoning (MFD): Defending multiple objectives against staggered attacks with fixed forces.
• Opponent Modeling (FS-F): Free-for-all skirmish requiring prediction of opponent targets.
• Collaboration (FS-T): Team-based skirmish requiring coordination without explicit communication.
• Adversarial Planning (SP): Breaching a static fortification within a time limit.
• Note: Fog-of-war is selectively applied based on whether partial observability is integral to the specific competency being tested.

3. Self-Evolving Game Generation Framework

To address the limitation of fixed scenarios, the authors propose a framework that converts free-form user queries into new diagnostic mini-games. This pipeline utilizes specialized VLM-based agents (Project Manager, Designer, Developer, Analyst) to orchestrate scenario planning, Game Design Document (GDD) generation, rule set construction, and implementation. The framework employs a self-evolution mechanism where a shared knowledge database stores validated artifacts, and a Project Manager conducts retrospective analysis to refine evaluation rubrics, allowing the benchmark to improve and expand over successive cycles.

4. RTSGameAgent (Baseline Agent)

To make large-scale RTS tractable for VLMs, the authors design RTSGameAgent, a baseline agent that combines:

• FSM-based Group Management: Replaces per-unit movement commands with group-level assignments (squads) and Finite State Machine (FSM) commands (move, move_force, stop). This delegates tactical execution (e.g., engaging enemies) to the game engine while the VLM handles strategic group coordination.
• Agentic Memory: A dual-memory system (short-term event logs and long-term experience summaries) managed by an LLM to maintain context across decision steps, mitigating the loss of inter-step information.

Key Results

The authors evaluated multiple state-of-the-art (SoTA) VLMs (including GPT-5.2, Gemini-3-Flash, Claude-4.5-Sonnet, and various open-source models) using RTSGameAgent.

• Performance Degradation with Scale: All models exhibited consistent performance degradation as task scale increased (larger map sizes and higher unit counts). For instance, in Free-for-All modes, top models like Gemini-3-Flash achieved a Rank Score of 0.66, but performance dropped significantly in Asymmetric Team modes (WR ~0.20).
• Coordination Challenges: Models struggled significantly in scenarios requiring tight coordination. In Symmetric Team modes, win rates dropped compared to 1v1 (e.g., Gemini-3-Flash dropped from 0.87 in 1v1 to 0.50 in 2v2). In Asymmetric Team modes, even top models failed to maintain sustained coordination under numerical disadvantage.
• Competency Gaps: Diagnostic mini-games revealed specific weaknesses. While some models approached human performance in Resource Management (TCP), they showed large gaps in Spatial & Temporal Reasoning (MFD) and Opponent Modeling (FS-F). For example, in MFD, the best VLM achieved a Damage Efficiency (DE) of 1.82, far below the human baseline of 3.46.
• Component Analysis: Ablation studies confirmed that both FSM-based group management and agentic memory are critical. Removing both components caused win rates in 1v1 to plummet from 0.87 to 0.10. Group management was identified as the primary bottleneck for scalability.
• Self-Evolving Framework: The framework successfully generated playable mini-games from free-form queries. Over successive batches, the framework improved playability and human preference scores, demonstrating that the self-evolution mechanism effectively refines generation quality and expands coverage.

Significance and Claims

The paper claims that RTSGameBench provides the first holistic, diagnostic, and extensible benchmark for evaluating strategic reasoning in VLMs within large-scale RTS environments. Its significance lies in:

1. Systematic Diagnosis: By isolating specific strategic competencies through mini-games, it allows researchers to pinpoint exact failure modes (e.g., poor opponent modeling vs. poor resource management) rather than relying on aggregate win rates.
2. Scalability: By leveraging BAR and the RTSGameAgent architecture, it enables the evaluation of VLMs in environments with orders of magnitude more units and complexity than previous benchmarks, revealing that current models struggle to scale their reasoning capabilities.
3. Extensibility: The self-evolving generation framework addresses the saturation problem of fixed benchmarks, offering a pathway to continuously expand evaluation coverage based on researcher needs without manual scenario design.

The authors conclude that while VLMs show promise, they currently lack the robustness required for complex, multi-agent strategic reasoning, particularly in scenarios demanding tight coordination and long-horizon planning under uncertainty. The benchmark serves as a rigorous tool to track progress in this domain.