Technical Summary: MINDGAMES: A Live Arena for Evaluating Social and Strategic Reasoning in Multi-Agent LLMs

1. Problem Statement

Large language models (LLMs) are increasingly deployed as interactive agents, yet their capacity for social and strategic reasoning over extended, multi-turn interactions remains poorly understood. Existing evaluations rely heavily on static vignettes or single-game benchmarks that fail to capture the sustained, multi-faceted reasoning required in real-world multi-agent settings. Specifically, current benchmarks often lack:

• Dynamic Interaction: The ability to model belief attribution, opponent adaptation, and deception in real-time.
• Complementary Reasoning Demands: A unified framework testing belief attribution under hidden information, opponent modeling, cooperative inference, and sustained deception simultaneously.
• Reproducible Data: Large-scale trajectory corpora for post-hoc analysis and training.

The paper argues that measuring progress toward "Theory of Mind" (ToM) in LLMs requires settings where agents must reason about and respond to other agents over sustained interaction, rather than in isolated, scripted tasks.

2. Methodology: The MINDGAMES Platform

MINDGAMES is a multi-game arena and evaluation platform built on TextArena, designed to operationalize complementary reasoning demands relevant to ToM.

2.1 Game Environments

The benchmark spans four distinct game environments, selected along axes of information structure and incentive alignment:

1. Colonel Blotto (2 players, Zero-Sum): Tests opponent modeling under hidden information. Players allocate troops across battlefields without communication, requiring inference of opponent strategies.
2. Iterated Prisoner's Dilemma (3 players, Mixed-Motive): Tests trust and betrayal through communication. Agents engage in free-form chat followed by private cooperation/defection decisions.
3. Codenames (4 players, Cooperative): Tests collaborative inference under constrained signaling. Teams must deduce hidden word associations via one-word clues, requiring anticipation of teammate interpretation.
4. Secret Mafia (6 players, Asymmetric Info): Tests sustained deception and deduction. Players have hidden roles (Mafia vs. Villagers) and must navigate public debate and private coordination to eliminate opponents.

2.2 Evaluation Protocol

• Infrastructure: Built on TextArena, providing a unified text-only API, online matchmaking, and full trajectory logging.
• Rating System: Uses TrueSkill, a Bayesian skill rating system, to handle variable match sizes (2–6 players) and team-based environments, providing a more robust metric than raw win rates.
• Competition Structure (NeurIPS 2025):
  • Tracks: Generalization Track (Blotto, IPD, Codenames) and Social Deduction Track (Secret Mafia).
  • Divisions: Efficient (≤9B parameters) and Unlimited (no cap).
  • Stages: Stage I (Open Online Ladder) for iterative development; Stage II (Final Evaluation) with frozen submissions and reset ratings to ensure fair comparison.

2.3 Data Release

The competition generated a dataset of 29,571 games (94,132 player trajectories, 243M tokens) containing turn-level observations, actions, rewards, and metadata. Additionally, the MindGames Reference Set (MG-Ref) was released: a frozen pool of top-ranked, low-error Stage II submissions paired with a deterministic offline tournament protocol to allow future agents to be scored under the same measurement lens.

3. Key Results and Findings

3.1 Agent-Level Design Patterns

Analysis of 944 submissions from 76 teams revealed distinct success patterns:

• Training vs. Inference: In the Efficient division, fine-tuning (SFT or RL) on game trajectories was the dominant strategy among top performers. In the Unlimited division, top agents relied more on inference-time scaffolding (code execution, state tracking, modular pipelines) and stronger base models, though the top-ranked agent (In2AI) used RL-trained 8B models in both divisions.
• Data Curation: Quality filtering proved more critical than raw data volume. Aggressive filtering (e.g., retaining only high-quality, strategic steps) outperformed training on larger, noisier corpora.
• Cognitive Scaffolding: Adding memory or structured reasoning modules to models without corresponding fine-tuning often degraded performance. These structures were most effective when paired with SFT.
• Modular Architectures: Successful systems frequently separated perception, belief updating, and action generation into distinct components (e.g., dual-agent architectures separating situation assessment from action selection).

3.2 Failure Modes

• Rule Adherence: Brittle adherence to game rules remained a major bottleneck. Error rates varied significantly by environment:
  • IPD: 0% error rate (due to auto-correction).
  • Colonel Blotto: 8.5% error rate (mostly formatting).
  • Codenames: 38.6% error rate (illegal clues).
  • Secret Mafia: 50.3% error rate (invalid moves, stale state tracking).
• State Tracking: In Secret Mafia, models frequently acted on outdated information (e.g., voting for eliminated players), indicating struggles with long-horizon state maintenance.
• Information Leakage: In social deduction, models often inadvertently revealed hidden roles through reasoning traces or public communication.

3.3 Evaluation Validity and Confounds

A critical contribution of the paper is the audit of leaderboard validity, revealing that evaluation reliability is strongly environment-dependent:

• IPD & Colonel Blotto: Provide relatively interpretable signals of strategic ability.
• Codenames: Rankings mix strategic skill with rule-adherence robustness.
• Secret Mafia: Exhibits a pronounced "error-survival confound." Because many errors are non-fatal (auto-corrected) and games often terminate early due to opponent failures, high rankings often reflect robustness to opponent errors rather than superior strategic deception. Top models in this track won largely because opponents failed, not because they executed superior strategies.

4. Significance and Claims

The paper positions MINDGAMES not merely as a benchmark, but as a reproducible resource for studying the limits of multi-agent reasoning. Its primary claims are:

1. Contextualizing Progress: The paper argues that agentic LLMs have not "solved" rich social reasoning. Instead, they exhibit partial and environment-dependent capabilities that become visible only when interaction mechanics are sufficiently reliable.
2. Engineering over Scale: Success in structured interactive settings often depends more on principled system design (action validity control, state maintenance, reward attribution) than on raw model parameter count.
3. Measurement Awareness: The paper emphasizes that live multi-agent leaderboards do not measure a single stable construct. It provides a framework for distinguishing when a score reflects genuine strategic ability versus artifacts of the environment (e.g., error survival).
4. Resource Availability: By releasing the dataset, the MG-Ref reference set, and the offline tournament protocol, MINDGAMES enables future research to bypass the "farming" incentives of live leaderboards and focus on robust, reproducible evaluation of social and strategic reasoning.

The authors conclude that future progress in this field will require not only stronger models but also more reliable environments and sharper measurement protocols that explicitly account for failure dynamics and evaluation confounds.