Technical Summary: Social Gym and SPARTAN

Problem Statement

Large Language Model (LLM) agents are increasingly deployed in multi-agent social settings requiring cooperation, negotiation, and adaptation. However, measuring and improving these social skills faces three critical limitations in existing evaluations:

1. Static Benchmarks: Many evaluations rely on static theory-of-mind questionnaires (e.g., FANToM, ToMi) that produce reproducible scores but fail to test sustained multi-turn behavior.
2. Subjective Judging: Open-ended interactive evaluations (e.g., SOTOPIA) assess multi-turn skills but rely on LLM-as-judge scoring, which is prone to position, verbosity, and self-enhancement biases, making results noisy and subjective.
3. Narrow Scope: Recent works using verifiable rewards often target single games or domains in isolation, failing to capture the breadth of social intelligence across different interaction structures.

There is a gap for an evaluation framework that is simultaneously multi-turn, broad in domain coverage, and verifiable (determined by interaction rules rather than subjective judgment). Furthermore, it remains unclear whether LLMs can improve their social reasoning capabilities without parameter updates (weight changes).

Methodology

Social Gym: A Verifiable Benchmark

The authors introduce Social Gym, an environment comprising 21 multi-agent social games designed to test social reasoning across five categories:

1. Normal-Form Games (6): Iterated matrix games with complete information and no communication (e.g., Prisoner's Dilemma, Chicken).
2. Economic Games (3): Multi-round resource allocation requiring strategic reasoning (e.g., Public Goods, Centipede).
3. Bluffing Games (4): Hidden-state games requiring misrepresentation or inference (e.g., Liar's Dice, Coup).
4. Hidden-Role Deduction (6): Games with secret role assignments requiring dialogue-based identification of allies/enemies (e.g., Werewolves, Resistance, Spyfall).
5. Social Strategy (2): Complete-information games relying on alliance formation and reputation (e.g., Survivor).

Key Architectural Features:

• Verifiable Outcomes: Every game has an algorithmically decided outcome (win/loss/score), eliminating the need for LLM judges.
• Partial Observability: A visibility layer filters messages into Public, Team-Private, or Private scopes, forcing agents to perform Theory-of-Mind reasoning.
• Unified Elo Tournament: A Bradley-Terry maximum-likelihood fit generates per-game win-rate tables and a cross-game leaderboard, allowing for objective ranking of models across diverse social structures.

SPARTAN: Training-Free Self-Improvement

To address whether LLMs can improve without weight updates, the authors propose SPARTAN (Self-Play and Reflect-Transfer), a three-stage loop:

1. Play: The model plays $N$ self-play games of a specific game $G$, generating trajectories.
2. Reflect: The model analyzes its own trajectories and outcomes to generate a first-person, transferable strategic playbook (covering deception, detection, persuasion, etc.). Crucially, the playbook is instructed to be game-agnostic (no references to specific game numbers).
3. Transfer: The generated playbook is prepended to the agent's system prompt for subsequent games.

This approach functions as "in-context fine-tuning" where the "training data" is the model's own gameplay and the "learned weights" are natural-language rules.

Key Results

Benchmarking Results (Social Gym)

• Leaderboard Inversions: While GPT-5-mini tops the overall leaderboard (1110 Elo), no model excels uniformly across all games or roles. Rankings invert sharply; for instance, Qwen3-32B ranks first in the specific game "Chicken" (1328 Elo) but last in "Werewolves" (817 Elo) among the seven evaluated models.
• Role Asymmetry: In hidden-role games, models often perform differently on minority/deceptive roles (Alt) versus majority/cooperative roles (Main). Generally, the minority role is structurally easier to win against a mixed pool of opponents, but this advantage reverses in matched-capability self-play for stronger models.
• Model Limitations: Smaller models (e.g., Qwen2.5-3B) exhibit a "parroting effect," frequently paraphrasing the previous speaker rather than generating independent arguments, contributing to lower performance.

SPARTAN Evaluation Results

The authors evaluated SPARTAN across four axes: within-game iteration, cross-game transfer, multigame transfer, and cross-model distillation.

1. Within-Game Iteration (GPT-5-mini):

   • Injecting a playbook ($R_1$) significantly lifts the win rate of the structurally weaker side (Alt) in asymmetric games (e.g., Werewolves Alt win rate rose from 23% to 36%).
   • Conversely, the structurally stronger side (Main) often sees a decline in performance when armed with the same playbook.
   • Gains are non-monotonic; the bulk of improvement occurs at $R_1$, with subsequent rounds ($R_2$–$R_4$) redistributing rather than accumulating gains.

2. Cross-Game and Multigame Transfer:

   • Playbooks generated from one game (e.g., Werewolves) transfer effectively to the weaker side of other hidden-role games (e.g., Spyfall, Resistance), often improving win rates by +7 to +27 percentage points.
   • Multigame playbooks (trained on multiple sources) do not consistently outperform single-game playbooks, suggesting that a single related training game provides sufficient signal for transfer.

3. Cross-Model Distillation:

   • Playbooks generated by GPT-5-mini successfully distill to weaker student models (e.g., Qwen3-4B, GPT-4o-mini), lifting their performance on structurally weaker roles (e.g., Spies in Resistance) by +13 to +24 percentage points.
   • The effect is role-dependent, not student-dependent: the playbook helps whichever model plays the disadvantaged role, regardless of the student's base capability.

4. Capacity Dependence (Qwen3-32B):

   • When applied to the open-weights Qwen3-32B, SPARTAN largely fails to improve performance in complex social deduction games (Werewolves, Spyfall).
   • The only exception is the Prisoner's Dilemma, where the playbook prescribes a discrete action (defect on the final round) that the model can execute.
   • In discussion-heavy games, Qwen3-32B fails to break its parroting behavior; the reflection process often codifies the parroting into the playbook rather than eliminating it.

Significance and Claims

The paper claims two primary contributions:

1. Social Gym provides a reproducible, verifiable foundation for measuring LLM social reasoning without relying on subjective LLM judges. It reveals that social ability is not a single scalar capability but is highly dependent on interaction structure, role, and game category.
2. SPARTAN demonstrates that LLMs can improve social performance without parameter updates by extracting transferable strategies from self-play. However, this improvement is capacity-dependent and structure-dependent: it effectively lifts structurally weaker roles in complex games for high-capacity models but fails for models that lack the reasoning capacity to process long-horizon social trajectories or break parroting patterns.

The authors conclude that Social Gym and SPARTAN offer a framework to separate structural properties of social settings from model-specific failures (e.g., weak deception, poor coalition tracking). They suggest this environment is a natural testbed for future Reinforcement Learning with Verifiable Rewards (RLVR) research, enabling the training of social reasoning skills at scale without LLM judges.

Limitations Acknowledged:

• External Validity: All games have fixed rules and win/loss criteria, which may not fully capture real-world social interactions like long-term trust building.
• Sample Size: Results are based on ~30 games per condition, yielding confidence intervals of roughly ±18 percentage points.
• No Placebo Control: The study lacks a placebo-playbook control to fully disentangle content effects from generic prompt perturbations, though structural patterns suggest the effects are content-driven.
• Reflection Constraints: The reflection is limited to natural language prose within the system prompt, lacking external retrieval or structured reasoning tools.