🤖 AI

GENSTRAT: Toward a Science of Strategic Reasoning in Large Language Models

This paper introduces GENSTRAT, a framework using procedurally generated strategic environments and a multi-axis capability profile to evaluate large language models' strategic reasoning, revealing that models with similar overall performance can exhibit distinct strengths and unpredictable behavioral volatility in specific deployment-relevant scenarios.

Original authors: Vartan Shadarevian, Kia Ghods, Alex Kenich, Anany Kotawala

Published 2026-05-25

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Vartan Shadarevian, Kia Ghods, Alex Kenich, Anany Kotawala

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a team of expert poker players to run a high-stakes casino. You want to know who is the best.

The Old Way: The "Fixed Menu" Problem
Traditionally, to test these players, you'd give them a fixed menu of five famous poker games (like "Texas Hold'em" or "Kuhn Poker"). You'd watch them play, tally the wins, and rank them.

The problem?

They memorized the menu: If the players studied those exact five games before the test, they aren't showing true skill; they're just reciting answers they already know.
The menu is too small: Just because someone is great at those five specific games doesn't mean they can handle the messy, weird, and unpredictable games that might actually show up in a real casino.

The New Way: GENSTRAT (The "Infinite Slot Machine")
The authors of this paper built a new testing ground called GENSTRAT. Instead of a fixed menu, they built a "game generator"—like a slot machine that never runs out of new, unique poker games.

Fresh Games on Demand: Every time they want to test a model, the machine spins and creates a brand-new game with different rules, card decks, and betting phases. No model can memorize the answers because the questions are always changing.
The "Six-Dimensional" Report Card: Instead of giving a single score (like "90/100"), GENSTRAT gives a detailed profile across six specific "axes" of difficulty:
1. State Space: How many different situations can happen? (Is the game simple or chaotic?)
2. Temporal Depth: Do early moves matter later? (Do you need to plan 10 steps ahead, or just react to the next card?)
3. Information Sensitivity: Does knowing your secret hand change your strategy?
4. Opponent Modeling: Do you need to guess what the other player is thinking?
5. Risk: Is it a safe game, or do you have to gamble big for a big reward?
6. Brittleness: Is the game "fragile"? If you make a tiny mistake, do you lose everything?

The Tournament: A 36,000-Hand Showdown
The researchers pitted 9 of the world's smartest AI models against each other in over 36,000 matches. Here is what they found:

The "Average" Winner: Newer, bigger models generally won more chips on average.
The "Specialist" vs. The "Generalist": Two models might have the same overall score, but their "profiles" look totally different.
- Example: One model (Claude) was amazing at "Brittle" games (where precision matters) but average elsewhere. Another (Gemini) was strong across almost all categories.
- Analogy: It's like two runners having the same total race time, but one is a sprinter who excels on straight tracks, while the other is a marathoner who excels on hilly terrain. If you only look at the total time, you miss the nuance.

The "Jaggedness" Test: The Bumpy Road
The paper introduced a new concept called Jaggedness.

Imagine driving a car. A "smooth" car handles bumps in the road consistently. A "jagged" car handles a bump perfectly, then hits the next bump and swerves wildly, even though the bumps are identical.
The researchers found that some top-tier models were "jagged." They performed incredibly well on one specific game, but then did surprisingly poorly on a very similar game right next to it.
Why it matters: If you deploy a "jagged" model in the real world, you might get great results today, but a slightly different situation tomorrow could cause a crash. A "smooth" model is more predictable and reliable.

The "Thinking" Test
They also checked if telling the AI to "think harder" (using more computing power) helped.

For most models, thinking longer did help them win more chips.
However, for some models, the extra thinking didn't seem to make a statistically significant difference, suggesting they might have hit a ceiling or that the extra effort wasn't being used efficiently.

The Bottom Line
The paper argues that simply ranking AI models by "who won the most" is dangerous. To truly understand how an AI will behave in the real world, you need to know:

What kind of problems it is good at (its capability profile).
How consistent it is when the situation changes slightly (its jaggedness).

GENSTRAT provides a way to see these details, ensuring that when we put AI into real-world markets or auctions, we aren't just hiring the model with the highest score, but the one that is actually reliable for the specific job.

Technical Summary: GENSTRAT

Problem Statement

Large language models (LLMs) are increasingly deployed as economic agents in marketplaces, auctions, and bidding settings. However, anticipating their behavior in specific deployments is difficult. Existing strategic-reasoning benchmarks rely on fixed canonical games (e.g., Poker variants, Diplomacy, Avalon). These benchmarks face two primary limitations:

Saturation and Contamination: As model capabilities improve, performance on fixed suites may saturate. Furthermore, the proximity of these canonical games to training data makes it difficult to rule out corpus contamination.
Limited Generalizability: Reducing strategic competence to performance on a small set of fixed games limits a deployer's ability to generalize to novel, messy strategic environments where information structures and variations differ significantly from the benchmark.

Methodology

The authors introduce GENSTRAT, a benchmarking framework based on procedurally generated strategic environments to address these limitations.

1. Procedural Game Generation

GENSTRAT utilizes a generator for Generalized Betting Games (GBGs), defined as two-player zero-sum extensive-form games with imperfect information.

Structure: Games consist of decks, private hands, card piles, structured phases (betting, simultaneous moves, auctions, observation), and conditional branches.
Randomization: The engine randomizes both the structural composition (phase graphs) and surface features (ranks, suits, hand sizes, betting orders).
Quality Control: Generated games undergo Monte Carlo simulations with random-playing agents to ensure playability. Acceptance criteria include limits on average moves per player, phase firing rates, and the elimination of dead branches.
Pool: From 12,351 candidate seeds, a pool of 2,000 accepted games was generated.

2. Complexity Axes and Benchmark Construction

To characterize game complexity and model performance, the authors define six axes derived from Monte Carlo simulations:

State Space: Logarithm of distinct observable information states.
Temporal Depth: The extent to which early decisions affect later payoffs.
Information Sensitivity: How strongly the optimal action depends on private information.
Opponent Modeling: The degree to which the best response shifts with changes in opponent policy.
Risk: The tradeoff between expected value and downside risk.
Brittleness: The sensitivity of payoffs to small policy perturbations.

From the 2,000-game pool, a 50-game benchmark was selected using Farthest-Point Sampling (FPS) in the six-axis embedding space to ensure broad coverage of the complexity landscape.

3. Evaluation Protocol

Models: Nine frontier and open-weight LLMs were evaluated, including gpt-5-4-high, gemini-3.1-pro-preview, claude-sonnet-4-6-max, and others.
Tournament Design: A head-to-head tournament involving over 36,000 matches. For each (Model A, Model B, Game) triplet, 40 matches were played, with each model occupying each seat (Alice/Bob) 20 times to control for seat asymmetry.
Strength Estimation: Model strength ( $\hat{\alpha}$ ) is estimated using an additive paired-comparison model on signed win margins (chips per game), constrained to sum to zero across models.

4. Diagnostic Metrics

Beyond a single leaderboard, the framework introduces:

Capability Profiles: Per-model OLS regressions of per-game strength against the six complexity axes. This decomposes overall strength into specific strategic competencies (e.g., performance in high-risk vs. high-brittleness games).
Local Jaggedness ( $J_m$ ): A measure of the smoothness of a model's performance surface. It quantifies how much a model's win margin fluctuates between strategically similar games (nearest neighbors in axis space). High jaggedness indicates unpredictable performance on games slightly different from the benchmark.

Key Results

Overall Leaderboard: Newer, larger, and reasoning-capable models generally scored higher. The top three models were gpt-5-4-high (+0.85 chips/game), gemini-3.1-pro-preview (+0.83), and claude-sonnet-4-6-max (+0.64). The ranking was stable across leave-one-game-out and complexity-tertile splits.
Divergent Capability Profiles: Models with near-identical overall strength exhibited qualitatively different profiles:
- gemini-3.1-pro-preview gained ground across the broadest set of axes, including state space and opponent modeling.
- claude-sonnet-4-6-max derived most of its advantage specifically from brittleness, showing a steep slope in high-brittleness games but flatter performance elsewhere.
- gpt-5-4-high showed strong performance on state space and information sensitivity but was notably locally jagged (high $J_m$ ), suggesting volatility in similar game contexts.
Jaggedness Findings: llama-3.3-70b-together was the most locally jagged model (lowest smoothness), while deepseek-v3.1-together was the smoothest. High jaggedness was observed even in top-tier models, indicating that average strength does not guarantee consistent performance across the procedural distribution.
Reasoning Ablation: Increasing reasoning budget (thinking mode) yielded positive point estimates for chip-margin returns across all four model families tested (gpt-5, claude, gemini-3.1-pro, gemini-2.5-pro). Two families showed statistically significant gains, while the others were underpowered by sample size rather than showing null effects.

Significance and Claims

The paper argues that overall ranking alone is insufficient for deployment decisions. A model's strategic capability is best understood as a full performance profile across different regions of the procedural game space, combined with its level of local smoothness (jaggedness).

Deployment Relevance: The capability profile and jaggedness measure provide a diagnostic that the overall ranking cannot. For instance, a model with high overall strength but high jaggedness may be less reliable in a deployment environment that lies just outside the benchmark distribution.
Evergreen Evaluation: Because the generator can draw fresh games on demand from the same distribution, the benchmark resists saturation and contamination. It can scale with the model frontier by relaxing complexity caps.
Generalization: The framework moves beyond fixed canonical games to evaluate how models generalize to novel strategic environments, addressing the "messiness" of real-world economic agent deployments.

The authors conclude that GENSTRAT reframes strategic-reasoning evaluation around a procedurally generated distribution and a multi-axis decomposition, allowing for a more nuanced understanding of model strengths, weaknesses, and stability.