Evaluating Language Models' Evaluations of Games

1. Problem Statement

Traditional AI evaluation focuses on problem-solving capabilities (e.g., playing chess, Go, or solving math problems). However, human reasoning involves a higher-order cognitive skill: problem evaluation—the ability to assess whether a problem, system, or game is worth engaging with before attempting to solve it.

The authors identify a gap in current AI research: while models are tested on how they play games, they are rarely tested on how they evaluate games. Specifically, the paper addresses two dimensions of evaluation difficulty:

1. Difficulty to Compute: How computationally intensive is the query? (e.g., calculating expected payoff requires simulating game states).
2. Difficulty to Quantify: How ambiguous is the metric? (e.g., defining "funness" is subjective and lacks a single ground truth).

The core research question is: To what extent can modern language models (LMs) and reasoning models evaluate the fairness (payoff) and "funness" of novel games, and how do these evaluations align with human judgments and game-theoretic optima?

2. Methodology

Dataset and Stimuli

• Novel Games: The study utilized a corpus of 121 two-player competitive strategy board games (from Collins et al., 2025). These are largely novel variants of Tic-Tac-Toe (e.g., different board sizes, win conditions, handicaps) designed to be unlikely to exist in training data, forcing models to rely on reasoning rather than memorization.
• Human Baseline: Approximately 20 human participants evaluated each game as "novices" (before playing) on two queries:
  1. Expected Payoff (Fairness): The likelihood of Player 1 winning vs. a draw.
  2. Funness: A subjective rating of how enjoyable the game is.
• Total Data: Over 450 human judgments across the dataset.

Models Evaluated

The authors tested a wide range of models, categorized by their reasoning capabilities:

• Non-Reasoning LMs: Standard models (e.g., GPT-4, DeepSeek-v3, LLaMA 3.1) prompted with or without Chain-of-Thought (CoT).
• Reasoning Models: Advanced models with explicit reasoning traces (e.g., OpenAI o1, o3, GPT-5, DeepSeek-R1, Gemini 2.5 Pro/Flash).
• Computational Baselines:
  • Game-Theoretic Optimal: Calculated where possible (78 of 121 games).
  • Simulated Agents: "Intuitive Gamer" (heuristic-based), "Expert Gamer" (depth-5 tree search), and Monte Carlo Tree Search (MCTS).

Evaluation Metrics

• Alignment: $R^2$ correlation between model predictions and human judgments.
• Rationality: Accuracy and deviation from the estimated game-theoretic optimal payoff.
• Resource Usage: Analysis of reasoning token counts to measure computational cost.
• Trace Analysis: Coding of reasoning traces to identify strategies (e.g., explicit simulation, analogical reasoning, mathematical computation).

3. Key Contributions

1. New Evaluation Paradigm: Proposes a framework for assessing AI not just on solving problems, but on evaluating whether problems are worth solving. This introduces a meta-reasoning layer to AI assessment.
2. Formalism for Evaluation Difficulty: Explicitly categorizes evaluation queries by their computational complexity (hard to compute) and quantification ambiguity (hard to quantify), providing a structured way to benchmark AI reasoning.
3. Large-Scale Empirical Study: Provides the first large-scale comparison of LMs and reasoning models against human novices and game-theoretic optima on novel game evaluations.
4. Resource-Rationality Analysis: Investigates the relationship between reasoning effort (token usage) and evaluation quality, highlighting the need for resource-rational meta-reasoning in AI.

4. Key Results

A. Expected Payoff (Fairness)

• Reasoning vs. Non-Reasoning: Non-reasoning models (even with CoT) showed poor alignment with human judgments and game-theoretic optima. They relied heavily on statistical inductive biases from training data.
• Reasoning Models: Models with explicit reasoning (e.g., o3, DeepSeek-R1) significantly improved alignment with both human judgments and game-theoretic optima.
• Non-Monotonic Alignment: A critical finding is a non-monotonic relationship between model sophistication and human alignment.
  • As models became more sophisticated (GPT-4 $\to$ o1 $\to$ o3), they aligned better with humans.
  • However, as models approached the game-theoretic optimal (e.g., GPT-5), their alignment with human judgments degraded. GPT-5 was highly rational but "too rational" to match human novice intuitions.
• Simulation Evidence: Trace analysis suggests that high-performing reasoning models engage in some form of explicit game simulation, though the frequency varies.

B. Funness Evaluation

• Quantification Challenge: Evaluating "fun" proved harder than evaluating payoff. Results were "jagged" across models; more advanced models did not consistently outperform less advanced ones.
• Human Variability: Human judgments on funness showed high variance (e.g., bimodal distributions), which most models failed to capture.
• Factors: Models varied in which factors they considered (balance, challenge, length, novelty). While most considered "challenge" and "strategic richness," they differed significantly on "balance" and "length," leading to inconsistent scalar scores.

C. Resource Usage

• Unpredictable Costs: Reasoning models exhibited highly variable token usage across games and queries.
• Lack of Correlation: There was no strong correlation between the number of reasoning tokens used and the accuracy of the evaluation (neither against humans nor the game-theoretic optimal).
• Novelty vs. Compute: Counter-intuitively, more "novel" games (those further from standard Tic-Tac-Toe) did not consistently demand more compute tokens, suggesting models do not dynamically adapt resources based on problem complexity.

5. Significance and Implications

• AI Alignment & Thought Partners: For AI to serve as effective "thought partners" (e.g., in education or mechanism design), it must align with human values and intuitions, not just mathematical optimality. The finding that "more rational" models can be less human-aligned is crucial for designing beneficial AI.
• Meta-Reasoning: The results highlight a deficiency in current models: they lack resource-rational meta-reasoning. They do not dynamically adjust their computational effort based on the difficulty of the evaluation query.
• Future Directions: The paper calls for:
  • Developing agents that can simulate different levels of rationality (e.g., simulating a "novice" vs. an "expert").
  • Creating benchmarks for problem-evaluation capabilities.
  • Investigating how to control the "precision" of AI evaluations to match specific human contexts.

In conclusion, this paper shifts the focus from AI as a solver to AI as an evaluator, revealing that while reasoning models are powerful, their ability to align with human subjective judgments (especially on ambiguous metrics like fun) is complex and non-linear, requiring new approaches to meta-reasoning and resource allocation.