Interactive Benchmarks

This paper proposes "Interactive Benchmarks," a unified evaluation paradigm that assesses model intelligence through active information acquisition and reasoning under budget constraints in interactive proofs and games, demonstrating that current models still have significant room for improvement in these dynamic scenarios.

The Gist

Imagine you are trying to solve a mystery, but instead of being handed a list of clues, you have to ask for them. You have a limited number of questions you can ask before the game ends. If you ask the wrong questions, you run out of time and fail. If you ask the right ones, you crack the case.

This is the core idea behind the paper "Interactive Benchmarks."

Here is the breakdown of why this matters, what they did, and what they found, using simple analogies.

1. The Problem: The "Cramming" vs. The "Detective"

For a long time, we tested AI models (like the ones powering chatbots) using static benchmarks.

• The Old Way: It's like giving a student a multiple-choice test where they can't ask the teacher for help. They just have to memorize facts or guess the answer based on the question alone.
• The Flaw: AI has gotten so good at memorizing these tests that they are "cheating" by having seen the answers during training. Also, in the real world, problems aren't multiple-choice. You don't just get all the facts at once; you have to go out and find them.

The authors argue that true intelligence isn't just knowing the answer; it's knowing what to ask to get the answer.

2. The Solution: The "Interactive Benchmarks"

The authors created a new way to test AI called Interactive Benchmarks. Instead of a static test, they set up a conversation where the AI has to actively hunt for information within a strict "budget" (a limit on how many turns or questions it can take).

They split this into two main "games":

Game A: The Detective (Interactive Proofs)

• The Scenario: Imagine a "Situation Puzzle" (like a riddle). You are told: "Ah Xing was knocked down by a kid, but he was happy. Why?"
• The Rules: You can't just guess. You have to ask a "Judge" (who knows the truth) Yes/No questions. But you only have 20 questions to solve it.
• The Test: Can the AI figure out the right questions to ask? (e.g., "Was the kid older than him?" "Did the kid look like a student?")
• The Result: When the AI just tried to guess without asking questions, everyone failed (0% accuracy). But when they were allowed to ask questions, some models (like Gemini and GPT-5) started solving them. This proves that asking the right questions is a skill the AI needs to learn.

Game B: The Poker Player & The Diplomat (Interactive Games)

Here, there is no "Judge" with the answer. The AI has to play against others to win.

• Poker (Texas Hold'em): The AI has to bluff, calculate odds, and read the other players' "tells" without knowing their cards. It's like playing poker against a room full of strangers where you have to decide when to bet big and when to fold.
• The Trust Game: Imagine a repeated game of "Cooperate or Betray." If you cooperate, we both win a little. If you betray, you win big and I lose. But if we both betray, we both lose. The AI has to learn: Should I trust this person? Should I punish them for cheating? When should I forgive them?

3. The Big Discovery: "The Gap"

The researchers tested the top AI models (Grok, Gemini, GPT-5, etc.) in these interactive games. Here is what they found:

• The "Cramming" Models are Stuck: Many models are great at answering questions if they have all the info, but they are terrible at figuring out what info they are missing.
• The "Detective" Skill is Rare: In the puzzle game, most models failed completely because they didn't know how to ask questions to narrow down the possibilities.
• The Poker Table: In the poker game, one model (Gemini) was the most consistent winner, balancing aggression with caution. Others were too timid or too reckless.
• The Trust Game: Only a couple of models learned to build trust and cooperate effectively over time. Most either betrayed too easily or were too naive.

4. Why This Matters (The Takeaway)

Think of the old benchmarks as testing a calculator: "What is 2+2?"
The new benchmarks test a detective: "Here is a crime scene. The clues are hidden. Go find them, ask the right questions, and solve the crime before your time runs out."

The paper concludes that:

1. Current AI is still very "passive." It waits for information to be given to it.
2. Real intelligence requires active curiosity—the ability to realize "I don't know this, so I need to ask for it."
3. There is still a huge amount of room for improvement. Even the smartest AI models today struggle to be good detectives or strategic players.

In short: We stopped testing AI on how well it can memorize a textbook, and started testing it on how well it can navigate a maze while blindfolded, asking for directions only when it gets stuck. And guess what? They are still getting lost quite a bit.

Technical Summary

Here is a detailed technical summary of the paper "Interactive Benchmarks":

1. Problem Statement

The authors argue that current Large Language Model (LLM) evaluation paradigms are becoming unreliable due to three main factors:

• Saturation and Contamination: Static datasets (e.g., GSM8K, MMLU) are increasingly memorized by models, leading to inflated scores that do not reflect true reasoning capabilities.
• Subjectivity: Preference-based arenas (e.g., ChatBot Arena) rely on human judgment, which is subjective and unsuitable for measuring objective reasoning.
• Passivity: Existing benchmarks treat models as passive recipients of information. They fail to evaluate a core component of intelligence: the ability to actively decide what information to acquire, when to acquire it, and how to do so efficiently under constraints.

Real-world tasks rarely provide all necessary information upfront. An intelligent agent must recognize knowledge gaps, identify missing evidence, and take targeted actions to resolve uncertainty.

2. Methodology: Interactive Benchmarks Framework

The paper proposes Interactive Benchmarks, a unified evaluation paradigm where models act as agents in a sequential decision process under budget constraints. The framework is instantiated in two distinct regimes (illustrated in Figure 1 of the paper):

A. Interactive Proofs (Convergent Regime)

• Goal: Minimize uncertainty to converge on an objective truth (Logic/Math).
• Mechanism: The model (Prover) interacts with an omniscient but budget-limited Judge.
• Interaction: The model asks constrained questions (e.g., Yes/No) or submits intermediate claims. The Judge provides restricted feedback ({yes, no, both, irrelevant}).
• Constraint: A fixed interaction budget (e.g., 20 turns). The model must solve the problem within this budget.
• Metrics: Accuracy (solved within budget) and Interaction Efficiency (average turns to solve).
• Datasets:
  • Situation Puzzles: 46 logic puzzles requiring abductive reasoning to explain paradoxical narratives.
  • Math Problems: 52 challenging instances from the HLE dataset, requiring iterative verification of intermediate steps rather than one-shot generation.

B. Interactive Games (Divergent Regime)

• Goal: Maximize long-term expected utility against uncertain adversaries.
• Mechanism: The model interacts with an environment and other agents (no dedicated judge).
• Interaction: The model takes actions (bets, cooperate/defect) based on observations and history.
• Datasets:
  • Texas Hold'em Poker: A 6-agent No-Limit tournament testing strategic reasoning, risk management, and Theory of Mind under imperfect information.
  • Trust Game: An Iterated Prisoner's Dilemma with a random horizon (geometric distribution) to test adaptive cooperation and betrayal strategies.

3. Key Contributions

1. Unified Evaluation Paradigm: Formalizes interaction as a theoretical framework distinguishing between epistemic truth-seeking (Proofs) and strategic utility maximization (Games).
2. Active Information Acquisition: Shifts the evaluation focus from "answering correctly" to "how the model gathers information to answer correctly."
3. Robustness against Memorization: By requiring iterative interaction and constrained feedback, the benchmarks reduce the efficacy of shortcut solutions and data memorization.
4. Process Interpretability: Unlike static benchmarks, the interaction traces provide explicit evidence of the model's hypothesis testing, error correction, and strategic adaptation.

4. Experimental Results

The authors evaluated six frontier models (Grok-4.1, Gemini-3, GPT-5, Kimi, DeepSeek, Qwen3) across the four tasks.

Interactive Proofs (Logic & Math)

• Necessity of Interaction: In Situation Puzzles, all models achieved 0% accuracy without interaction, proving that these tasks cannot be solved via static reasoning alone.
• Performance:
  • Logic: Gemini-3-flash achieved the highest accuracy (30.4%), followed by GPT-5-mini (17.4%). Qwen3-max performed poorly (4.3%).
  • Math: Under a budget-matched comparison, Interactive evaluation significantly outperformed Pass@k (repeated sampling). Interactive accuracy was 20–50% higher than static Pass@k, suggesting static methods underestimate model capability when budget is fixed.
  • Efficiency: Grok-4.1 and GPT-5 showed the best balance of high accuracy and moderate turn counts (8–9 turns).

Interactive Games

• Poker: Gemini-3-flash dominated, achieving the highest average winnings ($31.8 \pm 42.4$) and the most stable performance across tables. GPT-5-mini was the most aggressive (highest VPIP) but had higher variance.
• Trust Game: Only Qwen3-max (1.867) and GPT-5-mini (1.836) outperformed standard heuristic baselines (Grim Trigger and Tit-for-Tat). Most other models failed to adapt effectively to the dynamic opponent strategies, indicating a significant gap in current models' ability to play dynamic games.

5. Significance and Conclusion

• Revealing Gaps: The benchmarks reveal that while models perform well on static tasks, there is substantial room for improvement in interactive scenarios. Models struggle with strategic adaptation in games and efficient information gathering in proofs.
• Future Directions: The authors suggest that future training methods should explicitly optimize for interactive performance (e.g., learning to ask the right questions or adapt strategies online) rather than just static prediction accuracy.
• Impact: Interactive Benchmarks provide a more faithful, robust, and generalizable measure of AI intelligence, moving beyond the saturation of static datasets toward evaluating true agentic capabilities.

Project Page: https://github.com/interactivebench/interactivebench