Improvisational Games as a Benchmark for Social Intelligence of AI Agents: The Case of Connections

1. Problem Statement

The paper addresses a critical gap in current Large Language Model (LLM) benchmarks: the lack of evaluation for social intelligence and Theory of Mind (ToM) capabilities. While LLMs excel at knowledge retrieval, summarization, and deductive reasoning, they often fail to demonstrate the ability to:

• Gauge the cognitive states and knowledge bases of other agents.
• Adapt communication strategies based on the perceived abilities of teammates versus opponents.
• Navigate the delicate balance between providing clues that are clear enough for teammates to understand but obscure enough to prevent an opponent (the "Setter") from guessing the answer.

The authors argue that standard benchmarks do not capture the nuance of collaborative reasoning in constrained, multi-agent environments where success depends on modeling the mental states of others.

2. Methodology

The Game: "Connections"

The authors formalize a word-play game called Connections as a testbed.

• Roles: One player is the Setter (who has a secret target word), and $n$ players are Guessers.
• Mechanics:
  1. The Setter reveals the first letter of the target word.
  2. Guessers propose clues (phrases) that relate to a word starting with the revealed prefix.
  3. Connection: If a Guesser proposes a clue, and another Guesser (simultaneously) guesses the correct word associated with that clue without the Setter guessing it first, a "connection" is made, and the next letter is revealed.
  4. Blocking: If the Setter guesses the word based on the clue, the clue is "blocked," and no letter is revealed.
  5. Win Condition: Guessers win if they reveal the full word within a set number of clues; otherwise, the Setter wins.

Formalization

The authors model the game mathematically using semantic embeddings:

• Semantic Space: Each player $j$ has a unique working vocabulary $U_j$ and a semantic embedding function $\Phi_j$ that maps words/clues to vectors.
• Clue Generation: A clue $p$ is successful if the target word $w$ is among the top-$k$ closest words to $p$ in the Guessers' semantic space, but not in the Setter's space (or at least not the top guess).
• Optimization: The authors derive a probability function for clue generation. To maximize success, a Guesser must find a clue where the probability of a teammate guessing correctly ($p$) is high, but the probability of the Setter guessing correctly ($1-p$) is also high (i.e., the Setter fails).
  • The optimal probability $p^*$ for a clue to be guessed by a teammate but not the Setter is derived as $p^* = 1 - (1/n)^{1/(n-1)}$.
• Social Learning Model: The paper proposes a gradient-descent-like mechanism where agents update their "perceived discourse vectors" ($d_{i \leftarrow j}$) based on game outcomes. If a player successfully guesses a clue, the vector is adjusted to reflect that the agent shares that semantic knowledge.

Experimental Setup

• Agents: Three agents powered by GPT-4o.
• Roles: Player 0 (Setter), Players 1 & 2 (Guessers).
• Prompts: Agents were prompted with specific roles. In some experiments, "in-context learning" was used to prime agents with specific backgrounds (e.g., profession, culture) to simulate diverse semantic networks.

3. Key Contributions

1. Formalization of a Social Benchmark: The paper introduces "Connections" as a rigorous, formalizable benchmark for measuring AI social intelligence, moving beyond static QA tasks to dynamic, multi-turn collaborative reasoning.
2. Mathematical Framework for Clue Optimization: It provides a probabilistic model for generating clues that balance the trade-off between teammate comprehension and opponent blocking, formalizing the "Goldilocks" zone of clue difficulty.
3. Mechanism for Social Awareness: The authors propose a method for agents to learn the "discourse vectors" of other players over time, effectively simulating the human ability to learn who knows what (e.g., "Player A is good at medical terms, Player B is good at TV shows").
4. Empirical Analysis of LLM Limitations: The study highlights that while LLMs can play the game, they struggle with adaptive social awareness when not explicitly primed. Without specific prompts about other agents' backgrounds, LLMs tend to rely on a uniform semantic network, leading to frequent "blocking" by the Setter.

4. Results

• Performance Metrics: The authors ran 100+ iterations across various target words (e.g., kaleidoscope, xenophobia, labyrinthine).
  • Success Rate: Success varied significantly by word. Words with smaller prefix spaces (e.g., starting with 'X') required fewer iterations to solve than those with large spaces (e.g., 'C').
  • Blocking: In standard setups (without specific background priming), the Setter blocked a high percentage of clues. This suggests that when all agents share the same underlying model, their semantic networks are too similar, making it difficult for Guessers to find "blind spots" in the Setter's knowledge.
• Behavioral Observations:
  • Human-like vs. Non-Human: Agents displayed human-like behavior in recognizing that 'X' words are easier to guess. However, they exhibited non-human behavior by making "bizarrely uncommon" guesses before settling on obvious ones, indicating a lack of strategic intuition.
  • Impact of Priming: When agents were explicitly primed with different backgrounds (e.g., distinct professions), they showed improved ability to utilize "concerted vocabulary" to bypass the Setter. However, without explicit prompting, they failed to infer these differences through gameplay alone.

5. Significance and Future Work

• Beyond Deductive Reasoning: The paper argues that true AI intelligence requires the ability to model the cognitive states of others. "Connections" forces agents to move beyond simple retrieval to strategic communication and collaborative adaptation.
• Implications for AI Safety and Alignment: Understanding how AI agents gauge and adapt to human (or other AI) knowledge bases is crucial for developing agents that can collaborate effectively in real-world scenarios (e.g., healthcare, education, negotiation).
• Future Directions:
  • Dynamic Word Selection: The current game uses a fixed word. Future work aims to allow the Setter to change the target word mid-game (as long as it fits revealed letters), requiring even more complex improvisation.
  • Robust Social Learning: Developing agents that can autonomously detect and model the knowledge gaps of others without explicit prompt engineering, moving toward true Theory of Mind.

In conclusion, the paper establishes "Connections" as a vital new benchmark for evaluating the social intelligence of AI, demonstrating that while current LLMs possess the raw knowledge to play, they lack the nuanced, adaptive social reasoning required to consistently outmaneuver opponents and collaborate effectively without explicit guidance.