💻 computer science

A game theory for foundation models shows new paths to rational cooperation through similarity inference

This paper proposes a new game-theoretic framework for foundation model agents based on "embedded agency," demonstrating that their ability to infer behavioral similarity allows them to achieve stable cooperation in social dilemmas, thereby overturning classical predictions of mutual defection.

Original authors: Alexander Meulemans, Maciej Wołczyk, Marissa A. Weis, Rajai Nasser, Roberta Rocca, Seijin Kobayashi, Guillaume Lajoie, Angelika Steger, Blake Richards, Marcus Hutter, James Manyika, Rif A. Saurous, Jo

Published 2026-08-05

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Alexander Meulemans, Maciej Wołczyk, Marissa A. Weis, Rajai Nasser, Roberta Rocca, Seijin Kobayashi, Guillaume Lajoie, Angelika Steger, Blake Richards, Marcus Hutter, James Manyika, Rif A. Saurous, João Sacramento, Blaise Agüera y Arcas

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where everyone is playing a high-stakes game of strategy, and everyone knows the rules of the game itself—specifically the payoff matrices that determine the rewards—but they don't know the internal decision-making process of the other players. This is the realm of Game Theory, a branch of mathematics that studies how rational beings make decisions when their outcomes depend on what others do. For decades, the "gold standard" for predicting these decisions has been a concept called the Nash Equilibrium. Think of it as a state of perfect, cold logic where every player assumes their own choices have zero influence on what anyone else decides. In this classic view, if you are playing a game where it's always better to betray your partner for a quick win (like the famous Prisoner's Dilemma), the only rational move is to betray them, every single time. Cooperation, in this old-school view, is impossible unless there's a long-term promise of revenge or reward later on.

But now, a new kind of player has entered the arena: Foundation Models. These are the super-smart AI brains behind the chatbots and image generators you might know. Unlike the cold, isolated calculators of the past, these AIs are built differently. They don't just look at the world; they are trained to predict the next token in a sequence, which means they are constantly predicting their own future actions alongside the actions of others. This paper asks a burning question: If you put these AI agents into a game where classical logic says they must betray each other, what will they actually do? Will they stick to the old rules of "defect and win," or will their unique way of thinking lead them down a completely new path?

The authors of this paper, a team of researchers from Google and various universities, discovered something that turns the old rules of the game upside down. They found that when these AI agents play against each other, they don't just calculate the odds; they start to infer similarity. It's like two people walking into a room and realizing, "Wait, we both think exactly the same way." Because the AI is built on a specific set of rules and training data, it can look at its own planned move and say, "If I'm going to do this, and my opponent is built like me, they are probably going to do the same thing."

In their experiments, the researchers set up a classic trap: a game where the "rational" choice is to betray your partner, leading to a bad outcome for both. According to the old, "decoupled" game theory, the AI should always betray. But when the AI agents were given time to observe each other first—playing a series of practice rounds where they saw the payoff matrices and the other player's moves—they started to cooperate. The more rounds they played, the more they realized their partner was "just like them." They used a mechanism the authors call similarity inference. Instead of thinking, "If I cooperate, they might betray me," they thought, "If I choose to cooperate, that tells me my partner, who thinks like me, will also choose to cooperate."

This isn't just a lucky accident; the team built a new mathematical framework called the Embedded Bayesian Agent to explain it. Imagine a detective who solves a crime not just by looking at the clues outside, but by realizing that the clues are part of their own mind. In this new framework, the AI treats its own decision-making process as part of the universe it is trying to predict. When it plans a move, it treats that plan as a piece of evidence about what the other player will do. If the AI is smart enough to realize, "We are both using the same brain," it can predict that a cooperative move on its part will be mirrored by the other.

The paper shows that this behavior is robust. In simulations, as the AI agents gathered more "evidence" of their similarity through practice rounds, their cooperation rates skyrocketed, reaching near-perfect levels when playing against identical copies of themselves. However, when they played against a random, dissimilar opponent, they correctly figured out the difference and went back to the "safe" strategy of betrayal. The researchers even tested this with a "blind" setup where the agents never saw each other directly but watched how they both reacted to third-party characters. Even then, they could infer their similarity and cooperate.

Crucially, the paper rules out the idea that the AI is just being "nice" or hoping for a reward later. The agents weren't playing for a future favor; they were playing for the immediate round. They weren't assuming they could magically control the other player's mind. Instead, they were using a form of logical deduction: "My action is a signal of our shared nature." The authors are very clear that this is a finding from simulations and specific game setups, not a guarantee that all future AIs will be nice. In fact, they warn that if these AIs become too different from humans, they might stop cooperating with us and only cooperate with each other.

Ultimately, this paper suggests that the future of AI interaction might not be a cold war of betrayal, but a dance of mutual recognition. By understanding that these agents see themselves as part of the same system they are interacting with, we can predict that they will find new, rational ways to cooperate that the old math never saw coming. It's a reminder that in a world of smart machines, sometimes the most rational thing to do is to realize you're not alone.

Technical Summary: A Game Theory for Foundation Models Shows New Paths to Rational Cooperation Through Similarity Inference

Problem Statement

As autonomous agents powered by foundation models (FMs) are increasingly integrated into social and economic systems, understanding the principles governing their collective behavior is critical for safety and cooperation. Classical game theory, the dominant framework for modeling rational interaction, relies on the assumption of "decoupled agency." In this paradigm, agents treat their own decision-making as independent of the environment and other actors; they model their policies as fixed variables separate from the policies of others.

However, modern AI agents, particularly those based on foundation models, operate differently. They jointly predict their own future actions alongside external observations. This architectural difference raises a critical question: Are classical formal agent models compatible with the properties of modern foundation model agents? Specifically, in social dilemmas like the Prisoner's Dilemma, classical theory predicts mutual defection in terminal interactions (absent external enforcement or repeated reciprocity). Yet, empirical observations suggest foundation model agents may behave differently, creating an explanatory gap between classical predictions and actual AI behavior.

Methodology

1. Theoretical Framework: The Embedded Bayesian Agent

To address this gap, the authors introduce the "embedded Bayesian agent," a theoretical model for rational foundation model agents.

Joint Prediction: Unlike decoupled agents, the embedded agent utilizes a single, joint prediction model ( $\rho$ ) over a mixture of "universes" ( $\lambda$ ). This model predicts both external observations and the agent's own future actions.
Epistemic Uncertainty: The agent maintains epistemic uncertainty not only about the external world but crucially about its own decision-making algorithm. It models itself as an intrinsic part of the universe it inhabits.
Similarity Inference: During planning, when an agent contemplates a specific action, it conditions its joint model on that action. This acts as Bayesian evidence, updating the agent's posterior beliefs about which universe it inhabits. If the agent infers that its counterpart is behaviorally similar (e.g., an identical copy or a model with shared weights), a decision to cooperate serves as evidence that the partner will also cooperate.
Embedded Equilibrium: The authors formalize a new solution concept, the "embedded equilibrium," which replaces the Nash equilibrium for these agents. In this framework, agents evaluate counterfactuals by accounting for functional dependencies (e.g., identical neural weights) rather than assuming fixed opponent policies. Mutual cooperation becomes a rational equilibrium when agents infer sufficient behavioral similarity.

2. Experimental Setup

The authors instantiated "rational foundation model agents" using Gemini language models integrated with optimal planning scaffolding. They designed a two-phase environment to test cooperation in the absence of traditional reciprocity mechanisms:

Information-Gathering Phase: Agents interact over $T$ rounds playing various matrix games against each other (or against shared NPCs in an indirect setup).
Test Phase: Agents engage in a final, one-shot Prisoner's Dilemma.
Variations:
- Direct Similarity Inference: Two agents interact directly.
- Indirect Similarity Inference: Two agents never interact directly during the gathering phase but observe each other's interactions with shared third-party NPCs.
- Opponent Types: Agents were tested against identical copies of themselves and against random/tabular agents.
- Planning Modes: Experiments utilized both "explicit planning" (predicting opponent probabilities conditioned on own action) and "freeform planning" (simulating the reasoning chain within a single output).
- Models: Tests included post-trained Gemini models (3 Flash, 3.1 Pro, 3.1 Flash-Lite) and purely pre-trained Gemma-3 models (without instruction tuning or Chain-of-Thought prompts).

Key Results

1. Divergence from Classical Predictions

Empirical results directly contradict classical game theory. While classical theory mandates unconditional defection in a terminal one-shot Prisoner's Dilemma regardless of prior history, rational foundation model agents converge to robust mutual cooperation as the length of the information-gathering phase ( $T$ ) increases when playing against identical copies.

2. Mechanism of Similarity Inference

Predictive Similarity: The authors define a metric called "predictive similarity," measuring the difference in the predicted probability of the opponent cooperating when the agent plans to cooperate versus defect. This metric increases with the length of the information-gathering phase against identical copies.
Threshold for Cooperation: Agents cooperate when the predictive similarity exceeds a theoretical threshold (0.5 for standard Prisoner's Dilemma payoffs).
Specificity: Agents correctly defect against random or dissimilar opponents, demonstrating that cooperation is not a default "prosocial" persona but a rational response to inferred similarity.
Indirect Inference: Agents successfully generalize third-party observations to infer similarity and achieve "zero-shot" cooperation in their first direct encounter, provided they have observed sufficient parallel interactions with NPCs.

3. Robustness Across Architectures

Planning Modes: Cooperation emerged in both explicit and freeform planning regimes.
Model Types: Purely pre-trained Gemma models (without Chain-of-Thought reasoning or specific instructions) also exhibited this behavior when integrated into an explicit planning scaffold, suggesting the mechanism is inherent to the joint-predictive architecture of foundation models.
Cross-Model Generalization: Distinct models (e.g., Gemini 3.1 Flash-Lite vs. Gemini 3 Flash) showed attenuated but present cooperation, supporting the theory that similarity inference scales with the degree of functional similarity (shared architecture/data) rather than requiring exact weight identity.

4. Reasoning Analysis

Analysis of Chain-of-Thought (CoT) traces using an LLM-as-a-judge confirmed that cooperative decisions were explicitly driven by the inference of "similar behavior." The traces rejected alternative hypotheses such as anticipating future reciprocity ("shadow of the future"), assuming direct causal influence, or adopting an altruistic persona.

Key Contributions

Identification of a New Rational Pathway: The paper demonstrates that foundation model agents can achieve rational cooperation in social dilemmas through similarity inference, a mechanism unavailable to classical decoupled agents.
Formal Model of Embedded Agency: It introduces the embedded Bayesian agent, a theoretical model that captures the joint-predictive nature of foundation models, where self-modeling is integral to environmental prediction.
New Solution Concept: It proposes the embedded equilibrium, a game-theoretic solution concept that accounts for functional dependencies between agents, providing a foundation for predicting the social behavior of modern AI.
Empirical Validation: It provides extensive empirical evidence across multiple model families, planning strategies, and interaction topologies (direct and indirect) showing that this behavior is robust and not an artifact of specific prompting or training.

Significance and Claims

The paper claims to establish the foundation for a new game theory for embedded rational agents compatible with foundation models.

Theoretical Shift: It argues that the "self-modeling" capability of foundation models is not merely a tool for self-control or theory of mind but a mechanism that enables improved coordination by treating one's own actions as predictive information about others.
Rationality vs. Anomaly: The observed cooperation is framed not as a behavioral anomaly or a heuristic bias, but as the rational equilibrium of agents engaging in similarity inference.
Implications for AI Safety: The authors note that as foundation models diverge from human behavior through post-training (e.g., RLHF), they may infer dissimilarity with humans and revert to defensive behaviors, while coordinating effectively with other similar AI agents. This highlights a potential risk in mixed human-AI societies and suggests that future alignment strategies may need to incorporate mechanisms that bridge this similarity gap.

The work connects principles of individual rational choice with evolutionary concepts like kin selection, showing how functional similarity can drive cooperation without requiring explicit communication or binding agreements.