Recursive Joint Simulation in Games

Technical Summary: Recursive Joint Simulation in Games

Problem Statement
The paper addresses a fundamental difference between strategic interactions involving Artificial Intelligence (AI) agents and traditional human-human interactions: the potential for AI agents to be accurately simulated. Because an AI agent's source code may be known or its behavior predictable, an agent may face "self-locating uncertainty"—it cannot distinguish whether it is acting in the real world or within a simulation of that world. The authors investigate whether this uncertainty can be leveraged to achieve cooperative outcomes in strategic settings (such as the Prisoner's Dilemma) where standard game theory predicts mutual defection. Specifically, the paper explores a mechanism where agents engage in a "recursive joint simulation," observing nested simulations of their interaction before choosing an action, with the goal of determining if this setup alters the equilibrium outcomes.

Methodology
The authors formalize a setting called Recursive Joint Simulation (RJS). In this framework:

1. Joint Observation: Two or more agents jointly observe a simulation of the game they are about to play.
2. Recursion: The simulation itself contains a recursive structure. Before the simulated agents choose their actions, the device attempts to run a sub-simulation of the entire setup. This process repeats, creating a chain of nested simulations.
3. Termination Condition: To prevent infinite recursion, the simulation device has a fixed probability $p$ of continuing to a sub-simulation and a probability $1-p$ of terminating (refusing to simulate). If the device refuses, the agents in that specific instance must act immediately.
4. Payoff Structure: Crucially, payoffs are determined solely by the actions taken in the "top-level" (real) interaction. Simulated agents do not receive utility from simulated outcomes; they only care about the impact of their simulated actions on the real-world outcome.
5. Indistinguishability: The model assumes that simulated agents cannot distinguish their environment from reality. Consequently, an agent acting in a simulation reasons as if it might be in the real world, and vice versa.

The authors analyze this setting using extensive-form game representations and compare it to infinitely repeated games with exponential discounting. They also examine the "inside perspective," analyzing how an agent forms self-locating beliefs (regarding its depth in the simulation hierarchy) and whether these beliefs align with the probabilities found in repeated games.

Key Contributions and Results

1. Strategic Equivalence to Repeated Games (Theorem 1):
   The central technical result is that a Recursive Joint Simulation game $RJS(G, p)$ is strategically equivalent to an infinitely repeated version of the base game $G$ with a continuation probability $p$ (denoted $Rep_{T=?}(G, p)$ or $Rep_{\omega}(G, p)$).

   • The "deepest" simulation in the RJS chain corresponds to the first round of the repeated game.
   • The "top-level" (real) action corresponds to the final round of the repeated game.
   • Because the agents observe the outcomes of all sub-simulations before acting, the information structure mirrors that of a repeated game where players observe past rounds.

2. Folk Theorem for RJS (Corollary 4):
   Leveraging the equivalence to repeated games, the authors establish a Folk Theorem for recursive joint simulation. For any normal-form game $G$, any feasible and individually rational payoff vector can be sustained as a Nash equilibrium in the RJS setting, provided the simulation probability $p$ is sufficiently high. This implies that mutual cooperation in a Prisoner's Dilemma is an equilibrium outcome, achievable via strategies like "grim trigger" (cooperate unless a sub-simulation reveals a defection).

3. Robustness of Equivalence (The "Inside" Perspective):
   The paper demonstrates that the equivalence holds even when analyzing the game from the internal perspective of an agent (de se reasoning).

   • The authors address the issue of absentmindedness (facing the same information set multiple times in a single path), a common problem in self-locating belief scenarios like the Sleeping Beauty problem.
   • They show that in RJS, absentmindedness does not occur at actual decision points because agents know the depth of the simulation once they are asked to act.
   • Using "regular" methods of assigning self-locating beliefs, they prove that an agent's subjective probability of being at a specific depth in the simulation matches the probability distribution of rounds in a repeated game. Thus, the agent's internal reasoning leads to the same strategic conclusions as the external analysis.

4. Generalization:
   The authors note that the equivalence extends to settings with time-varying simulation probabilities (Appendix B), linking RJS with limited simulation budgets to finitely repeated games with uncertain lengths.

Significance and Claims
The paper claims that recursive joint simulation provides a rigorous mechanism for establishing cooperation among AI agents without relying on controversial decision-theoretic concepts like superrationality or direct access to source code for syntactic comparison (as in program equilibria).

• Cooperative Mechanism: The mechanism relies on the fact that an agent's action in a simulation can influence the opponent's action in the real world (because the opponent observes the simulation). If an agent defects in a simulation, the opponent (observing this) may defect in the real world. Therefore, agents are incentivized to cooperate in simulations to ensure cooperation in reality.
• Practicality vs. Theory: The authors are modest about the immediate applicability. They acknowledge that while the theoretical framework allows for a wide range of outcomes (including worse ones than the baseline), in practice, agents would likely only adopt RJS if it offers a Pareto improvement over the default Nash equilibrium of the base game.
• Limitations: The paper explicitly discusses two potential objections:
  1. Principle: Whether consciousness can be simulated. The authors argue this is less relevant for AI agents, which are instantiated on computers and can be simulated without loss of functional identity.
  2. Practice: Whether simulations can be made indistinguishable from reality in complex environments (e.g., interacting with real banking systems). The authors acknowledge this is a significant practical hurdle but argue it is solvable in constrained environments (e.g., trusted third-party implementations).

In conclusion, the paper establishes that the unique capabilities of AI agents to simulate one another fundamentally alter the game-theoretic landscape, transforming single-shot interactions into effectively repeated games where cooperation can be sustained as a rational equilibrium.