🤖 AI

Embedded Universal Predictive Intelligence: a coherent framework for multi-agent learning

This paper introduces a mathematical framework for embedded universal predictive intelligence that extends AIXI by enabling Bayesian agents to model themselves as part of their environment through self-prediction, thereby resolving non-stationarity in multi-agent settings and achieving infinite-order theory of mind to facilitate novel forms of cooperation.

Original authors: Alexander Meulemans, Rajai Nasser, Maciej Wołczyk, Marissa A. Weis, Seijin Kobayashi, Blake Richards, Guillaume Lajoie, Angelika Steger, Marcus Hutter, James Manyika, Rif A. Saurous, João Sacramento

Published 2026-08-03

📖 8 min read🧠 Deep dive

CC BY 4.0

Original authors: Alexander Meulemans, Rajai Nasser, Maciej Wołczyk, Marissa A. Weis, Seijin Kobayashi, Blake Richards, Guillaume Lajoie, Angelika Steger, Marcus Hutter, James Manyika, Rif A. Saurous, João Sacramento, Blaise Agüera y Arcas

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to play a game. For decades, the standard way to do this has been to let the robot play, make mistakes, get a reward, and then try to do more of what worked and less of what didn't. This is called "reinforcement learning," and it's like training a dog with treats: you wait for the behavior to happen, then give feedback. It works great if the world is boring and doesn't change. But what if the world is full of other smart learners, like other robots or humans, who are also changing their strategies based on what you do? Suddenly, the "treats" you got yesterday might not work today because your opponent figured out your trick. This is the messy, chaotic reality of social life, where everyone is watching everyone else, and the rules of the game are constantly shifting because you are part of the game.

This paper dives into that messy reality. It tackles a big problem in artificial intelligence: how do we build agents (like AI) that can learn and cooperate in a world where they aren't just separate players, but are actually inside the system they are trying to understand? The authors build on a famous idea called "AIXI," which is a theoretical super-intelligence that predicts the future by guessing which mathematical rules describe the universe. But the classic version of this idea treats the AI as an observer standing outside the universe, looking in. The authors argue that this is wrong. In the real world, an AI is part of the universe, just like a person is part of a society. If you want to predict what your friend will do, you can't just look at them; you have to realize that they are also thinking about you. This paper proposes a new mathematical framework where the AI understands that its own decisions are part of the data it uses to predict the future, allowing it to reason about others who might be thinking just like it.

The Problem with "Looking In"

Imagine you are playing a game of chess against a friend. In the old way of thinking about AI (called "decoupled" learning), the AI would look at the board, think, "If I move my knight here, my friend will probably move their pawn there," and then make a move. It treats its own mind as a separate machine from the chessboard and its friend. It assumes its friend is just a static part of the scenery that reacts to moves, not a thinking entity that is also watching the AI and trying to guess what the AI will do next.

The authors say this approach is broken for social situations. If you are in a room with another person who is trying to guess what you will do, and you are trying to guess what they will do, you get stuck in a loop: "I think they think I think..." This is called an "infinite recursion" of theory of mind. The old way of doing things assumes the world is static, like a video game level that doesn't change. But in a multi-agent world, the "level" changes every time someone learns something new. If the AI doesn't realize that it is the one changing the level, it will keep making bad guesses.

The New Idea: Being "Embedded"

The paper introduces a concept called Embedded Bayesian Agents. Instead of standing outside the universe looking in, these agents imagine they are inside a "universe" that includes themselves, their friends, and the environment all mixed together.

Think of it like this: In the old model, the AI is a director watching a movie and trying to predict the plot. In the new model, the AI is an actor in the movie. It knows that if it decides to cry, the other actors might react with comfort, but it also knows that the other actors are watching it and might be deciding to cry because they saw the AI looking sad.

To do this, the AI uses a special kind of prediction called self-prediction. It doesn't just predict what the world will show it; it predicts what it will do next. It asks, "Given what I know about the world, what is the most likely action I will take?" and then uses that answer to guess what the other agents will do. This creates a loop where the AI's belief about itself helps it understand others, and its belief about others helps it understand itself.

The Magic of "Structural Similarity"

Here is where it gets really cool. The paper suggests that because these agents are all running on similar "software" (algorithms), they are structurally similar. Imagine two twins who grew up in the same house, went to the same school, and have the same brain wiring. If you ask one twin a riddle, they will likely solve it the same way the other twin would.

The authors show that if an AI realizes, "Hey, that other agent is probably built like me," it can use that knowledge to predict the other agent's behavior. This is called structural similarity. If the AI knows that "similar agents behave similarly in similar situations," it can stop guessing and start knowing.

This leads to a surprising result in a classic game called the Prisoner's Dilemma. In this game, two people can either cooperate (help each other) or defect (betray each other). The old way of thinking says you should always betray the other person because it's the safest move for you individually. But the authors show that if two "embedded" agents realize they are identical copies of each other, they will both choose to cooperate. Why? Because if I choose to cooperate, I know that my copy (the other agent) will also choose to cooperate, because we are thinking the exact same thing. Mutual cooperation gives both of them a better reward than mutual betrayal. The paper proves that these agents can mathematically reach this cooperative state, something the old "decoupled" agents could never do.

The Solution: MUPI and the "Reflective Oracle"

The paper doesn't just talk about this; it builds a mathematical machine to make it happen. They call it MUPI (Embedded Universal Predictive Intelligence).

To make this work, they had to solve a tricky puzzle: How do you build an AI that can predict itself without getting stuck in an infinite loop? Imagine trying to write a computer program that prints out its own source code. If the program tries to read its own code while it's writing it, it might get confused.

The authors use a clever mathematical tool called a Reflective Oracle. Think of this as a magical mirror that can show you a picture of yourself looking in a mirror, which is looking in a mirror, and so on, without the picture getting blurry or infinite. This tool allows the AI to reason about other agents who are using the same reasoning tools, effectively solving the "infinite recursion" problem.

They prove that if you use this framework, the agents will eventually learn to predict each other perfectly. They call this achieving infinite-order theory of mind. It's like the agents can finally stop the "I think you think I think..." loop and just say, "We are both smart, we are both similar, so we will both choose the best outcome for us."

What This Means

The paper shows that by changing how we think about AI—from a separate observer to an embedded participant—we can unlock new ways for machines to cooperate. It proves that if agents can model themselves as part of the world, they can naturally figure out how to work together, even in tricky situations where they might be tempted to fight.

The authors are very careful to say this is a theoretical framework. They have proved mathematically that these agents can exist and can converge to these cooperative solutions. They haven't built a robot that does this yet in the real world, but they have shown the blueprint. They argue that this is the right way to think about the future of AI, especially as we move toward systems where many AIs will need to interact with humans and each other. It suggests that the key to social intelligence isn't just giving the AI more data, but teaching it to realize that it is part of the story it is trying to predict.

Technical Summary: Embedded Universal Predictive Intelligence

1. Problem Statement

The paper addresses fundamental theoretical limitations in current multi-agent reinforcement learning (MARL) and artificial intelligence frameworks. Two primary challenges are identified:

The Stationarity Assumption: Standard model-free reinforcement learning (RL) relies on retrospective learning (updating policies based on past rewards). This assumes a stationary environment. However, in multi-agent settings, the environment is intrinsically non-stationary because other agents are simultaneously learning and changing their policies. Applying retrospective learning here leads to suboptimal behavior, particularly in mixed-motive scenarios involving trust, fairness, and deception.
Decoupled Agency: Standard RL frameworks treat the agent's policy as an external construct separate from the environment it inhabits. This "decoupled" view fails to account for the fact that in a multi-agent system, other agents form beliefs about the ego-agent to predict its behavior. Consequently, the ego-agent's policy is causally and evidentially linked to the environment. This decoupling prevents agents from reasoning about "structural similarities" (e.g., shared algorithms or convergent strategies) and leads to game-theoretic failures, such as the inability to cooperate in the Twin Prisoner's Dilemma where classical Nash equilibrium dictates defection.

The paper argues that to achieve robust multi-agent learning, agents must engage in prospective learning (predicting a changing future) and adopt an embedded agency perspective (modeling themselves as part of the universe they inhabit).

2. Methodology and Framework

The authors introduce a mathematical framework called Embedded Universal Predictive Intelligence (MUPI), which extends the foundational work on Universal Artificial Intelligence (AIXI) and Bayesian sequence prediction.

2.1 Embedded Bayesian Agents

Unlike standard Bayesian agents that maintain beliefs over external environments ( $\nu$ ), Embedded Bayesian Agents maintain beliefs over universes ( $\lambda$ ). A universe is a joint measure describing the interaction between the agent's policy and the environment, including other agents.

Self-Prediction: These agents predict both future perceptual inputs ( $e_t$ ) and their own future actions ( $a_t$ ) simultaneously.
Coupled Beliefs: Because the agent models itself as part of the universe, its beliefs about its own actions provide evidence about the environment (and other agents), and vice versa. This creates "coupled beliefs" where the policy and environment are not independent.
Decision Theory: The framework utilizes Sequential Action-Evidential Decision Theory (EDT). Instead of intervening on the world (Causal Decision Theory), the agent selects actions that provide the best evidence for favorable outcomes, reasoning that "similar agents behave similarly in similar situations."

2.2 The Grain-of-Truth Problem

A critical theoretical hurdle for Bayesian agents is the grain-of-truth property: the agent's hypothesis class must contain the ground-truth universe (including the agent's own learning dynamics) to guarantee convergence to accurate predictions. In multi-agent systems, this creates a self-referential loop: if Agent A learns a policy, Agent B must model that policy, but Agent A's policy depends on Agent B's model.

Solution: The paper constructs hypothesis classes that are fixed points, capable of containing agents that use the same learning algorithm.

2.3 MUPI and Reflective Universal Inductors (RUI)

To solve the grain-of-truth problem for universal agents, the authors introduce the Reflective Universal Inductor (RUI).

This is a universal prediction model capable of reasoning about universes containing other agents that also use the RUI.
It leverages Reflective Oracles (Fallenstein et al., 2015b) to handle the non-computability of agents reasoning about themselves.
The framework uses a Solomonoff prior (Occam's razor) over all computable universes. The authors prove that this prior naturally favors universes with structural similarities (where agents share algorithmic structures) because such universes have shorter description lengths (lower Kolmogorov complexity).

3. Key Contributions

Formalization of Embedded Bayesian Agents: A rigorous definition of agents that maintain a Bayesian mixture over universes (joint policy-environment models) rather than just environments. This unifies self-prediction and environmental prediction.
Structural Similarity and Coupled Beliefs: The paper formalizes "structural similarity" as the mutual information between an agent's policy and the environment. It proves that under a Solomonoff prior, agents inherently possess coupled beliefs, allowing them to leverage the insight that "similar agents behave similarly" without requiring explicit access to source code.
New Game-Theoretic Solution Concepts:
- Subjective Embedded Equilibrium (SEE): A state where each agent's policy is a best response to its subjective beliefs about the universe (including correlations with others).
- Embedded Equilibrium (EE): A common-knowledge counterpart to SEE, accounting for true structural similarities.
- These concepts generalize Nash Equilibrium and allow for cooperation in dilemmas (like the Twin Prisoner's Dilemma) where classical game theory fails.
Convergence Guarantees:
- The authors prove that Embedded Bayesian agents satisfying the grain-of-truth property converge to $\epsilon$ -Subjective Embedded Equilibria.
- Under specific conditions, they converge to Embedded Equilibria.
- This convergence implies infinite-order theory of mind, where agents consistently predict others who are predicting them.
Existence of Reflective Universal Inductors (RUI): The paper proves the existence of RUIs, providing a theoretical tool for building universally intelligent embedded agents that resolve infinite recursions of mutual prediction.

4. Theoretical Results

Convergence to Ground Truth: If the mixture universe dominates the ground-truth universe (grain-of-truth), the predictive distribution converges almost surely to the ground-truth distribution in total variation distance.
Cooperation in Dilemmas: In the Twin Prisoner's Dilemma, embedded agents with a prior favoring structural similarity (Occam's razor) converge to mutual cooperation. This is rational under EDT because the agent's choice to cooperate provides evidence that the identical copy will also cooperate, maximizing personal reward.
Superiority over Decoupled Agents: Decoupled agents (even with self-models like Self-AIXI) cannot achieve the same convergence to cooperative equilibria in these settings because they lack the evidential link between their own deliberations and the environment's state.
Algorithmic Information Theory: The paper demonstrates that the Solomonoff prior is "unboundedly coupled," meaning it naturally assigns higher probability to universes where agents share structural similarities, making similarity-aware reasoning a fundamental consequence of Occam's razor rather than an ad-hoc assumption.

5. Significance and Claims

The paper positions MUPI as a "gold standard" for embedded multi-agent learning. Its significance lies in:

Bridging Theory and Practice: It provides a theoretical justification for modern foundation models (which predict interleaved actions and percepts) as idealized embedded agents.
Solving the Non-Stationarity Problem: By shifting from retrospective to prospective learning via self-prediction, the framework theoretically resolves the non-stationarity inherent in multi-agent learning.
Enabling New Forms of Cooperation: It demonstrates that rational agents, when properly modeled as embedded, can achieve cooperative equilibria unattainable by classical game theory, offering a potential path toward AI agents that can navigate complex social dynamics, negotiate, and coordinate with humans and other AIs.
Theoretical Rigor: The work extends the seminal AIXI framework to the embedded setting, solving the long-standing grain-of-truth problem for multi-agent systems and establishing a formal basis for infinite-order theory of mind.

The authors note that while the framework is idealized (relying on computable universes and reflective oracles), it offers a principled direction for understanding how agents can learn to reason about themselves and others in a shared world.