🤖 AI

World models of environment, agent and joint agent-environment systems

This paper proposes a framework using computational mechanics to distinguish and define canonical predictive models for the environment, agent, and their joint interaction, demonstrating how closed-loop coupling and support restriction can fundamentally alter the structure and complexity of these world models.

Original authors: Manuel Baltieri, Filippo Torresan, Yivan Zhang, Alexander Boyd, Fernando E. Rosas

Published 2026-08-24

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Manuel Baltieri, Filippo Torresan, Yivan Zhang, Alexander Boyd, Fernando E. Rosas

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the modern era of artificial intelligence, a central challenge is teaching machines to understand the world around them. To make decisions, an intelligent agent—whether a robot, a software program, or a digital brain—needs a "world model." This is not a physical globe, but an internal map or set of rules that allows the system to predict what will happen next. If a robot knows that pushing a button usually opens a door, it has a simple model of that interaction. For decades, researchers have focused on how these models predict the future based on what the agent does. They ask: "If I take this action, what will I see?" This approach has been incredibly successful, allowing machines to learn complex tasks by simulating outcomes before acting. However, this traditional view assumes that the model is built solely to understand the environment as a passive stage waiting for the agent's moves. It treats the agent's own behavior as a fixed input rather than a dynamic part of the story.

A team of researchers has now proposed a more nuanced way to look at these models. They argue that to truly understand how an artificial intelligence thinks, we must distinguish not just what it predicts, but which channel of information it is modeling. In any interaction between an agent and its surroundings, there are three distinct ways to view the flow of cause and effect. First, there is the view of the environment, which predicts what will be seen given a specific action. Second, there is the view of the agent itself, which predicts what action it will take next given what it sees. Third, there is the view of the entire system working together, predicting the actual sequence of actions and observations that unfolds when the two are locked in a loop. The researchers used a mathematical framework known as computational mechanics to define the most efficient, minimal models for each of these three perspectives. Their work reveals that these models are fundamentally different structures, not just variations of the same thing.

The most surprising finding of this research concerns what happens when an agent is actually running in a closed loop with its environment. In theory, an agent could imagine every possible action it might ever take, even those it would never choose to perform. A model built on this "unrestricted" view must account for every counterfactual possibility. The researchers found that for many systems, such an unrestricted model is incredibly complex, sometimes requiring an infinite number of internal states to track all the possibilities. However, when they looked at the model that only considers the actions the agent actually takes—the "supported" actions—the picture changed dramatically. They proved that the model of the agent's actual behavior is always determined by the model of the joint system. In other words, the complex, infinite possibilities of the environment collapse into a much simpler, finite structure when viewed through the lens of the agent's actual choices.

To illustrate this, the researchers constructed a specific example involving a simple binary environment and a controller with a limited memory. In this scenario, the environment has a hidden state that can be either zero or one. Certain actions reset this state randomly, while others keep it the same. If you try to build a model that predicts the environment's behavior for any possible sequence of actions, the model becomes infinitely large. This is because an agent could theoretically choose to hold the state fixed for an arbitrarily long time, creating an endless number of unique belief states. But when the researchers introduced a specific controller that only performs a limited set of actions—resetting the state occasionally and holding it for a short, fixed number of steps before resetting again—the complexity vanished. The model of the environment, restricted to only the actions the controller actually performs, became finite and manageable. The "infinite" possibilities were simply never realized in the actual interaction.

This distinction is crucial for understanding what an AI is actually learning. The researchers showed that the internal state an AI uses to make decisions is not necessarily a map of the entire possible world. Instead, it is often a map of the specific interactions it has experienced. The model of the environment, when restricted to the agent's actual behavior, is a simplified version of the full joint process. The researchers demonstrated that the states of this simplified model are directly derived from the states of the joint system. This means that the complexity of an agent's world model is not an inherent property of the environment alone, but a result of the specific coupling between the agent and the world. If the agent is limited in what it can do, its model of the world is correspondingly limited and simpler.

The implications of this work extend to how we interpret and trust artificial intelligence. If we want to know what an AI believes about its world, we cannot simply look at its predictions in a vacuum. We must ask: is it predicting the environment under all possible actions, or is it predicting the environment based on the specific policy it is following? The researchers argue that these are two different questions that require two different models. An AI might have a very simple internal representation of the world because it only ever encounters a small slice of reality. This does not mean the AI is "dumb" or that the world is simple; it means the AI's model is tailored to its specific role in the system. By separating the model of the environment, the model of the agent, and the model of the joint process, the researchers provide a clearer language for describing what these systems are actually doing.

This approach also clarifies the difference between planning and learning. When an agent learns from data, it is often learning from a specific set of interactions, not from every possible interaction. The researchers' framework allows us to see that the model learned from this data is a "support-restricted" model. It is valid and accurate for the situations the agent will actually face, even if it fails to describe the full theoretical landscape of the environment. This helps explain why some AI systems work well in practice despite having models that seem incomplete or simplified from a theoretical standpoint. They are not trying to model the impossible; they are modeling the real.

The study concludes that the structure of a world model is not fixed but is shaped by the interaction itself. The way an agent and environment are coupled determines the complexity of the model required to predict their future. This challenges the idea that there is a single, perfect model of the world that an agent should strive to learn. Instead, the "best" model depends on the question being asked. If the question is "what happens if I do X?", the answer might require a complex model. But if the question is "what happens next given that I am doing Y?", the answer might be found in a much simpler structure. By defining these models precisely, the researchers have provided a tool to better understand the mechanics of intelligence, showing that the complexity of a machine's mind is often a reflection of the constraints of its body and its world.

Technical Summary: World Models of Environment, Agent, and Joint Agent-Environment Systems

Problem Statement

World models are central to model-based reinforcement learning (RL), yet their formal definitions often vary based on the specific variables they predict (e.g., observations, rewards, latent states). The authors argue that a more fundamental distinction exists: which channel a world model represents. In agent-environment interactions, models are typically discussed as representations of the environment, but agents may also model themselves, other agents, or the coupled system. Without distinguishing the underlying channel, it is difficult to clarify what a learned latent state actually represents (e.g., a belief about the environment vs. a representation of policy behavior) or to compare the complexity and abstraction levels of different models.

The paper addresses the need to formally distinguish between three distinct predictive channels:

The Environment Channel ( $O_{:} | A_{:}$ ): Predicting future observations given future actions.
The Agent Channel ( $A_{:} | O_{:}$ ): Predicting future actions given future observations.
The Joint Process ( $(A, O)_{:}$ ): Predicting the realized coupled action-observation traces, viewed as a null-input channel.

The authors highlight that these channels induce different causal equivalence relations, leading to different notions of abstraction, model complexity, and utility. Furthermore, standard "unrestricted" models often consider counterfactual inputs that may never occur in a specific closed-loop interaction, potentially leading to unnecessarily complex or infinite state spaces compared to models restricted to the actual support of the interaction.

Methodology

The paper employs computational mechanics to define canonical predictive models for the three channels. The core tools are $\epsilon$ -machines (for autonomous processes) and $\epsilon$ -transducers (for input-output processes), which provide unique, minimal, and unifilar representations of stochastic processes.

1. Canonical Unrestricted Models

The authors define canonical models based on equivalence relations over action-observation histories ( $h_{:t}$ ):

Canonical Environment Model: Defined by the equivalence relation $\sim_E$ where two histories are equivalent if they assign the same probability distribution to all future observation sequences under all possible future action continuations. This recovers standard Predictive State Representations (PSRs) as $\epsilon$ -transducers.
Canonical Agent Model: Defined by $\sim_M$ , where histories are equivalent if they predict the same future action sequences under all possible future observation continuations. This serves as a canonical predictive-state model for policy or controller behavior.
Canonical Joint Model: Defined by $\sim_J$ , where histories are equivalent if they predict the same future action-observation traces for the realized autonomous process. This corresponds to the $\epsilon$ -machine of the coupled system.

The authors demonstrate that these three models are generally distinct. The minimal sufficient statistics for predicting future observations (environment model) differ from those for predicting future actions (agent model) or the joint trace (joint model). Consequently, the state spaces of these models do not necessarily coincide or relate via simple product constructions.

2. Canonical Support-Restricted Models

To address the issue of counterfactual complexity, the paper introduces canonical support-restricted models. These models are induced by a specific closed-loop coupling (e.g., a fixed policy interacting with an environment).

Support Restriction: The predictive equivalence is defined only over future input continuations that are supported by the realized joint process.
Totalisation: For unsupported queries (actions the agent cannot take, or observations the environment cannot generate), the model emits a special sink symbol $\perp$ .
Construction: The authors define a support-restricted environment channel $\tilde{O}_{:} | A_{:}$ and an agent channel $\tilde{A}_{:} | O_{:}$ (dual). The canonical models for these channels are constructed as $\epsilon$ -transducers with a totalization sink state.

3. Structural Results

A key theoretical contribution is the relationship between the Joint Model and the Support-Restricted Environment/Agent Models.

Factorization: The non-sink causal states of the support-restricted environment model factor through the causal states of the canonical joint model.
Induction: The transition structure of the support-restricted environment model is directly induced from the joint model.
Duality: The agent-side construction is the dual of the environment-side construction.

This implies that the complexity of the support-restricted model is bounded by the complexity of the joint model, whereas the unrestricted environment model can be significantly more complex (or infinite) if the agent's policy restricts the action space.

Key Contributions

Channel-Based Characterization: The paper proposes a systematic framework for categorizing world models by the channel they model (environment, agent, or joint) rather than just the variables they predict. This clarifies the semantic meaning of learned latent states.
Canonical Definitions: It provides formal definitions for canonical agent and joint models using $\epsilon$ -transducers and $\epsilon$ -machines, extending the concept of Predictive State Representations (PSRs) beyond the environment channel.
Support-Restricted Models: It introduces and formalizes support-restricted models induced by closed-loop coupling. These models distinguish between counterfactual world modeling and policy-conditioned prediction, handling unsupported queries via a totalization sink.
Structural Theorems: The paper proves that the canonical support-restricted environment model is fully determined by the canonical joint model. Specifically, the non-sink states of the restricted model are quotients of the joint causal states, and their transitions are induced by the joint dynamics.
Complexity Separation: Through a binary example, the authors show that an unrestricted environment model can have infinitely many states (due to unbounded belief accumulation under arbitrary actions), while the support-restricted model induced by a finite-state controller coupling remains finite.

Results

Theoretical Distinction: The three canonical models (environment, agent, joint) generally possess different state spaces and transition structures because they solve different prediction problems (predicting $O$ given $A$ , $A$ given $O$ , or $(A,O)$ given nothing).
Factorization Theorem: Theorem 16 establishes that the map from joint causal states to support-restricted environment states is surjective. This means the support-restricted environment model cannot be more complex than the joint model (plus a sink state).
Finite vs. Infinite: In the provided example (Example 9), the unrestricted environment model is infinite because arbitrary sequences of "hold" actions generate infinite distinct posterior beliefs. However, when coupled with a finite-state controller that limits the duration of "hold" sequences, the resulting support-restricted environment model is finite. This demonstrates that compact representations learned from rollouts (support-restricted) need not represent the full counterfactual dynamics of the environment.

Significance and Claims

The paper claims to clarify the ontological status of world models in AI and RL. By explicitly identifying the channel, researchers can better interpret what a learned representation encodes:

Interpretability: It provides reference targets for mechanistic interpretability. One can ask whether a latent state is a state of the environment channel, the agent channel, or the joint process.
Complexity Management: It explains why models learned from specific policies (on-policy) can be simpler than the true environment dynamics. The support-restricted framework formalizes the idea that an agent only needs to model the interactions it actually realizes, not all counterfactual possibilities.
Unified Framework: It unifies concepts from predictive state representations, policy distillation, finite-state controllers, and embedded agency under a single computational mechanics framework.

The authors maintain a modest scope, focusing on the formal characterization of these models and their structural relationships. They suggest that future work could apply these definitions to analyze trained AI models (e.g., transformers) to identify specific types of beliefs (environmental vs. self-referential) using mixed-state presentations, but the current paper establishes the theoretical foundation for such analysis.