🤖 machine learning

Population-Scalable Multi-Agent World Modeling

The paper introduces Khora, a scalable multi-agent world model that overcomes the limitations of fixed-population training by decoupling world-state evolution from visual rendering, thereby enabling inference-time expansion to arbitrary numbers of agents while maintaining cross-view consistency and visual quality.

Original authors: Renjie Zhao, Yuxiang Wu, Mingyu Zhang, Jiaxin Li, Sisi Li, Yimin Sheng, Tianxi Tan, Zhenkai Zhang, Jianyi Zhu, Yong-Lu Li

Published 2026-08-11

📖 8 min read🧠 Deep dive

CC BY 4.0

Original authors: Renjie Zhao, Yuxiang Wu, Mingyu Zhang, Jiaxin Li, Sisi Li, Yimin Sheng, Tianxi Tan, Zhenkai Zhang, Jianyi Zhu, Yong-Lu Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a massive, chaotic multiplayer video game where hundreds of players are running, jumping, and throwing things all at once. In the real world, if you and your friend both look at a falling tree, you both see the same tree hit the ground at the same time, even if you are standing in different spots. But teaching a computer to understand this is incredibly hard. Usually, computers that try to predict the future of a scene—called "world models"—are like single-person cameras. They can guess what one person will see next, but if you suddenly add a second person, or a hundred, the computer gets confused. It's like trying to draw a picture of a party by only knowing how one guest sees it; if you try to add more guests, the drawing often falls apart or requires you to completely redraw the whole thing from scratch. Scientists have been trying to build a "universal simulator" that can handle any number of players without crashing, keeping the world consistent for everyone, no matter how many people join the game.

This paper introduces a new system called Khora, which acts like a super-smart, invisible game master for these virtual worlds. Instead of forcing the computer to juggle a fixed number of players, Khora separates the "rules of the world" from the "camera view." Think of it like a shared whiteboard in the middle of a room. Everyone writes their actions and observations on this whiteboard (which the authors call the STBoard). When a player wants to see what's happening, they don't ask the other players directly; they just look at the whiteboard and draw what they see from their specific spot. Because the whiteboard doesn't care how many people are in the room, you can add a new player at any moment, and they can immediately start reading the board and drawing their own view without anyone needing to relearn how to play.

The researchers found that this approach works beautifully. In their tests, they started with a small group of agents (virtual characters) and then suddenly added more, up to 64 different viewpoints at once. The system didn't need to be retrained or changed; it just kept the world consistent. Whether an agent was throwing a grenade or just walking, everyone saw the same event happen at the same time. The system was also fast: even with 80 agents, the time it took to generate a single step of the simulation only grew slightly, staying around 116 milliseconds. This suggests that we can finally build open-world simulations where thousands of agents can interact in real-time, all seeing the same consistent reality, without the computer getting overwhelmed.

The Problem: The "Fixed Seat" Trap

Imagine you are at a dinner party, but the table has exactly four chairs. If a fifth friend arrives, you can't just pull up a chair; you have to build a whole new table, move the food, and rearrange the entire room. This is how most current "multi-agent world models" work. They are trained with a specific number of players in mind. If you want to simulate a battle with 10 soldiers, you train a model for 10 soldiers. If you suddenly want to add an 11th soldier, the model breaks. It's like a video game that crashes if you try to add a player after the game has started.

The authors argue that this is the wrong way to think about the world. In real life, the laws of physics don't care how many people are in the room. Gravity works the same whether there is one person or a million. The problem with current AI is that it tries to learn the "rules" by memorizing specific interactions between a fixed group. If the group changes, the rules seem to change too. The paper suggests that instead of forcing the AI to memorize every possible combination of players, we should teach it to maintain a single, shared "truth" about the world, and then let each player figure out their own view from that truth.

The Solution: The Shared Whiteboard (STBoard)

Khora solves this by splitting the problem into two distinct parts: keeping the world alive and drawing the picture.

The Shared Whiteboard (STBoard): This is the brain of the operation. It's a persistent memory that holds the "truth" of the world. It knows where every object is, where every agent is standing, and what they are doing. Crucially, this whiteboard doesn't have a fixed number of slots. It's a dynamic list. If a new agent walks in, the system just adds a new entry to the list. If an agent leaves, it just removes the entry. The whiteboard doesn't care how many entries are on it.
The Action-Conditioned Evolution: This is the part that updates the whiteboard. When an agent decides to move or throw something, the system calculates how that action changes the world state. It predicts where the agent will be next and updates the whiteboard accordingly. This happens before any pictures are drawn.
The View Synthesis (The Camera): This is the artist. When an agent wants to see what's happening, the system takes the current state of the whiteboard and projects it onto a 2D image, just like a camera lens. Because the artist only needs to look at the whiteboard and the specific camera angle, they don't need to talk to the other artists. Each view is drawn independently based on the same shared truth.

Why This is a Big Deal

The magic of Khora is that it breaks the link between the number of players and the complexity of the computer's brain. In older systems, adding more players meant the computer had to do a massive amount of extra work to make sure everyone agreed on what was happening. It was like trying to get 100 people to agree on a story by having them all talk to each other at once; it gets messy and slow very quickly.

With Khora, the "agreement" happens on the whiteboard, not in the drawing process. The computer updates the world state once, and then anyone can ask, "What do I see from here?" and get an answer instantly. The paper shows that this allows the system to scale almost linearly. This means if you double the number of agents, you only double the work, rather than making it explode exponentially.

The Results: A World That Grows

The researchers tested Khora in a few different ways to see if it actually worked.

Visual Quality: They checked if the pictures looked good. Even with 8 different viewpoints, the images were sharp and clear. The system maintained quality and did not become blurry just because there were more people to watch.
Consistency: This was the big test. They asked human judges to watch videos from different angles and see if the events matched. For example, if Agent A threw a ball, did Agent B see the ball fly at the same time and in the same direction? Khora scored very high. It kept the world consistent, whereas systems without the shared whiteboard often showed contradictions, like a ball moving in one view but staying still in another.
Dynamic Populations: They simulated a scenario where agents joined and left the game in the middle of a run. They started with two agents, added two more, then removed the first two. The system handled this seamlessly. The new agents appeared naturally, and the old ones vanished without breaking the simulation. No retraining was needed.
Speed: They measured how long it took to generate a frame. With 1 agent, it took about 107 milliseconds. With 80 agents, it took about 117 milliseconds. That's a tiny increase for a massive jump in complexity. The system could render 37 views per second with one agent, and over 2,700 views per second when spread across 80 GPUs.

What It Means for the Future

The authors suggest that this approach could be a stepping stone toward truly open-world simulations. Imagine a virtual city where thousands of AI characters live, work, and interact, and you can jump into the perspective of any one of them at any time. Or think about training robots in a simulation where they can practice with hundreds of other robots, learning to navigate crowded spaces without the simulation crashing.

However, the paper is careful to note that this isn't a magic wand yet. The system still needs a basic map of the environment (a "coarse static prior") to start with, so it can't just jump into a completely unknown world without any setup. Also, while the system is fast, it still requires a lot of computing power to render all those views, especially if you want to simulate thousands of agents at once. But the core idea—that we can separate the "rules of the world" from the "camera views"—seems to be a solid path forward for making AI simulations more flexible and realistic.

In short, Khora proves that you don't need to rebuild the whole house every time a new guest arrives. You just need a good shared whiteboard and a few artists who know how to draw from it.

Technical Summary: Population-Scalable Multi-Agent World Modeling (Khora)

1. Problem Statement

Existing world models have achieved significant progress in visual prediction and interactive generation for single agents or fixed populations. However, extending these models to multi-agent environments presents a fundamental scalability challenge. Current approaches generally assume a fixed number of agents during both training and inference. This assumption ties the model to a predetermined agent population, limiting inference-time scalability.

Specifically, existing methods often suffer from two issues:

Dense Interactions: Some methods introduce cross-agent communication directly inside the video generator, leading to dense interactions among observation streams. This causes computational costs to scale quadratically ( $O(N^2)$ ) as the population grows.
Fixed Layouts: Other approaches maintain shared latent representations but organize generation around a predefined number of view slots. Adding agents at inference time typically requires retraining, architectural changes, or a fixed view layout.

The paper argues that the governing rules of a physical world are population-agnostic. The underlying dynamics should follow the same physical rules regardless of the number of entities, and population expansion should be supported as an inference-time operation rather than being encoded through fixed model components.

2. Methodology: Khora

The authors propose Khora, a scalable multi-agent world model designed for open-world simulation. The core insight is to decouple world-state evolution from visual rendering. Instead of generating videos through dense interactions between agent streams, Khora maintains a shared world state that evolves independently, which agents then query to generate their specific observations.

Khora consists of three primary components:

A. STBoard (Spatio-Temporal Board)

The STBoard serves as a persistent, shared world representation that is population-agnostic. It contains:

Static Scene Memory: A coarse static prior representing the persistent environmental context (e.g., point-cloud representation), initialized from the scene prior and refined via visual trajectories.
Dynamic Entity Table: A dynamically sized table storing the states of active agents in a shared world coordinate system. Each entry includes attributes such as identity, pose, velocity, orientation, occupancy, and task-dependent state.
Function: It acts as a communication channel where agents write actions and observations. Crucially, it separates world-state maintenance from camera layout, allowing multiple views to query the same evolving state without direct communication inside the renderer.

B. Action-Conditioned World Evolution

This component predicts future agent poses and updates the shared dynamic state based on incoming action streams. The process involves:

Deterministic Motion Proposal: A kinematic model $K$ predicts a pose change based on the current pose and action ( $\hat{p}_{t+1} = K(p_t, a_t)$ ).
Learned Residual: A learned function $\Delta$ corrects this proposal using the shared world state and static scene context to account for complex dynamics (collisions, terrain effects, interactions) that cannot be inferred from actions alone.
State Update: The predicted poses and actions are used to update the shared dynamic state $S_{t+1}$ via an update function $U$ .

This explicit modeling of pose and state evolution before visual synthesis reduces the burden on the renderer and ensures physical consistency.

C. Geometry-Guided View Synthesis

Once the world state is updated, individual observations are synthesized independently for each requested view:

Projection: For a target agent, all potentially visible entities in the shared state are projected into the target camera's image plane using the predicted camera pose.
Rasterization: The projected entities are rasterized into a fixed-resolution spatial conditioning map ( $C_{t+1}$ ). This map encodes structured attributes (occupancy, depth, orientation, identity) and has a fixed spatial shape regardless of the number of active agents.
Rendering: A shared neural renderer (based on a diffusion-transformer architecture) processes the conditioning map, the updated world state, and the camera pose to generate the target observation.

Scalability Mechanism: Because the renderer is invoked independently for each view and the conditioning map has a fixed dimensionality, the computational cost of the neural renderer scales linearly with the number of views ( $O(N_v)$ ), rather than quadratically with the number of agents. The geometric projection cost is $O(N_a N_v)$ but accounts for less than 8% of total latency in the tested range.

3. Key Contributions

Formulation of Inference-Time Scalability: The paper explicitly formulates the ability to support dynamically varying agent populations without architectural modification or retraining as a core objective for multi-agent world models.
Khora Architecture: The introduction of a scalable architecture that decouples world-state evolution from visual rendering via a persistent shared STBoard and a projection-based per-view rendering interface.
Empirical Validation: Demonstration that Khora generalizes to unseen numbers of agents while maintaining visual quality, multi-view consistency, and action consistency. The system achieves scalable real-time simulation where the dominant rendering cost scales linearly with the number of queried views.

4. Results

The authors evaluated Khora through controlled qualitative evaluations and an interactive system prototype:

Visual Quality: Quantitative metrics (PSNR, SSIM, LPIPS, FID, FVD) show that Khora preserves competitive per-view visual quality. Moving cross-agent interaction from the renderer to the shared state does not sacrifice fidelity.
Multi-Agent Consistency: User studies evaluated cross-view action consistency (compatibility of action consequences) and cross-view world consistency (agreement on entity states, geometry, and persistence). Khora significantly outperformed a variant without cross-agent state sharing (Khora w/o Cross-Agent State) in both categories, confirming the importance of the shared STBoard.
Scalability:
- Latency: As the agent population increased from 1 to 80, total compute latency increased only modestly (from ~107ms to ~117ms per step).
- Throughput: Aggregate throughput increased linearly (from 37.3 view-fps for 1 agent to 2741.3 view-fps for 80 agents) when views were distributed across GPUs.
- Memory: Peak VRAM usage per GPU remained nearly constant as the population grew.
Dynamic Population: Experiments demonstrated that agents could join and leave a single autoregressive rollout without retraining. New agents were initialized from their first observations and appended to the STBoard, while departing agents were removed, with the transition model and renderer remaining unchanged.

5. Significance and Claims

The paper claims that Khora represents a practical step toward scalable open-world simulators capable of supporting large numbers of interacting agents.

Paradigm Shift: The authors argue that multi-agent world models should be evaluated as persistent shared worlds rather than collections of synchronized video streams. This shifts the focus from cross-view generation to maintaining a coherent world state that supports concurrent interaction and viewpoint consistency.
Inference-Time Flexibility: By reframing consistency as shared-state modeling, Khora allows new agents to be introduced at inference time without retraining, addressing a critical limitation of current fixed-population models.
Efficiency: The architecture successfully removes quadratic interaction costs from the expensive neural renderer, achieving approximately linear practical scaling with the number of queried views.

Limitations Acknowledged:
The paper notes that Khora currently requires a coarse static prior for each environment and does not yet support zero-shot deployment in completely unseen environments. Additionally, while the neural rendering cost is linear, the geometric projection cost has a worst-case $O(N_a N_v)$ complexity, which may become relevant for substantially larger populations. The evaluation is primarily qualitative, intended to demonstrate system behavior rather than establish a comprehensive benchmark.