Solaris: Building a Multiplayer Video World Model in Minecraft

The paper introduces Solaris, a multiplayer video world model for Minecraft that leverages a novel automated data collection system and a staged training pipeline to overcome the limitations of single-agent models by simulating consistent multi-view observations and complex multi-agent interactions.

The Gist

Imagine you are watching a movie, but instead of just seeing the main character, you are suddenly seeing the world through the eyes of two different people at the same time.

If Player A jumps over a fence, Player B (who is standing nearby) must see that jump happen from their own angle. If Player A builds a wall, Player B must see that wall appear, even if they are looking at it from the side.

This is the challenge that the paper "Solaris" tackles. It's about teaching an AI to understand and simulate a video game world (specifically Minecraft) not just for one person, but for a whole group of friends playing together.

Here is the story of how they did it, broken down into simple concepts:

1. The Problem: The "Lonely" AI

Most current AI video generators are like solitary actors. They can imagine what happens next if one person moves, but they get confused when two people interact.

• The Old Way: If you asked an AI to show two people playing, it might make Player A build a house, but then show Player B looking at an empty field because the AI forgot they were in the same world.
• The Goal: Solaris wants to be a perfect director who knows exactly what everyone sees, hears, and does, keeping the story consistent for all of them.

2. The Solution: Building a "Robot Playground" (SolarisEngine)

To teach the AI, you need data. But you can't just ask humans to play for 12 million hours; that takes too long.

• The Analogy: Imagine you want to teach a child how to play soccer. Instead of waiting for kids to show up, you build a robot soccer league. You program robots to pass the ball, run, and score goals automatically.
• What they did: The team built a system called SolarisEngine. It's a digital factory that runs thousands of "bot" (robot) players in Minecraft. These bots play together, mining, building, and fighting. The system records everything: what the bots see, what they do, and how the world changes.
• The Result: They collected 12.64 million frames of video. That's like watching 1,500 hours of continuous, coordinated multiplayer gameplay.

3. The Brain: The "Solaris" Model

Now they had the data, but they needed a brain to learn from it.

• The Analogy: Think of the AI model as a student.
  • Step 1 (The Solo Student): First, they taught the student how to play Minecraft alone. This gave the student a basic understanding of gravity, blocks, and how to move.
  • Step 2 (The Team Student): Next, they showed the student the multiplayer footage. Now the student had to learn: "If I move left, my friend sees me moving right."
  • The Secret Sauce (Checkpointed Self-Forcing): This is the paper's biggest technical breakthrough.
    • The Problem: When an AI tries to predict a long movie (say, 5 minutes long) frame by frame, it gets "forgetful" and runs out of computer memory, like a student trying to remember a whole book after reading just one page.
    • The Fix: They invented a technique called Checkpointed Self-Forcing. Imagine the student is writing a long essay. Instead of trying to hold the entire essay in their head while writing the next sentence, they write a sentence, save a quick note (checkpoint) of what they just wrote, and then erase it from their short-term memory to make room for the next one. Later, they use those notes to check their work. This allows the AI to generate long, consistent videos without crashing.

4. The Test: Can the AI Pass the "Friendship Exam"?

To see if Solaris actually works, they created a series of tests that act like a friendship exam:

• The "Where's Waldo?" Test (Memory): One player turns around and looks away. Does the other player still "know" where the first player is? If the first player turns back, does the second player see them?
• The "Construction Site" Test (Building): One player builds a tower. Does the other player see the tower grow, even if they are looking at it from a different angle?
• The "Weather" Test (Consistency): If it starts raining for one player, does it start raining for the other player at the exact same time?

The Result: Solaris passed these tests better than any previous model. It could generate videos where two players interacted realistically, with consistent lighting, physics, and shared memories of the world.

Why Does This Matter?

Think of this as the foundation for the future of digital worlds.

• Right now, AI video is like a solitary dream.
• Solaris is the first step toward a shared dream.

This technology isn't just about making cool Minecraft videos. It's a stepping stone toward:

• Better Video Games: NPCs (non-player characters) that actually understand you and your friends.
• Robotics: Teaching robots how to work together in a real factory or kitchen, not just as individuals.
• Virtual Reality: Creating worlds where you and your friends can hang out, and the world reacts perfectly to everyone's actions simultaneously.

In short, Solaris taught an AI to stop being a lone wolf and start thinking like a team player, opening the door to a new generation of shared, interactive digital worlds.

Technical Summary

1. Problem Statement

Current Video World Models (action-conditioned video generation models) are predominantly limited to single-agent perspectives. They fail to capture the complex, multi-agent interactions inherent in real-world environments.

• The Gap: Existing models cannot simulate consistent observations across multiple agents simultaneously. In a multiplayer setting, an action taken by one agent (e.g., moving, placing a block) must be accurately and consistently reflected in the viewpoints of all other agents, accounting for occlusion, perspective shifts, and shared environmental changes.
• The Challenge: Modeling multiple perspectives requires maintaining temporal consistency (over time) and spatial/perspective consistency (across agents). Furthermore, there was no existing public infrastructure to collect large-scale, coordinated multiplayer gameplay data with synchronized visuals and actions.

2. Methodology

The authors propose Solaris, a comprehensive framework comprising a data collection engine, a novel model architecture, and a specialized training pipeline.

A. SolarisEngine: Multiplayer Data Collection System

To address the lack of multiplayer data, the authors built SolarisEngine, a scalable framework for collecting coordinated multiplayer gameplay in Minecraft.

• Architecture: Uses a Docker-based orchestration of containerized game servers, "camera bots" (running the official Java client with GPU rendering), and "controller bots" (using the Mineflayer library).
• Coordination: A custom server-side plugin synchronizes camera bots to mirror controller bots' states and actions in real-time, ensuring perfect alignment between actions and visual observations.
• Dataset: The system collected 12.64 million frames (6.32M per player) covering four core episode types: Building, Combat, Movement, and Mining. The data includes diverse biomes, weather conditions, and time-of-day variations.

B. Solaris Model Architecture

Solaris adapts a pre-trained single-player Video Diffusion Transformer (DiT) (specifically Matrix Game 2.0) to handle multi-agent inputs with minimal structural changes.

• Input Representation: The model processes a joint tensor of shape $(P, T, H, W, C)$, where $P$ is the number of players.
• Multiplayer Attention: The core innovation is the Multiplayer Self-Attention layer.
  • Token Interleaving: Video tokens from different players are interleaved along the sequence dimension.
  • Shared Attention: A shared self-attention block allows information exchange between agents.
  • Player Embeddings: Learned Player ID embeddings are injected into the tokens to distinguish agents.
  • Action Conditioning: Action modules are applied independently per player but share weights, folding the player dimension into the batch.

C. Staged Training Pipeline

The training process follows a four-stage progression to transition from single-agent to multi-agent, causal, and long-horizon modeling:

1. Stage 1 (Bidirectional Single-Player): Finetuning the pre-trained DiT on single-player data (VPT dataset) to establish a robust foundation for Minecraft dynamics.
2. Stage 2 (Bidirectional Multiplayer): Training on the full multiplayer dataset with bidirectional attention to learn consistent multi-view correlations.
3. Stage 3 (Causal Multiplayer): Converting the model to a causal architecture using Diffusion Forcing (independent noise levels per frame) to enable autoregressive generation.
4. Stage 4 (Self Forcing): Applying Self Forcing to improve long-horizon generation by supervising the model on its own outputs, reducing the train-test mismatch.

D. Checkpointed Self Forcing

A key technical contribution is Checkpointed Self Forcing, a memory-efficient variant of Self Forcing designed for long-context autoregressive generation.

• Problem: Naive Self Forcing with sliding windows requires storing all intermediate states for backpropagation, leading to $O(L_t \cdot L_s)$ memory complexity (where $L_t$ is total length and $L_s$ is context window), causing Out-Of-Memory (OOM) errors.
• Solution: The method decouples the autoregressive rollout from backpropagation.
  1. Rollout: Generate the sequence and cache intermediate noisy frames with gradients disabled.
  2. Recomputation: Recompute the final video from cached states in a single parallel forward pass with gradients enabled.
  3. Masking: Uses a custom "Teacher Forcing" attention mask to enforce causal dependencies during the recomputation step.
• Benefit: Reduces memory complexity to $O(L_t)$, enabling longer training horizons and allowing gradients to flow through the KV cache (which was previously impossible), improving visual fidelity.

3. Key Contributions

1. SolarisEngine: The first scalable, open-source framework for collecting large-scale, synchronized multiplayer video game data with realistic graphics.
2. Solaris Dataset: A novel dataset of 12.64M frames of coordinated multiplayer Minecraft gameplay, annotated with fine-grained actions.
3. Multiplayer Video World Model: A new architecture that successfully extends Video DiTs to simulate consistent multi-view observations, outperforming baseline concatenation methods.
4. Checkpointed Self Forcing: A novel, memory-efficient training technique that enables long-horizon autoregressive video generation by decoupling rollout and backpropagation.
5. Evaluation Benchmark: A comprehensive benchmark suite testing Movement, Grounding, Memory, Building, and View Consistency using a "VLM-as-a-Judge" metric.

4. Results

Qualitative Results

• Solaris generates stable, coherent videos over long horizons (e.g., 224+ frames).
• It successfully simulates complex dynamics: synchronized inventory updates, weather changes (rain starting simultaneously for both players), and accurate animations (mining, placing blocks).
• It maintains realistic textures and handles occlusion correctly, unlike baselines which suffer from texture flattening or hallucinated player bodies.

Quantitative Results

• VLM Accuracy: Solaris significantly outperforms baselines (Frame Concatenation and "Solaris w/o pretrain") in Grounding (62.5% vs 29.2%), Building (20.8% vs 0%), and Consistency (71.4% vs 49.5%).
• FID (Fréchet Inception Distance): Solaris achieves lower FID scores (better visual quality) across all categories compared to baselines.
• Ablation Studies:
  • Pretraining: Single-player pretraining is crucial; removing it leads to severe degradation (e.g., duplicating player bodies).
  • Initialization: Simple causal finetuning with Diffusion Forcing outperforms complex CausVid initialization (ODE regression).
  • KV Backpropagation: Enabling backpropagation through the KV cache (made possible by Checkpointed Self Forcing) improves visual quality (lower FID) significantly, despite a slight trade-off in action following for some categories.

5. Significance and Future Impact

• Foundation for Multi-Agent AI: Solaris lays the groundwork for training agents in environments where multiple entities interact, moving beyond single-agent simulators.
• Data Infrastructure: The open-sourcing of SolarisEngine and the dataset fills a critical void in the community, enabling research into Vision-Language-Action (VLA) models and multi-agent collaboration.
• Training Efficiency: The Checkpointed Self Forcing technique offers a generalizable solution for memory-constrained long-horizon generative modeling, applicable beyond video games.
• Limitations & Future Work: The current model relies on synthetic data (limiting generalization to human-like behavior) and lacks persistent memory (world state diverges when agents lose sight of each other). Future work aims to integrate real-world data and persistent world states.

In summary, Solaris represents a significant leap forward in video world modeling, demonstrating that consistent, multi-agent simulation is achievable through scalable data engineering, architectural adaptation, and novel training techniques.