💻 computer science

minWM: A Full-Stack Open-Source Framework for Real-Time Interactive Video World Models

The paper introduces minWM, a full-stack open-source framework that transforms existing bidirectional video foundation models into real-time, camera-controllable, few-step autoregressive world models through a comprehensive pipeline of fine-tuning, causal forcing training, and distillation.

Original authors: Min Zhao, Hongzhou Zhu, Bokai Yan, Zihan Zhou, Yimin Chen, Wenqiang Sun, Kaiwen Zheng, Guande He, Xiao Yang, Chongxuan Li, Fan Bao, Jun Zhu

Published 2026-05-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Min Zhao, Hongzhou Zhu, Bokai Yan, Zihan Zhou, Yimin Chen, Wenqiang Sun, Kaiwen Zheng, Guande He, Xiao Yang, Chongxuan Li, Fan Bao, Jun Zhu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant, high-end movie director (a Video Foundation Model) who can create stunning, high-quality films based on a script. However, this director works very slowly: they plan the entire movie scene-by-scene before showing you a single frame. If you want to change the camera angle halfway through, they have to stop, re-plan the whole movie, and start over. This is great for making a polished film, but terrible for a video game or a live chat where you need instant reactions.

minWM is a new "training camp" and "toolkit" designed to turn this slow, perfectionist director into a real-time interactive director who can react to your commands instantly, without losing their artistic quality.

Here is how minWM works, broken down into simple steps:

1. The Problem: The "Slow Director"

Current video AI models are like the slow director. They are amazing at making beautiful videos, but they are bidirectional. This means they look at the beginning, middle, and end of a video all at once to ensure everything fits perfectly.

The Issue: If you want to move the camera or change the scene in real-time, this model can't do it quickly. It takes too long to "think" about the whole video before showing you the first frame.

2. The Solution: The minWM "Training Camp"

minWM is a full-stack framework (a complete set of tools) that takes an existing slow director and trains them to become a fast, interactive performer. It does this in two main phases:

Phase 1: Learning to Move the Camera (The "Camera Control" Lesson)

First, the framework teaches the slow director how to follow specific camera instructions.

The Analogy: Imagine teaching the director to hold a camera on a gimbal. Instead of just guessing where the camera should be, they learn to follow a precise path you draw for them.
How: The team uses a special technique called PRoPE. Think of this as giving the director a pair of "smart glasses" that understand exactly where the camera is in 3D space. They practice on videos where the camera movement is perfectly known (like a 3D animation), so the director learns the exact relationship between "move left" and "the view shifts left."

Phase 2: Speeding Up the Process (The "Distillation" Lesson)

Now that the director can follow camera moves, they are still too slow. They still plan the whole movie before showing a frame. minWM uses a technique called Causal Forcing to speed them up.

The Analogy: Imagine a student (the new fast model) sitting next to a master teacher (the slow, high-quality model).
1. Teacher Forcing: The student learns to write the story one sentence at a time (frame by frame) instead of planning the whole book at once. This makes them faster.
2. The "Cheat Sheet" (Distillation): To make the student even faster, they are trained to skip steps. Instead of taking 50 small steps to draw a picture, they learn to do it in just 4 big, confident strokes.
3. The Safety Net (Asymmetric DMD): There's a risk that by speeding up, the student might start drawing messy pictures. To fix this, the student occasionally checks their work against the original master teacher. If the student's fast drawing looks a bit off, the teacher gently corrects them, ensuring the final result is still high-quality.

3. The Results: From "Cinema" to "Video Game"

The paper shows that this process works incredibly well:

Speed: The new models are 200+ times faster at showing the first frame than the original slow models.
- Before: You wait over 10 seconds to see the first frame.
- After: You see the first frame in about 1 second.
Control: The fast models can still follow your camera commands perfectly. You can tell the video to "pan left" or "zoom in," and it happens instantly.
Flexibility: The team tested this on two different types of "directors" (the Wan2.1 and HY1.5 models) and it worked for both, proving the method is a universal toolkit, not just a one-time fix.

4. Important Lessons Learned (The "Ablation Studies")

The paper also shares some practical tips they discovered while building this, which are like "rules of thumb" for anyone trying to do this:

Good Data is Key: You can't just teach the director with blurry, guessed camera movements. You need "ground truth" data—videos where the camera path is mathematically perfect (like 3D reconstructions). If you use messy data, the director gets confused.
Practice Makes Perfect: The director needs to train for a while (around 8,000 steps) before they truly understand how to move the camera. If you stop too early, they won't listen to your commands.
Don't Skimp on the Class Size: You need a decent number of examples (a "batch size") for the director to learn. If the class is too small (fewer than 4 examples), they fail to learn the camera moves.

Summary

minWM is an open-source recipe that takes a high-quality, slow video AI and transforms it into a real-time, interactive video engine. It teaches the AI to follow camera commands and then speeds it up so it can generate video as you watch, making it possible to build interactive video worlds (like video games or live simulations) that were previously impossible with standard video AI.

Technical Summary: minWM

Problem Statement
Recent diffusion-based foundation models have achieved significant success in high-quality text-to-video (T2V) and text-and-image-to-video (TI2V) generation. However, transforming these offline, bidirectional generators into real-time interactive video world models remains a substantial challenge. An effective interactive world model must support causal rollout, respond to user actions (such as camera trajectories), and generate future frames with sufficiently low latency. While recent works have explored autoregressive (AR) diffusion distillation to bridge this gap, these techniques are currently scattered across separate pipelines. Building a complete interactive system still requires substantial, disjointed effort in data construction, controllable fine-tuning, AR training, few-step distillation, and inference alignment. There is a lack of a unified, reproducible, and extensible framework to manage this full pipeline.

Methodology
The authors present minWM, a full-stack open-source framework designed to convert existing bidirectional T2V/TI2V foundation models into camera-controllable, few-step autoregressive world models. The pipeline is modular and consists of two primary phases:

Phase 1: Camera-Controllable Training for Bidirectional Diffusion Models
- Objective: Equip a multi-step bidirectional diffusion backbone with the ability to follow prescribed camera trajectories while preserving original visual quality.
- Technique: The framework adopts PRoPE (Projective Rotation and Position Embedding) as the injection method. Given camera parameters (intrinsic matrix $K_i$ and extrinsic transformation $T^{cw}_i$ ), PRoPE constructs a lifted projective matrix for each frame. This transformation is injected into the self-attention mechanism in a GTA (Generalized Token Attention) form, allowing the model to condition on camera trajectories by encoding relative camera intrinsics and poses without disrupting the original self-attention generative structure.
Phase 2: AR Diffusion Distillation for Real-Time Interactive Models
- Objective: Transform the camera-controllable multi-step bidirectional model into a few-step autoregressive generator suitable for low-latency inference.
- Pipeline: The framework implements the Causal Forcing or Causal Forcing++ pipeline, comprising three stages:
  - Stage 1 (AR Diffusion Training): The bidirectional model is fine-tuned into an AR diffusion model via teacher forcing. This involves concatenating clean video with noisy counterparts and training under a causal attention mask. This grants autoregressive capability but retains multi-step latency and exposure bias issues.
  - Stage 2 (Initialization): The model is distilled into a few-step generator. The framework offers two options:
    - Causal ODE: Regressing from noisy intermediate frames to clean frames using PF-ODE trajectories generated by the AR teacher.
    - Causal CD (Consistency Distillation): A theoretically equivalent alternative that eliminates the need for offline ODE data generation, reducing storage and time overhead.
  - Stage 3 (Asymmetric DMD): To overcome the quality limitations inherited from the AR teacher, an asymmetric Distillation with a Diffusion Model (DMD) stage is applied. The few-step AR student model self-rollouts to generate full sequences, which are then optimized against a frozen, high-quality bidirectional teacher model (the "real" score) using standard DMD gradients.
- Camera Integration: Throughout all distillation stages, camera conditions are consistently fed into the AR teacher, the student model, and the score estimators ( $s_{real}$ and $s_{fake}$ ), ensuring the final few-step model remains camera-controllable.

Key Contributions

minWM Framework: The release of a fully open-source, end-to-end pipeline for building real-time interactive video world models. It covers the entire workflow from camera-conditioned data construction and controllable fine-tuning to Causal Forcing distillation and low-latency inference.
Architecture Generalization: Demonstration that the framework is architecture-agnostic. The authors instantiated minWM on two distinct open backbones:
- Wan2.1-T2V-1.3B: Utilizing cross-attention-based condition injection.
- HY1.5-TI2V-8B: Utilizing an MMDiT-style architecture.
Adaptability: The framework supports adapting existing world models (e.g., HY-WorldPlay) to new data distributions, training recipes, and latency targets.
Practical Ablations: The paper provides actionable guidance on critical training factors, including the necessity of ground-truth camera trajectories (vs. perception-estimated poses), the minimum training steps required for controllability, and minimal batch-size requirements.

Results

Latency Reduction: The few-step AR models achieved dramatic reductions in first-frame latency compared to multi-step bidirectional baselines.
- HY1.5: Reduced latency from 771.04s to 3.446s (a 223.75× speedup).
- Wan2.1: Reduced latency from 269.055s to 1.137s (a 236.64× speedup).
- This low latency allows for real-time interaction where users can view frames while generation is ongoing.
Controllability Preservation: Visual results confirm that the distilled few-step AR models retain the ability to generate videos under different camera actions, validating the effectiveness of the distillation algorithm in preserving controllability.
Ablation Findings:
- Data Quality: Training on datasets with perception-estimated camera poses (SpatialVid) failed to yield reliable controllability. Successful training required ground-truth trajectories obtained via 3D reconstruction/re-rendering (DL3DV) or generation by WorldPlay.
- Training Steps: Camera controllability emerged progressively; models were largely uncontrollable at 1–2k steps, began responding at ~5k steps, and achieved strong controllability at 8k steps.
- Batch Size: A batch size smaller than 4 often resulted in failure to learn controllability. A batch size of 16 was found to be necessary for stable, high-controllability training.

Significance
The paper positions minWM not as a single trained checkpoint, but as a reproducible and extensible recipe for the community. It addresses the fragmentation in current research by providing a unified pipeline that converts powerful offline foundation models into real-time interactive agents. By releasing runnable scripts, checkpoints, and documentation, minWM aims to lower the barrier for researchers to build, adapt, and extend real-time interactive video world models, particularly for applications requiring low-latency, camera-controllable generation. The authors explicitly note that while they currently focus on camera control, the framework is designed to be extended to other control conditions (e.g., pose) and additional models in the future.