🤖 machine learning

MotionCraft: Latent World Modeling with Sparse Attention for Visual Upscaling

MotionCraft is a controllable video super-resolution framework that leverages motion-aware latent state prediction and adaptive sparse attention to achieve high-fidelity, temporally consistent reconstructions while balancing local detail, long-range dependencies, and computational efficiency.

Original authors: Rong Fu, Chunlei Meng, Yangchen Zeng, Xiaowen Ma, Yongtai Liu, Wangyu Wu, Shuo Yin, Zijian Zhang, Sicheng Li, Yingrui Ji, Chenhao Wang, Simon Fong

Published 2026-08-11

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Rong Fu, Chunlei Meng, Yangchen Zeng, Xiaowen Ma, Yongtai Liu, Wangyu Wu, Shuo Yin, Zijian Zhang, Sicheng Li, Yingrui Ji, Chenhao Wang, Simon Fong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a blurry, shaky video on your phone. Maybe it's a shaky recording of a concert or a grainy old home movie. You wish you could zoom in and see the singer's face clearly or make the shaky camera movement smooth. This is the problem of "video super-resolution." It's like trying to guess what a high-definition photo looks like when you only have a tiny, fuzzy sketch. For years, computers have tried to solve this by looking at how pixels move from one frame to the next. Some methods are like careful painters who only look at the pixels right next to each other, which is great for small details but gets confused when things move fast. Others are like detectives who look at the whole picture to find connections, but they get so overwhelmed by the amount of data that they slow down to a crawl. The big challenge has always been finding a way to be both fast and smart enough to handle big, fast movements without the video looking like a glitchy mess.

Enter MotionCraft, a new approach that tries to solve this puzzle by treating the video not just as a stack of pictures, but as a "world" that is moving and changing. Instead of just guessing what the next frame looks like, MotionCraft tries to predict the "state" of the world, much like how a video game engine predicts where a character will be a split second later based on their current speed and direction. The researchers found that by combining a smart way of tracking motion (even when the motion is messy or hidden) with a "sparse attention" system—which is like a spotlight that only shines on the most important parts of the video rather than the whole stage—they could create high-quality, smooth videos much faster than before. They also built a special "control knob" that lets users decide if they want the video to look super sharp (even if it's a bit jittery) or super smooth (even if it loses a tiny bit of detail).

The Problem: The "Blurry vs. Fast" Dilemma

Think of trying to fix a blurry video like trying to restore a torn map. If the map is just a little crumpled, you can smooth it out easily. But if the map is being thrown around in a storm (fast motion) or if parts of it are covered in mud (occlusions), it gets really hard to know where the roads go.

Previous methods had a tough time with this. Some methods, called convolutional techniques, are like a person who only looks at the immediate neighborhood. They are great at keeping the edges of buildings sharp, but if a car zooms past, they get confused and the car looks like it's melting. Other methods, based on Transformers, are like a person who looks at the entire map at once. They can see how the car moves across the whole screen, but they get so tired from looking at every single detail that they take forever to finish the job. Then there are generative methods that try to "dream up" the missing details. They can make the video look incredibly realistic, but sometimes they "hallucinate" things that weren't there, making the video flicker or jump between frames.

The Solution: A "World Model" with a Spotlight

The authors of this paper, MotionCraft, decided to build a system that acts like a predictive world model. Instead of just looking at the pixels, the system tries to understand the "latent state" of the video. Think of this as the video's "internal GPS." It doesn't just look at the picture; it asks, "Where is this object going to be in the next second?"

Here is how they made it work:

The "Trust Me" Sensor (Reliability-Guided Fusion):
One of the biggest headaches in video restoration is that the tools used to track motion (like optical flow) often lie. If a car drives behind a tree, the tracker might get confused and say the car is moving sideways. MotionCraft solves this by having a "Trust Me" sensor. It checks the motion data from two sources: an external tracker and an internal guess based on the image itself. If the external tracker looks shaky (like near a tree or in a blurry area), the system automatically trusts its own internal guess more. It's like having a GPS that knows when the satellite signal is weak and switches to your map instinct.
The Spotlight (Adaptive Sparse Attention):
To avoid getting overwhelmed by data, MotionCraft uses Adaptive Sparse Attention. Imagine you are in a crowded room and you need to find your friend. A "full attention" system would look at every single person in the room, which takes forever. MotionCraft uses a spotlight. It keeps looking at the people right next to you (local details) but also quickly scans the room to find the one or two people far away who are actually your friend (long-range connections). It ignores everyone else. This makes the system incredibly fast without losing the ability to see the big picture.
The Control Knob (Controllability Interface):
Usually, you have to choose between a video that looks super sharp but jittery, or one that is smooth but blurry. MotionCraft introduces a Controllability Interface. This is like a slider on a music app. You can slide it one way to prioritize fidelity (keeping every tiny detail sharp) or the other way to prioritize smoothness (making the motion look fluid). The system adjusts the "latent state" in real-time to give you exactly the balance you want.

What They Found

The researchers tested MotionCraft on many different types of videos, from synthetic computer-generated clips to real-world footage from movies and shaky phone recordings.

It's Faster and Sharper: In their tests, MotionCraft was able to process video at 18.63 frames per second (FPS) on a standard graphics card, which is faster than many other top methods. At the same time, it achieved a PSNR of 27.05 dB on the REDS dataset, a score that indicates a very high level of detail recovery compared to other methods which scored around 24–25 dB.
It Handles Motion Better: When they tested videos with fast movements or complex scenes, MotionCraft showed a Temporal Consistency score of 0.96 (on a scale where 1 is perfect), beating the next best method which scored 0.90. This means the video didn't flicker or jump as much.
It's Robust to Bad Data: Even when they swapped out the motion tracker for a different, less accurate one, the system's performance only dropped by a tiny amount (about 0.14 dB), proving that its "Trust Me" sensor works well.
The Trade-off is Predictable: When they turned up the "smoothness" control knob, the video got smoother, and the sharpness (PSNR) went down in a very predictable, gentle way. This means users can choose their preference without the video suddenly breaking.

Why It Matters

MotionCraft suggests that we don't have to choose between speed, quality, and control anymore. By treating video restoration as a problem of predicting a "world state" and using a smart spotlight to focus only on what matters, the authors have created a system that is efficient enough for real-time streaming (like watching a movie on your phone) but powerful enough to restore old, damaged films.

The paper doesn't claim this is the final answer to every video problem, but it shows that combining motion reliability checks, sparse attention, and user control creates a much more practical and powerful tool than what we had before. It's a step toward making high-quality video restoration something that can happen instantly, right on your device, without needing a supercomputer.

Technical Summary: MotionCraft

Problem Statement

Video Super-Resolution (VSR) aims to reconstruct high-fidelity, high-resolution video sequences from low-resolution inputs. While critical for applications ranging from mobile photography to archival restoration, existing methods face a fundamental trade-off among local-detail fidelity, long-range spatio-temporal modeling, perceptual realism, and computational efficiency.

Convolutional approaches preserve local structures but suffer from misalignment errors when motion is large or degradations are complex.
Transformer-based methods capture long-range dependencies but incur prohibitive computational costs for high resolutions or long sequences due to quadratic attention complexity.
Generative (latent/diffusion) methods synthesize rich textures but often struggle with temporal coherence and require specialized conditioning to prevent frame-to-frame discontinuities.

Furthermore, current systems often rely on external motion cues (e.g., optical flow) that fail in occluded or textureless regions, leading to temporal inconsistencies. There is a lack of frameworks that simultaneously offer robust motion handling, efficient long-range modeling, and user-controllable trade-offs between smoothness and fidelity in a streaming context.

Methodology

MotionCraft formulates video restoration as motion-aware latent state prediction inspired by world models. It operates within a compact latent space and is designed for strictly causal, streaming inference. The framework consists of four core components:

1. Causal Motion Encoding and Reliability-Guided Fusion

Instead of relying solely on external motion estimates, MotionCraft fuses two motion streams:

External Motion: Aggregated from external cues (e.g., optical flow) over the causal history.
Internal Motion: Extracted directly from the observed image history via a reduced-resolution encoder.
Reliability Gating: A photometric inconsistency check evaluates the quality of external motion cues per location. A confidence map ( $c_t$ ) is generated to gate the fusion, weighting the external and internal branches. A compact residual corrector further compensates for systematic biases in external motion.

2. Latent World Transformer (LWT) with Adaptive Sparse Attention

The core processing unit is the Latent World Transformer, which predicts high-resolution latent states from low-resolution inputs.

Adaptive Sparse Attention: To avoid quadratic complexity, spatial tokens are partitioned into blocks. The model employs a differentiable soft top-k selection during training and hard top-k retrieval during inference. This mechanism preserves local context while selectively retrieving a small set of highly relevant remote blocks, balancing locality with targeted non-local interactions.
Causal Memory: A Key-Value Memory Cache ( $M_{t-1}$ ) stores summaries from previous frames, ensuring strict causality (no future frame look-ahead).

3. Controllability Interface

MotionCraft introduces an explicit, user-accessible interface to modulate the latent state before decoding.

Global Scalar ( $\gamma$ ) and Spatial Gate ( $g_t$ ): These parameters allow users to dynamically balance temporal smoothness against reconstruction fidelity. Increasing $\gamma$ monotonically increases motion smoothness while gracefully decreasing PSNR, offering a predictable trade-off.

4. Conditional Decoding and Training

Tiny Conditional Decoder: A compact decoder reconstructs the high-resolution frame using the predicted latent and the current low-resolution input.
Distillation: The decoder is refined via distillation from a stronger teacher model, transferring fine details while maintaining pixel-space supervision.
Training Objectives: The model is trained using a combination of JEPA-style representation loss, latent distillation, flow-based temporal coherence loss, reconstruction loss, perceptual loss, and adversarial loss. Training follows a curriculum where a dense teacher is distilled into a sparse-causal student.

Key Contributions

Unified Motion-Aware Latent Prediction: The paper unifies motion-aware prediction with reliability-guided sparse attention. Unlike prior works that treat motion correction and attention in isolation (e.g., RealBasicVSR or RealViFormer), MotionCraft couples confidence gating with block selection, allowing motion quality to directly inform non-local retrieval.
Adaptive Sparse Attention Mechanism: A novel attention design that preserves locality while retrieving remote blocks, jointly optimized with the motion branch to ensure computational feasibility for high-resolution streaming.
Compact Controllability Interface: A mechanism enabling fine-grained, user-level control over the smoothness-fidelity trade-off via a global scalar and spatial gate, a feature largely absent in previous VSR frameworks.
Streaming-Optimized Architecture: The system is designed for strictly causal inference with zero future-frame look-ahead, utilizing a key-value cache and sparse attention to enable efficient, real-time processing.

Experimental Results

MotionCraft was evaluated against state-of-the-art methods (including RealBasicVSR, RealViFormer, Upscale-A-Video, STAR, SeedVR, and FlashVSR) on multiple datasets (UDM10, MovieLQ, YouHQ40, REDS, SPMCS, VideoLQ, AIGC30).

Reconstruction Fidelity: MotionCraft achieved superior PSNR and SSIM scores across all datasets. For instance, on the REDS dataset, it reached 27.05 dB PSNR and 0.805 SSIM, outperforming the next best method (RealViFormer at 25.96 dB).
Perceptual Quality: The model demonstrated significant improvements in no-reference metrics (MUSIQ, CLIP-IQA, DOVER) and reduced warping errors ( $E^*_{warp}$ ), indicating better temporal coherence and texture synthesis.
Temporal Stability: Quantitative analysis showed lower temporal artifact rates and higher flow agreement compared to baselines. The model remained robust even when external motion estimators were swapped (e.g., from RAFT to GMFlow), with negligible performance degradation.
Efficiency: MotionCraft achieved the highest throughput (18.63 FPS) and lowest peak memory consumption (10.25 GB) among compared methods for 768×1408 resolution videos, while maintaining a competitive parameter count (~1.7B).
Ablation Studies: Incremental ablation confirmed that motion-aware fusion, adaptive sparse attention, and the controllability interface each contributed positively to fidelity, stability, and user control.

Significance and Claims

The paper claims that MotionCraft addresses the critical barriers preventing high-quality VSR from being viable in everyday streaming contexts. By formulating restoration as a motion-aware latent state prediction problem, the framework successfully integrates robust motion estimation, efficient long-range modeling, and explicit user control into a single pipeline.

The authors emphasize that the system enables predictable trade-offs between temporal smoothness and reconstruction fidelity, a capability previously difficult to achieve without sacrificing either quality or efficiency. The use of a "world model" inspired approach allows the system to maintain temporal consistency without the heavy computational burden of full attention or the stochastic discontinuities of standard diffusion models. The work positions MotionCraft as a practical, efficient, and interactively controllable solution for high-fidelity visual upscaling in real-world scenarios.