💻 computer science

SkyReels-V4: Multi-modal Video-Audio Generation, Inpainting and Editing model

SkyReels V4 is a unified multi-modal foundation model that leverages a dual-stream MMDiT architecture and MLLM-based instruction following to simultaneously generate, inpaint, and edit high-fidelity, 1080p, 15-second video with synchronized audio from diverse inputs, utilizing an efficient two-stage strategy for long-duration, cinematic-quality output.

Original authors: Guibin Chen, Dixuan Lin, Jiangping Yang, Youqiang Zhang, Zhengcong Fei, Debang Li, Sheng Chen, Chaofeng Ao, Nuo Pang, Yiming Wang, Yikun Dou, Zheng Chen, Mingyuan Fan, Tuanhui Li, Mingshan Chang, Hao

Published 2026-03-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Guibin Chen, Dixuan Lin, Jiangping Yang, Youqiang Zhang, Zhengcong Fei, Debang Li, Sheng Chen, Chaofeng Ao, Nuo Pang, Yiming Wang, Yikun Dou, Zheng Chen, Mingyuan Fan, Tuanhui Li, Mingshan Chang, Hao Zhang, Xiaopeng Sun, Jingtao Xu, Yuqiang Xie, Jiahua Wang, Zhiheng Xu, Weiming Xiong, Yuzhe Jin, Baoxuan Gu, Binjie Mao, Yunjie Yu, Jujie He, Yuhao Feng, Shiwen Tu, Chaojie Wang, Rui Yan, Wei Shen, Jingchen Wu, Peng Zhao, Xuanyue Zhong, Zhuangzhuang Liu, Kaifei Wang, Fuxiang Zhang, Weikai Xu, Wenyan Liu, Binglu Zhang, Yu Shen, Tianhui Xiong, Bin Peng, Liang Zeng, Xuchen Song, Haoxiang Guo, Peiyu Wang, Max W. Y. Lam, Chien-Hung Liu, Yahui Zhou

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a movie director. In the old days, you had to hire a cameraman to shoot the scene, a sound engineer to record the dialogue, a composer to write the music, and a special effects team to fix mistakes or add new characters later. It was a massive, expensive, and disjointed process.

SkyReels-V4 is like a super-powered "One-Person Movie Studio" that fits in your computer. It doesn't just make a video; it makes the video and the sound at the same time, understands your specific instructions, and can even edit the movie while it's being made.

Here is how it works, broken down into simple concepts:

1. The "Twin Brain" Architecture (Dual-Stream MMDiT)

Think of the model as having two brains working in perfect sync, connected by a shared "translator."

Brain A (The Visuals): This brain paints the movie frames.
Brain B (The Sound): This brain composes the music, dialogue, and sound effects.
The Translator (MLLM): Both brains share a giant, smart language center (based on a Large Language Model). When you say, "A cat fighting a dog in a cyberpunk city with jazz music," this translator understands the meaning of "jazz" and "cyberpunk" and tells both brains exactly what to do so the sound matches the visuals perfectly.

The Magic: Before, if you asked for a video, the sound might be added later and feel out of sync (like a bad dub). SkyReels-V4 generates them simultaneously, so the "jazz" swells exactly when the camera pans, and the "fighting" sounds match the punches.

2. The "Magic Paintbrush" (Unified Inpainting & Editing)

Imagine you have a painting, but you want to change the sky, add a bird, or extend the canvas to make it wider. Usually, you'd need different tools for each job.

SkyReels-V4 uses a single "Magic Paintbrush" that handles everything by treating every task as "filling in the blanks."

Text-to-Video: You give it a blank canvas and a description. It paints the whole movie.
Image-to-Video: You give it the first frame (the blank canvas is half-filled). It paints the rest of the movie.
Video Editing: You give it a movie and a "mask" (a stencil) over the part you want to change (e.g., covering the background). It erases what's under the mask and paints something new, keeping the rest of the movie intact.
Reference: You can show it a photo of a specific shirt and say, "Put this shirt on the character." The model looks at your photo and "inpaints" that exact shirt onto the character in the video.

3. The "Smart Sketch" Strategy (Efficiency)

Creating a 1080p, 32-frames-per-second movie for 15 seconds is like trying to paint a massive mural in real-time. It usually requires a supercomputer and takes forever.

SkyReels-V4 uses a clever trick called "Sketch then Refine":

The Sketch: First, it quickly generates the whole movie at a low resolution (like a rough pencil sketch) to get the movement and timing right.
The Keyframes: At the same time, it paints a few "Keyframes" (important moments) in high definition.
The Refiner: Finally, a specialized "Refiner" module takes that rough sketch and the high-quality keyframes and fills in the gaps, turning the whole thing into a crisp, cinema-quality masterpiece.

This is like an artist quickly sketching a pose to get the anatomy right, then spending time on the details, rather than trying to paint every single hair perfectly from the very first second.

4. What Can It Actually Do?

Make Movies from Scratch: Type "A dragon flying over a volcano," and it gives you a 15-second clip with roaring fire and wind sounds.
Fix Mistakes: If a video has a weird glitch or you want to remove a person from the background, it can "heal" that part of the video.
Change the Story: You can take a video of a person walking and change the background to a beach, or change their clothes, all while keeping the movement smooth.
Sync Sound: It can make a character speak a specific line of dialogue, and their lips will move perfectly to match the words.

Why Is This a Big Deal?

Before this, you needed different AI tools for making videos, adding sound, and editing. They often didn't talk to each other well. SkyReels-V4 is the first "All-in-One" studio that can do all three at a professional, cinema-level quality (1080p resolution) in a single go. It's like going from building a car with a hammer and a saw to driving a fully automated factory that builds the car, paints it, and installs the engine all at once.

1. Problem Statement

Current video generation models face three primary limitations:

Fragmentation: Existing systems typically handle video generation, audio generation, inpainting, and editing as separate tasks or pipelines. Few models unify these capabilities within a single architecture.
Modality Asynchrony: While some models generate video and audio, they often suffer from audio-visual asynchrony (lip-sync mismatches, temporal misalignment) and lack deep semantic alignment between the two modalities.
Input Rigidity: Most state-of-the-art (SOTA) models rely heavily on text prompts. They struggle to accept rich, multi-modal conditioning inputs (e.g., reference images, video clips, masks, and audio references) simultaneously to guide fine-grained generation or editing.
Computational Scalability: Generating high-resolution (1080p), long-duration (15s), high-frame-rate (32 FPS) video with synchronized audio is computationally prohibitive for standard diffusion transformers due to memory and time constraints.

2. Methodology

SkyReels-V4 addresses these challenges through a unified Dual-Stream Multimodal Diffusion Transformer (MMDiT) architecture and a novel efficiency strategy.

A. Architecture: Dual-Stream MMDiT

The model consists of two parallel branches (Video and Audio) sharing a common semantic backbone:

Shared Text Encoder: A powerful Multimodal Large Language Model (MLLM) acts as a shared text encoder. It processes combined prompts (text, image descriptions, video descriptions, audio descriptions) to create a unified semantic context.
Hybrid Transformer Blocks:
- Dual-Stream (Early Layers): Video/Audio tokens and text tokens have separate parameters for LayerNorm, QKV projections, and MLPs but interact via joint self-attention. This ensures strong cross-modal alignment early in the process.
- Single-Stream (Later Layers): Tokens are concatenated and processed with shared parameters for computational efficiency.
- Reinforced Text Conditioning: Additional cross-attention layers allow the video stream to query text embeddings, preserving fine-grained semantic control.
Bidirectional Cross-Attention: To ensure temporal synchronization, the audio stream attends to video features and vice versa at every layer, exchanging synchronization cues throughout the network depth.
Temporal Alignment: Rotary Positional Embeddings (RoPE) are scaled (audio frequencies scaled by ~0.096) to align the different temporal resolutions of video (21 frames) and audio (218 tokens).

B. Unified Inpainting & Editing Framework

SkyReels-V4 unifies generation, inpainting, and editing using a Channel Concatenation formulation on the video branch:

Input Formulation: The input to the video MMDiT is a concatenation of three tensors: Noisy Video Latent ( $V$ ), Conditional Frames ( $I$ ), and a Binary Mask ( $M$ ).
Task Unification:
- Text-to-Video: Mask is all zeros.
- Image-to-Video: First frame is masked as condition ( $M=1$ ).
- Video Extension/Editing: Specific spatiotemporal regions are masked as conditions or targets.
In-Context Learning: Reference images and video clips are encoded and prepended to the latent sequence with Offset 3D RoPE (negative temporal indices) to distinguish them from generated content, allowing the model to learn fine-grained visual patterns (identity, texture) directly.

C. Efficiency Strategy: Joint Generation & Refinement

To achieve 1080p/32FPS/15s generation without prohibitive costs:

Base Model: Generates a low-resolution full sequence and high-resolution keyframes simultaneously.
Refiner Module: A dedicated Super-Resolution (VSR) and Frame Interpolation model takes the base outputs. It uses Video Sparse Attention (VSA) to reduce computational cost by ~3x while reconstructing a temporally consistent, high-fidelity video.

D. Training Pipeline

The model undergoes a progressive multi-stage training strategy:

Video Pretrain: T2I $\to$ T2V $\to$ Inpainting/Editing $\to$ Multi-modal conditioning (Image/Video Ref).
Audio Pretrain: Trained from scratch on hundreds of thousands of hours of speech and audio data.
Joint Training: Simultaneous training on T2V, T2A, and T2AV tasks to enforce synchronization.
Supervised Fine-Tuning (SFT): High-quality curated data with multi-modal inputs to refine instruction following and alignment.

3. Key Contributions

First Unified Foundation Model: SkyReels-V4 is the first model to simultaneously support rich multi-modal inputs (text, images, video, masks, audio), joint video-audio generation, and a unified framework for generation, inpainting, and editing.
Dual-Stream MMDiT: Introduces a symmetric twin-backbone architecture with bidirectional cross-attention and shared MLLM encoders, achieving superior audio-visual alignment.
Channel-Concatenation Inpainting: A novel formulation that treats diverse video manipulation tasks (I2V, extension, editing) as a single inpainting problem under a unified interface.
Scalable High-Fidelity Generation: Proposes a joint low-res/high-res keyframe generation strategy combined with a VSA-based refiner, enabling cinematic-quality (1080p, 32 FPS, 15s) generation with synchronized audio.

4. Results

Artificial Analysis Arena:
- Rank #1 in Text-to-Video with Audio (surpassing Kling 3.0 and Veo 3.1).
- Rank #2 in Text-to-Video without Audio.
- Rank #4 in Image-to-Video with Audio.
SkyReels-VABench (Human Evaluation):
- Evaluated against SOTA baselines (Veo 3.1, Kling 2.6, Seedance 1.5 Pro, Wan 2.6).
- Achieved the highest overall average score on a 5-point Likert scale.
- Demonstrated particular strength in Instruction Following and Motion Quality.
- Consistently outperformed baselines in Good-Same-Bad (GSB) comparisons across all five dimensions (Instruction Following, AV Synchronization, Visual Quality, Motion Quality, Audio Quality).

5. Significance

SkyReels-V4 represents a paradigm shift in video generation by moving from fragmented, unimodal, or single-task pipelines to a holistic, multi-modal foundation model.

Creative Workflow: It enables complex, cinema-level storytelling where users can generate, edit, and extend videos with synchronized sound using natural language, visual references, and audio cues in a single step.
Technical Benchmark: It sets a new standard for efficiency and quality in high-resolution, long-duration video generation, proving that joint audio-video synthesis at 1080p is computationally feasible.
Future Impact: By unifying generation, inpainting, and editing, it paves the way for next-generation creative tools that seamlessly blend visual and auditory storytelling, addressing the historical gap between "silent film" and "talkies" in the AI era.