💻 computer science

Bernini: Latent Semantic Planning for Video Diffusion

Bernini is a unified video generation and editing framework that leverages a division of labor where an MLLM-based planner performs semantic reasoning in ViT embedding space to guide a DiT-based renderer, achieving state-of-the-art performance through efficient, separate training and novel components like Segment-Aware 3D Rotary Positional Embedding.

Original authors: Bernini Team, Chenchen Liu, Junyi Chen, Lei Li, Lu Chi, Mingzhen Sun, Zhuoying Li, Yi Fu, Ruoyu Guo, Yiheng Wu, Ge Bai, Zehuan Yuan

Published 2026-05-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Bernini Team, Chenchen Liu, Junyi Chen, Lei Li, Lu Chi, Mingzhen Sun, Zhuoying Li, Yi Fu, Ruoyu Guo, Yiheng Wu, Ge Bai, Zehuan Yuan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: The Architect and the Builder

Imagine you want to build a custom house. You have two experts:

The Architect (The MLLM): This person is brilliant at understanding your vague ideas, reading complex blueprints, and figuring out what the house should look like. They are great at reasoning but terrible at actually laying bricks.
The Builder (The Diffusion Model): This person is a master craftsman. They can lay bricks, paint walls, and install windows with photorealistic perfection. However, if you just say "make it nice," they might build a castle when you wanted a cottage. They need very specific instructions.

Bernini is a system that unites these two experts. Instead of trying to force the Architect to lay bricks or the Builder to do complex reasoning, Bernini gives them a specialized language to talk to each other.

How It Works: The "Secret Blueprint"

In the past, trying to combine these two types of AI was messy. Bernini solves this by creating a "division of labor":

The Planning Phase (The Architect):
When you give Bernini a command (like "Change the weather to a stormy night and make the dog happy"), the MLLM Planner doesn't try to draw the video. Instead, it acts like an architect sketching a high-level blueprint.
- The Secret Sauce: This blueprint isn't a picture or a sentence. It's a set of mathematical codes (called "ViT embeddings") that represent the meaning of the scene. It's like the Architect handing the Builder a list of coordinates and emotional tones rather than a drawing.
The Rendering Phase (The Builder):
The DiT Renderer (the Builder) receives this blueprint. It also looks at the original video (if you are editing) to keep the background consistent. Using the blueprint as a guide, it generates the actual pixels, frame by frame, creating a photorealistic video.

Why is this cool? Because the Architect and Builder can be trained separately. The Architect keeps getting smarter at understanding human language, and the Builder keeps getting better at making pretty pictures. They only need to learn how to read each other's "secret blueprint" language, which is much easier than retraining the whole system from scratch.

The New Tools: "Name Tags" and "Thinking Steps"

To make this work perfectly, the researchers added two clever tricks:

The "Name Tag" System (SA-3D RoPE):
Imagine you are in a crowded room where everyone is wearing the same outfit. If you shout "Look at the person in the red hat," it's confusing if there are three people in red hats.
In video editing, the AI often has to look at the original video, a reference image, and the new video all at once. Sometimes, a pixel in the original video is in the exact same spot as a pixel in the new video.
Bernini gives every piece of the puzzle a "Name Tag" (a segment index). This tells the AI, "This red hat belongs to the original video, and that red hat belongs to the new video." This prevents the AI from getting confused and mixing up the source with the result.
The "Thinking Aloud" Step (Chain-of-Thought):
Sometimes, a simple command isn't enough. If you say "Make it look like a cartoon," the AI might just change the colors.
Bernini's planner is taught to think out loud before it draws the blueprint. It breaks the task down: "First, I need to understand the lighting. Second, I need to change the texture of the skin. Third, I need to adjust the motion." This step-by-step reasoning helps the AI handle complex tasks, like changing the weather so that a fire in the video naturally goes out (causal reasoning).

What Can It Do?

The paper shows Bernini is a "Swiss Army Knife" for video. It can:

Text-to-Video: Create a video from scratch based on a description.
Video-to-Video Editing: Change the style, add or remove objects, or change the weather in an existing video.
Reference-Guided Editing: "Make the person in this video look like the person in this photo."
Motion Transfer: "Make the person in this photo dance like the person in this video."

The Results: Winning the Race

The researchers tested Bernini against other top-tier video AI models (like Kling, Wan, and others) on a new benchmark they created called Bernini-Bench.

The Score: Bernini won the "Video Editing Leaderboard," beating the competition in consistency and following instructions.
The Key Win: It was particularly good at keeping the video consistent. When you edit one part of a video, the rest of the video doesn't warp or glitch. It stays true to the original scene while changing only what you asked for.

Summary

Bernini is like hiring a genius architect to write a perfect set of instructions for a master builder. By letting them speak a specialized "blueprint language" and giving them tools to keep track of which part of the video is which, Bernini creates videos that are not only beautiful but also smart enough to understand complex, logical changes.

Technical Summary: Bernini – Latent Semantic Planning for Video Diffusion

1. Problem Statement

Multimodal Large Language Models (MLLMs) and diffusion models have matured along largely independent trajectories. MLLMs excel at semantic reasoning, interpreting complex instructions, and grounding answers in multimodal contexts, while diffusion models (specifically Diffusion Transformers or DiTs) have become the standard for photorealistic image and video synthesis. However, unifying these two families into a single system that can both understand intent and generate desired outputs remains an open challenge. Existing approaches often struggle to transfer the deep semantic understanding of MLLMs directly into the pixel-level generation of diffusion models without compromising the pretrained strengths of either component or requiring excessive joint training that leads to catastrophic forgetting.

2. Methodology

Bernini proposes a unified framework that decouples semantic planning from pixel rendering, utilizing a simple division of labor:

The Planner (MLLM): Handles semantic reasoning and predicts the target visual representation.
The Renderer (DiT): Synthesizes pixels based on high-level semantic guidance and low-level visual features.

2.1 Architecture

The framework consists of two primary components connected via a semantic interface in the ViT embedding space.

MLLM-based Planner:
- Unified Input: Text, source images, source videos, and target outputs are serialized into a unified 1D token sequence.
- Masked Generative Modeling: Instead of generating text tokens, the planner predicts target visual semantics as dense ViT embeddings. During training, target tokens are randomly masked, and the model infers them from the context. During inference, the model iteratively refines the target representation from fully masked tokens to a complete sequence.
- Chain-of-Thought (CoT): The planner incorporates a CoT mechanism, performing reasoning in the latent space (via self-text and self-vision-text reasoning) before producing the final semantic embedding. This allows the model to decompose complex editing tasks into intermediate visual states.
- ViT Embedding Decoder: A lightweight decoder (MLP + ResNet head) maps the planner's hidden states back to the ground-truth ViT embedding space, trained via flow matching.
DiT-based Renderer:
- Flow Matching: The renderer operates in the VAE latent space, performing flow-matching denoising to synthesize the final video.
- Conditioning: It is conditioned on the semantic embeddings from the planner (via cross-attention), text features, and source VAE features (for detail preservation in editing tasks).
- Segment-Aware 3D RoPE (SA-3D RoPE): To handle multiple visual inputs (e.g., source video, reference image, target) within a unified sequence, Bernini introduces SA-3D RoPE. Standard 3D Rotary Positional Embeddings (RoPE) struggle when tokens from different segments share the same $(t, h, w)$ coordinates. SA-3D RoPE assigns a unique segment index to each visual input and incorporates this into the rotary frequency vector, introducing a segment-dependent global phase modulation. This allows the attention mechanism to distinguish tokens from different segments while preserving spatiotemporal modeling properties.

2.2 Training Strategy

Bernini employs a three-stage training pipeline to preserve the pretrained capabilities of both components while aligning them:

Stage I (MLLM Pretraining): The planner and ViT decoder are trained on a mix of text-to-image, text-to-video, and editing data (image/video pairs). The objective is to transform the MLLM into a semantic planner capable of inferring target visual representations. A task-dependent mask ratio strategy is used to control information leakage, forcing the planner to infer semantics from context rather than visible target tokens.
Stage II (DiT Pretraining): The renderer is trained independently on a large mixture of generation and editing data to ensure high-fidelity synthesis and source preservation.
Stage III (Joint Training): The planner and renderer are lightly co-trained. This stage aligns the semantic planning (ViT space) with visual rendering (VAE space) without excessive interference, allowing the system to translate multimodal reasoning into faithful visual outputs.

2.3 Data Construction

The paper details a comprehensive data construction pipeline to support diverse tasks:

Pre-training Data: 20 million video pairs and 30 million image pairs extracted from general corpora and tutorial videos, filtered by similarity and annotated with dense prompts.
Editing Data: High-quality video-to-video (V2V) data is synthesized using propagation-based methods combined with strong image editing models. A dual-branch framework (Image-to-Video + Video-to-Video) is used to generate motion-aware editing data, ensuring natural human-object interactions.
Reference-Guided Data: Pipelines for Reference-to-Video (R2V) and Reference+Video-to-Video (RV2V) ensure identity preservation (especially for humans) by sourcing references from real footage rather than generated images.
Reasoning-Augmented Data: Explicit Chain-of-Thought data is constructed, where MLLMs rewrite instructions into structured intermediate prompts or generate visual intermediate states to guide the editing process.

3. Key Contributions

Unified Framework with Latent Semantic Interface: Bernini unifies generation and editing by using the MLLM's native ViT embedding space as a semantic bridge to the diffusion generator. This allows pretrained understanding to transfer directly into generation, enabling strong generalization across diverse video tasks.
Scalable Data Construction: The authors designed a suite of data pipelines yielding a large-scale, multi-task corpus. This includes video/image-pair pretraining, high-quality propagation-based editing data, motion-aware data, and reasoning-augmented datasets, addressing the scarcity of high-quality video editing data.
Segment-Aware 3D RoPE: A novel positional embedding mechanism that disambiguates visual tokens from different segments (source, reference, target) sharing the same spatiotemporal coordinates, crucial for multi-modal input handling.
State-of-the-Art Performance: Bernini achieves top-tier results across a wide range of benchmarks, including OpenVE-Bench, OpenS2V-Eval, and the newly proposed Bernini-Bench.

4. Experimental Results

Video Editing (Bernini-Bench): Bernini outperforms strong baselines like Kling O3 and Wan2.7. On the Bernini-V2V benchmark, it achieves an Overall Score (OS) of 3.49 (vs. 3.30 for Wan2.7) and demonstrates superior Video Consistency (VC), preserving non-edited regions better than competitors. In human side-by-side evaluations, Bernini shows significant advantages in consistency and instruction following.
Public Benchmarks:
- OpenVE-Bench: Bernini scores 4.04 overall, significantly outperforming the previous best (VINO at 3.18).
- EditVerse: Achieves the highest editing quality score (8.02) and text alignment.
- FiVE: Attains state-of-the-art performance in structure preservation, background preservation, and editing accuracy.
Video Generation:
- Text-to-Video (VBench): Bernini maintains the base quality of its foundation model (Wan2.2-A14B), achieving a Total score of 84.64 compared to the base's 84.79, proving the unified design does not degrade generation quality.
- Subject-to-Video (OpenS2V-Eval): Bernini achieves the highest Total score (62.94) among all competitors. Notably, it achieves a FaceSim score of 78.20, exceeding the next-best baseline (Kling O3 at 57.20) by over 20 points, demonstrating superior face-identity preservation.
Ablation Studies: Experiments confirm the necessity of the ViT semantic interface, the MLLM planner, and the SA-3D RoPE. Removing SA-3D RoPE leads to reference leakage artifacts, while removing the planner degrades instruction following.

5. Significance and Claims

The paper claims that Bernini represents a significant step toward unifying multimodal understanding and generation. By anchoring the interface in the MLLM's ViT embedding space, the framework allows the two components to be trained largely independently, preserving their respective pretrained strengths (reasoning for the MLLM, synthesis for the DiT) while enabling effective collaboration.

The authors emphasize that Bernini's ability to generalize to challenging editing tasks—such as causal reasoning ("What happens when it rains?"), focus shifts, and complex motion changes—stems from the integration of Chain-of-Thought reasoning and the semantic planning paradigm. The model does not merely memorize transformations but learns a transferable, compositional instruction-following ability.

Limitations: The authors modestly note that Bernini is limited by the capabilities of its underlying foundation models (Qwen2.5-VL and Wan2.2). In complex scenarios, it still relies on an external LLM rewriter to provide sufficiently detailed instructions, suggesting its native reasoning is not yet fully sufficient for the most challenging edits. Additionally, while it leads in consistency, its raw visual quality in subject-to-video generation still lags slightly behind the strongest closed-source systems like Wan2.7.