💻 computer science

From Dense Prediction to Visual Editing: Structured Supervision for Unified Image and Video Creation

This paper proposes a unified image and video creation framework that enhances precise, temporally consistent editing by incorporating depth and surface-normal prediction as structured visual supervision within a shared multimodal diffusion transformer backbone, thereby transferring useful structural knowledge to downstream generation tasks.

Original authors: Zhefan Rao, Bin Zou, Haoxuan Che, Xuanhua He, Chong Hou Choi, Yanheng Li, Rui Liu, Qifeng Chen

Published 2026-08-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhefan Rao, Bin Zou, Haoxuan Che, Xuanhua He, Chong Hou Choi, Yanheng Li, Rui Liu, Qifeng Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where a computer can not only paint a picture from a description but also take an existing video and change a specific detail—like turning a rainy day into a sunny one or adding a running dog—without altering the rest of the scene or making the video flicker. This is the promise of unified visual creation, a field where artificial intelligence learns to generate and edit both images and videos from a single set of instructions. For these systems to work well, they must understand not just the meaning of words like "add a dog," but also the physical structure of the scene: where objects are, how they overlap, and how they move through time. Without this structural understanding, the computer might obey the instruction to add a dog but accidentally erase the person standing next to it, or make the new animal flicker in and out of existence as the video plays. The challenge has been teaching a single model to follow diverse commands while keeping the underlying geometry and identity of the visual world intact.

Researchers have developed a new approach that teaches the computer to see the world in three dimensions while it learns to create it. Instead of just showing the model pairs of images or videos and asking it to guess the changes, the team added a new layer of training. They asked the model to predict two specific things about every image it processed: the depth of the scene, which tells how far away objects are, and the surface normals, which describe the direction a surface is facing, like the tilt of a wall or the curve of a ball. These predictions are not the final goal; the researchers are not trying to build the best possible 3D scanner. Instead, they use these tasks as a way to force the model to pay attention to the hidden skeleton of the image. By learning to reconstruct these structural maps, the model gains a better sense of where boundaries are and how objects relate to one another, which helps it preserve those details when it is asked to edit the content later.

To make this work, the team built a system that separates the meaning of a command from the visual evidence it needs to follow. One part of the system reads the text instructions and understands the intent, while another part looks at the actual pixels of the source image or video to keep the spatial details sharp. These two streams of information are combined in a shared brain that learns to generate new content. Crucially, the researchers also created a special method for generating training data. Rather than just editing the first frame of a video and copying that change forward, which often leads to errors, they taught the system to generate pairs of videos where the change happens at different times. One video might show a scene, and the paired video shows the same scene with a change that starts halfway through or ends before the clip is over. This teaches the model exactly when and where to apply a change, ensuring that the rest of the video remains stable and consistent.

The results of this approach show that adding these structural lessons significantly improves the quality of the edits. When tested on a standard set of video editing tasks, the model that received this extra training scored higher than previous systems, particularly in tasks that required adding or changing local objects without disturbing the background. The improvement was most noticeable in the ability to keep the unedited parts of the video looking natural and steady. The researchers found that this method worked well across a wide range of tasks, from creating new videos from scratch to editing existing ones based on text instructions or reference images. They demonstrated that a single model could handle all these different jobs, achieving a high overall score that surpassed other unified systems currently available.

However, the researchers are careful to define the limits of their success. They explicitly state that their goal was not to create a superior tool for measuring depth or surface angles, and they did not test their model on those specific tasks. The value of the depth and normal predictions lies entirely in how they help the creation process, not in the accuracy of the maps themselves. The study suggests that this transfer of structural knowledge is real and measurable, but it does not claim that the model has achieved a general understanding of the physical world. The system still struggles with very long videos, where characters might drift or change appearance, and it can have trouble with very small objects or when multiple references conflict. The work represents a significant step toward more reliable visual editing, showing that teaching a model to understand the structure of a scene makes it a better artist, even if the artist is not yet a perfect architect.

Technical Summary: From Dense Prediction to Visual Editing

1. Problem Statement

Unified visual creation aims to enable a single model to generate and edit both images and videos based on diverse instructions (text, visual references, keyframes). While large video diffusion models have established strong synthesis priors, existing unified systems struggle to balance task breadth with faithful control.

The core challenge lies in the conflict between semantic intent (what to create/change) and structural preservation (where, when, and relative to which geometry the change should occur). Current approaches often rely on semantic-only conditioning or creation-only training. This leads to recognizable failures in editing tasks, such as:

Drifting from reference identities.
Modifying unedited regions.
Distorting scene geometry.
Temporal flickering.

These failures occur because high-level conditioning (e.g., abstract visual tokens) does not explicitly retain aligned evidence regarding object boundaries, occlusion, surface orientation, or motion continuity. Furthermore, while existing editors use attention control or feature propagation to mitigate symptoms, the shared backbone is rarely explicitly supervised to predict the dense structure required for such preservation.

2. Methodology

The authors propose a unified framework that treats depth and surface-normal prediction as structured visual supervision within the same image-form denoising interface used for creation. The framework is built upon three main pillars:

A. Unified Visual Creation Framework

The system casts image/video generation, editing, reference-conditioned creation, and dense prediction as conditional denoising in a single latent space using a shared Multimodal Diffusion Transformer (MMDiT).

Decoupled Conditioning: The framework separates semantic interpretation from spatially aligned visual injection.
- Semantic Pathway: Uses frozen vision-language features (Qwen2.5-VL, SigLIP, Glyph-ByT5) to interpret instructions and cross-modal relations.
- Visual Pathway: Injects spatially aligned VAE latents from source images, videos, and references directly into the MMDiT.
Shared Objective: Both creation outputs and dense prediction targets (depth/normal maps) share the same target slots and denoising objective ( $L_{denoise}$ ). No perception-specific output head or inference-time module is added.

B. Structured Visual Supervision

Instead of treating dense prediction as a separate task, the model is trained to reconstruct depth and surface normals as image-form denoising targets.

Mechanism: Normalized depth is rendered as an RGB image; surface normals are represented as three-channel orientation maps.
Goal: To explicitly teach the backbone spatial organization (layout, occlusion, boundaries, surface geometry) without requiring these maps at inference time. The hypothesis is that learning to reconstruct dense structure improves structure-sensitive creation behavior (editing locality and preservation).

C. Mutual Context Attention (MCA) and Data Construction

To address the need for paired evidence of permitted changes, the authors introduce Mutual Context Attention (MCA) for constructing aligned source-target video pairs.

MCA Mechanism: Two video-denoising branches (source and target) share context during synthesis while allowing the target branch to realize specified edits. This is achieved by manipulating Key/Value interactions (e.g., SWAP KV, CONCAT KV) at specific layers and denoising steps.
- Early steps: Strong coupling (e.g., SWAP KV) aligns scene structure and identity.
- Middle steps: Softer coupling allows for editability.
- Late steps: Release prevents copying texture artifacts.
Temporal Flexibility: Unlike standard pipelines that edit the first frame and propagate, MCA allows edits to begin after the first frame, end before the last, or occupy intermediate intervals.
Pipeline: An automated pipeline expands prompts, generates MCA-coupled pairs, filters invalid responses, and verifies alignment via VQA.

D. Progressive Training Curriculum

The training follows a three-stage curriculum to stabilize the learning of new condition pathways without disrupting pretrained priors:

Stage I (Conditioning Alignment): Mixes text-guided generation, VLM-guided reconstruction, and consistency preservation to align conditions.
Stage II (Editing & Structure): Introduces instruction editing, depth/normal targets, and MCA pairs.
Stage III (Reference-Conditioned Creation): Expands to subject/reference generation, reference-based editing, and keyframe propagation.

3. Key Contributions

Dense Prediction as Supervision: Introducing depth and surface-normal prediction as structured visual supervision for a shared image/video creation backbone, improving downstream structural preservation without adding inference-time components.
Unified Denoising Framework: Developing a framework that decouples semantic intent from spatially aligned visual evidence, supporting seven generation and editing task types (including T2I, T2V, S2V, TI2I, TV2V, II2I, IV2V) in a single checkpoint.
Structure-Aware Training Recipe: Establishing a training methodology combining dense supervision, MCA-generated editing pairs, and progressive curriculum design, validated through unified and editing-focused experiments with controlled ablations.

4. Results

The framework was evaluated across unified creation tasks and specific editing benchmarks:

Unified Performance: The single checkpoint achieved the highest reported Overall score of 4.15 in unified system comparisons (outperforming VINO at 4.12 and UniVideo at 4.03). It maintained competitive generation scores while significantly improving editing scores.
Impact of Dense Supervision: Adding dense supervision (depth/normal targets) improved the OpenVE Overall score from 3.98 to 4.06. The most significant gains were in Local Add (3.92 to 4.18) and Local Change (3.89 to 4.11), supporting the hypothesis that dense tasks improve structure-sensitive editing.
Editing Benchmarks:
- OpenVE-Bench: Achieved 4.43 Overall.
- InsEdit-Bench: Achieved 4.61 Overall, with high scores in Instruction Compliance (4.50), Temporal Visual Quality (4.54), and Unedited Region Preservation (4.80).
Data Efficiency: Replacing half of 4K conventional pairs with 2K MCA pairs improved the downstream score from 3.96 to 4.08, demonstrating the high quality of MCA-generated data. MCA pairs showed superior alignment (0.84 vs 0.67 for independent generation) and lower artifact rates (9.8% vs 18.6%).

5. Significance and Claims

The paper claims that perception-oriented dense supervision transfers useful structural knowledge to downstream creation, specifically enhancing editing locality and the preservation of identity and geometry.

Modest Scope: The authors explicitly state they do not claim superiority as a standalone dense predictor. The evaluation focuses on the transfer of structural knowledge to generation and editing, not on general perceptual intelligence or state-of-the-art dense prediction accuracy.
Mechanism: The results support a "deliberately bounded conclusion": learning to reconstruct dense structure improves structure-sensitive creation behavior. The dense tasks teach the backbone what scene structure is, while MCA pairs teach where and when that structure should be preserved or changed.
Limitations: The authors acknowledge limitations, including the lack of independent depth/normal benchmarks, the reliance on VLM judges for evaluation, and practical limits such as identity drift in longer videos and insufficient precision for small objects. They do not claim the framework solves all production-level challenges.