Technical Summary: RVEDiT – Implicit Reasoning in Diffusion Transformers for Video Editing

Problem Statement

Instruction-based video editing requires transforming a source video according to a natural-language instruction while preserving irrelevant content and maintaining temporal coherence. The authors argue that existing Diffusion Transformer (DiT) editors struggle with this task due to two structural limitations in current pipelines:

1. Undifferentiated Conditioning: In standard DiTs, textual instructions and visual tokens are concatenated into a single joint sequence processed uniformly by every transformer block. This forces a single token stream to encode both high-level editing intent and low-level visual targets simultaneously, leaving no dedicated "workspace" for reasoning.
2. Unsupervised Internal Reasoning: The cross-attention patterns that govern the edit are supervised only indirectly through pixel-level reconstruction losses (e.g., flow-matching). Consequently, the model's internal reasoning trace (the attention maps encoding which text tokens read which visual regions) is underconstrained. The model learns to minimize reconstruction error but is not explicitly taught to establish the fine-grained correspondences required for instruction-faithful edits.

The paper posits that instruction-based video editing is fundamentally a latent reasoning problem that must be solved natively within the DiT backbone, rather than outsourcing reasoning to external Multi-Modal Large Language Models (MLLMs) which can introduce hallucinations, or treating the backbone as a black box.

Methodology: RVEDiT

The authors propose RVEDiT (Reasoning Video Editing DiT), a framework designed to structure the conditioning stream and supervise the internal reasoning process. It consists of two complementary components:

1. Granularity-Routed Token Conditioning (GRTC)

This component addresses the undifferentiated conditioning bottleneck by introducing a depth-aware routing mechanism:

• Learnable Editing Tokens: A set of $K$ learnable editing tokens ($X_e$) is distilled from an MLLM (specifically Qwen2.5-VL) given the source video and instruction. These tokens act as an abstract, globally-contextualized carrier of the editing intent.
• Depth-Based Routing: The tokens are routed to different layers of the DiT based on their semantic granularity:
  • Shallow Layers ($\ell_{idx} \le L_s$): Attend only to the learnable editing tokens ($X_e$). This allows the early blocks to establish a global editing intent without being diluted by thousands of pixel-level tokens.
  • Deep Layers ($\ell_{idx} > L_s$): Attend to the full sequence $[X_e; X_v; X_\ell]$, where $X_v$ are native visual tokens and $X_\ell$ are textual tokens. This allows the deeper blocks to ground the global intent into specific regions, textures, and motions.
• Effect: This creates a coarse-to-fine editing process within the backbone, separating the formation of global intent from local refinement.

2. Reference-Anchored Attention Alignment (RAAA)

This component addresses the unsupervised reasoning bottleneck by introducing a training-time regularizer that does not incur inference costs:

• Reference Branch: During training, a second forward pass is performed through the same DiT (with shared weights) conditioned on the ground-truth target video ($v^*$) and its caption ($\ell^*$). This branch generates a "successful" reasoning trace (cross-attention maps) that serves as an anchor.
• Mutual Information Maximization: The method maximizes the mutual information between the cross-attention features of the editing branch (conditioned on source + instruction) and the reference branch (conditioned on target).
• Objective: Using a symmetric InfoNCE estimator, the model is encouraged to align the attention geometry of the editing branch with the reference branch at a randomly sampled depth. This teaches the model where and how to attend to reflect a successful edit, rather than just minimizing pixel reconstruction error.
• Inference: The reference branch is discarded after training, adding no computational overhead to deployment.

The final training objective combines the standard flow-matching loss ($\mathcal{L}_{fm}$) with the alignment term ($\mathcal{L}_{align}$):
$$\mathcal{L} = \mathcal{L}_{fm} + \lambda_{align} \mathcal{L}_{align}$$

Key Contributions

1. Problem Formulation: The paper identifies and formalizes two structural bottlenecks in existing DiT pipelines (undifferentiated conditioning and unsupervised reasoning) that hinder implicit reasoning in video editing.
2. RVEDiT Framework: A DiT-native framework that internalizes reasoning through:
   • Granularity-Routed Token Conditioning: Establishes a coarse-to-fine editing process by routing MLLM-derived tokens to shallow layers and native tokens to deep layers.
   • Reference-Anchored Attention Alignment: Actively supervises the reasoning pathway by maximizing mutual information between editing and reference attention traces.
3. Empirical Validation: Extensive experiments demonstrate that RVEDiT outperforms state-of-the-art open-source baselines, particularly in localized, compositional, and fidelity-sensitive edits.

Experimental Results

The authors evaluated RVEDiT on the OpenVE-Bench against baselines including VACE, OmniVideo, ICVE, Lucy-Edit, DITTO, and Kiwi-Edit. Evaluation was conducted using an MLLM-as-a-judge protocol (Gemini-3.1-Pro and Qwen3.5-VL-35B) scoring three axes: Instruction Compliance (IC), Consistency Fidelity (CF), and Visual Quality (VQ).

• Quantitative Performance: RVEDiT achieved the best overall scores under both judges (77.42 with Qwen3.5-VL-35B and 62.48 with Gemini-3.1-Pro).
• Specific Gains: The model showed particularly strong performance on Local Add and Local Change tasks, which require precise modification and preservation of surrounding content. It also performed well on Global Style transfer.
• Ablation Studies:
  • Removing GRTC degraded performance on tasks requiring clear global intent (Local Add, Global Style).
  • Removing RAAA caused a drop in tasks requiring fine-grained positional correspondence (Local Add, CF, and VQ metrics).
  • Removing both components resulted in significant failure, particularly in Local Remove tasks (IC dropped by nearly 17 points compared to the full model).
• Mechanistic Analysis: Attention mass analysis confirmed that shallow layers focus on editing tokens (global intent) while deeper layers shift to native tokens (local refinement), validating the coarse-to-fine design.

Significance and Claims

The paper claims that RVEDiT successfully demonstrates that explicitly shaping the internal reasoning process of generative backbones is a viable path toward robust and controllable video editing. By treating the task as a latent reasoning problem and addressing the structural bottlenecks of standard DiTs, the method achieves:

• Instruction Faithfulness: Better adherence to complex natural language instructions.
• Source Preservation: Improved retention of non-edited content and scene structure.
• Temporal Coherence: Reduced flickering and artifacts across frames.

The authors emphasize that these improvements are achieved without external planning modules or additional inference costs, suggesting that end-to-end generative passes can be endowed with implicit reasoning capabilities through architectural design and targeted training objectives.