Technical Summary: Aurora – Unified Video Editing with a Tool-Using Agent

1. Problem Statement

Recent unified video editing models have converged on a design where a single diffusion transformer jointly consumes text instructions, source videos, and reference images (or masks) to perform diverse tasks such as replacement, removal, style transfer, and insertion. While flexible, this architecture assumes that users provide model-ready conditioning inputs: precise text, specific reference images for visual patterns, and spatial masks for localized edits.

In practice, raw user requests often suffer from underspecification along two axes:

1. Visual Underspecification: The user names a specific entity or pattern (e.g., "a Burberry check scarf" or "The Great Wave off Kanagawa") but provides no reference image.
2. Textual Underspecification: The edit is described indirectly (e.g., "change the blue item to a vehicle that runs much faster"), requiring the model to infer concrete goals.

Current unified models lack the mechanism to infer, ground, or fill these missing conditions before generation begins. This creates a gap between the rich conditioning channels available to the model and the incomplete information provided by the user.

2. Methodology: The Aurora Framework

Aurora addresses this gap by decoupling condition construction from video generation. It introduces an agentic framework pairing a tool-augmented Vision-Language Model (VLM) agent with a unified video diffusion transformer (DiT).

2.1 The VLM Agent (Condition Construction)

The agent, implemented as a LoRA-finetuned Qwen3-VL-8B-Instruct, acts as a pre-processor. It takes a source video ($V_{src}$), a raw user instruction ($y$), and optional user references ($R$) to generate a structured edit plan:
$$(y', c, q, m) = \pi_\phi(V_{src}, y, R)$$
Where:

• $y'$ (Rewritten Instruction): A standardized, model-ready instruction that resolves textual underspecification.
• $c$ (Task Label): A canonical edit type (e.g., replace_object, remove_object) used for routing.
• $q$ (Image Search Query): An optional query to trigger a web image search tool (via Serper API) if a reference image is missing.
• $m$ (Mask Phrase): An optional phrase to trigger a segmentation tool (GroundingDINO + SAM) if spatial grounding is missing.

Tool Execution:
If $q$ or $m$ are non-empty, the agent invokes external tools:

• Image Search: Retrieves candidate images and selects the most relevant one.
• Segmentation: Generates a mask for the target region and composites it onto the source video.

The final output is a unified conditioning tuple $\mathbf{x} = (y', V_{src}, R^+)$, where $R^+$ includes the original user references plus any retrieved images or masked images. The task label $c$ is used during planning but is not fed to the video model.

2.2 The Video DiT (Video Generation)

The video generation backbone is a unified Diffusion Transformer (DiT) built upon Wan2.2-TI2V-5B. It consumes the conditioning tuple $\mathbf{x}$ via two paths:

1. Multimodal Context Path: A frozen Qwen3.5-4B encoder processes the rewritten instruction ($y'$), sampled source frames, and reference images ($R^+$) into a shared multimodal context ($T_{ctx}$) for cross-attention.
2. Latent Token Path: Noisy video tokens, source tokens, and reference tokens are concatenated along the token-sequence dimension.
   • Zero-Timestep Modulation: To preserve unedited content and reference identity, source and reference tokens are modulated with a zero-timestep embedding ($a(0)$), while noisy tokens follow the standard flow-matching trajectory ($a(t)$). This ensures conditioning tokens remain fixed while the video denoises.

2.3 Training Objectives

• Video DiT: Trained with flow matching on a mixture of image editing, instruction-based video editing, and reference-guided video editing data. It learns to handle empty, user-provided, and tool-retrieved references uniformly.
• VLM Agent:
  • Supervised Fine-Tuning (SFT): Trained on 25k planning examples and 10k reference-image selection examples. The target is the complete 4-field plan, teaching the agent to rewrite instructions, route tasks, and select tools.
  • Preference Alignment (DPO): Trained on 1.8k preference pairs to resolve boundary cases (e.g., false image search triggers, ambiguous mask phrases, constraint-losing rewrites). This sharpens decision boundaries without mixing video sampling variance into the learning objective.

3. Key Contributions

1. Aurora Framework: An agentic video editing system that maps raw, underspecified requests to complete, model-ready conditioning tuples using a tool-augmented VLM agent.
2. AgentEdit-Bench: A new benchmark specifically designed to evaluate video editing under textual and visual underspecification. It contains 150 cases across five task types (IP replacement, IP addition, IP background change, reasoning, and removal) where the raw request omits necessary conditions.
3. Training Data & Alignment: A dataset construction pipeline for VLM agent planning and reference selection, combined with DPO-based preference alignment to handle edge cases in tool triggering and instruction rewriting.
4. Model Transferability: Demonstration that the Aurora VLM agent can be paired with frozen, existing unified video editing models (e.g., UniVideo, Kiwi-Edit) to improve their performance on underspecified requests.

4. Experimental Results

• Performance on AgentEdit-Bench:
  • The Aurora video DiT alone (without the agent) scores 74.7, outperforming UniVideo (67.0) and Kiwi-Edit (69.7) on raw prompts.
  • Adding the VLM agent boosts the score to 87.9. The gains are most significant in IP tasks (replacement and addition), where the agent successfully retrieves missing reference images.
  • Ablation studies show SFT contributes the majority of the gain (85.0), while DPO adds further refinement (87.9) by correcting boundary errors.
• Performance on Existing Benchmarks (EditVerse-Bench, OpenVE-Bench):
  • On benchmarks with well-specified inputs, Aurora achieves state-of-the-art results among open-source methods (7.61 on EditVerse-Bench, 3.38 on OpenVE-Bench), closing most of the gap with the closed-source Runway Aleph baseline.
  • Even when tools are disabled (only instruction rewriting occurs), the agent improves performance, indicating the value of instruction refinement.
• Transferability:
  • When the Aurora VLM agent is paired with frozen versions of UniVideo and Kiwi-Edit, it consistently improves their scores on both AgentEdit-Bench and existing benchmarks, proving the agent's utility is model-agnostic.

5. Significance and Limitations

Significance:
The paper argues that progress in video editing depends not only on stronger video DiTs but also on the reliable translation of raw user requests into model-ready conditions. Aurora demonstrates that an agentic layer can effectively resolve underspecification, enabling unified models to handle real-world, incomplete user inputs without requiring the video model itself to be retrained for every new tool or condition.

Limitations:

• Training Paradigm: The VLM agent is aligned via offline DPO. The authors note that online RL (e.g., GRPO) over the full pipeline, using the final edited clip as a reward signal, is currently impractical due to compute constraints.
• Motion Complexity: The 5B video DiT handles removal, stylization, and insertion of objects with limited motion reliably. However, adding new subjects with large, physically plausible motion remains difficult, attributed to both the backbone's capacity and the scarcity of such cases in the training data.

Broader Impact:
The authors acknowledge risks regarding non-consensual identity manipulation and copyright issues arising from the web image search tool, positioning Aurora as a creative tool rather than an evidence-producing one.