Technical Summary: InstructAV2AV

Problem Statement

Recent advances in diffusion-based video editing have enabled complex visual manipulations guided by text instructions. However, these methods typically operate solely on the visual modality, leaving the accompanying audio unchanged. In real-world media, visual content is tightly synchronized with audio; consequently, when editing alters a speaker's identity, an action, or an event source, the original audio becomes temporally or semantically disjointed from the edited visual result. Existing audio-video editing methods struggle with fine-grained controllability, often relying on global scene transformations, tedious manual spatial masks, or dense auxiliary conditions (e.g., Canny edges). There is a critical need for an end-to-end framework capable of open-world, instruction-guided audio-video joint editing that modifies specific targets while preserving non-target contexts and maintaining strict audio-visual synchronization.

Methodology

1. Scalable Data Synthesis and InsAVE-80K

To address the scarcity of high-quality source-to-target paired data for audio-video editing, the authors propose a scalable data synthesis pipeline to construct InsAVE-80K, the first large-scale dataset for this task.

• Pipeline: The process involves three stages: (1) Data Source Collection from open-world platforms (e.g., YouTube) with rigorous multi-stage filtering for video motion, quality, and audio clarity; (2) Data Editing Engine, which uses a mask-guided model (based on Wan2.2-5B) to automatically synthesize edited targets and corresponding audio tracks while preserving non-target elements; and (3) Verification, employing a multimodal LLM and human verification across five criteria (instruction fidelity, content preservation, perceptual quality, synchronization, and safety).
• Dataset: InsAVE-80K comprises 79,000 training pairs and 1,000 human-curated evaluation pairs, featuring 5-second clips at 720p/24FPS with 16kHz audio.

2. InstructAV2AV Framework

The core framework is an end-to-end instruction-guided audio-video joint editing model built upon a pre-trained audio-video generation backbone (Ovi).

• Dual-Stream Editing Backbone: The model adopts a symmetric dual-stream diffusion transformer architecture. Source audio and video are encoded into latent codes and concatenated with Gaussian noise along the channel dimension to anchor the source context.
• Source-Instruction Gated Attention (SIGA): To handle the lack of explicit spatial masks and balance content preservation with instruction following, the authors introduce SIGA. This module allows the noisy latent features to concurrently attend to both source features (for preservation) and instruction embeddings (for modification). A learnable soft gate adaptively fuses these two streams, shifting focus based on the token's requirement.
• Two-Stage Training Strategy (TSTS): To effectively transfer generative priors to the editing domain and ensure smooth convergence, the authors employ a two-stage strategy:
  1. Modality-specific training: The audio and video branches are trained independently to learn source preservation and instruction following without cross-modal interference.
  2. Joint training: The full architecture is fine-tuned with bidirectional cross-attention enabled to learn fine-grained visual-acoustic correlations and strict synchronization.

Key Contributions

1. InstructAV2AV: The first end-to-end framework for open-world, instruction-guided audio-video joint editing that operates solely on text instructions without requiring manual masks.
2. InsAVE-80K: The construction of the first large-scale audio-video editing dataset with source-to-target pairs and rigorous quality filtering.
3. Architectural Innovations: The proposal of the SIGA module for adaptive content preservation and instruction following, and the Two-Stage Training Strategy for robust model convergence.

Results

Extensive experiments were conducted on the InsAVE-80K evaluation set and the public AvED-Bench.

• Quantitative Performance: InstructAV2AV outperforms state-of-the-art methods (AvED, AVI-Edit, CoherentAVEdit) across 11 metrics spanning video quality (FVD, TV-A, TC, SSIM), audio fidelity (FAD, TA-A, LPAPS), and joint audio-visual alignment (AV-A, PEAVS, Sync-C/D). Notably, it achieves superior performance even when compared against baselines that were granted ground-truth instance masks during evaluation.
• Qualitative Performance: Visual comparisons demonstrate that InstructAV2AV successfully avoids the frame-wise flickering, audio artifacts, and semantic misalignment observed in competing methods. It accurately modifies target content (e.g., changing a speaker's identity and voice, inserting/removing objects with sound) while faithfully preserving non-target backgrounds and ambient sounds.
• User Preference: In a user study, InstructAV2AV received the highest preference scores (approx. 46-49%) across Audio-Video Synchronization, Text Alignment, and Overall Preference dimensions.
• Ablation Studies: Removing source concatenation, the SIGA module, or the two-stage training strategy resulted in significant degradation in background preservation, instruction following, and visual consistency, respectively.

Significance and Claims

The paper claims that InstructAV2AV represents a significant step forward in controllable content creation by addressing the inherent multimodal nature of real-world media. By enabling fine-grained attribute disentanglement (e.g., preserving identity while changing speech, or preserving speech while changing appearance) and operating without manual spatial constraints, the framework facilitates flexible and realistic audio-video editing. The authors acknowledge that the framework inherits limitations from its underlying generative backbone (e.g., challenges with complex physical interactions or lighting consistency in multi-source environments) but posit that as foundational models improve, InstructAV2AV will seamlessly integrate these advancements to further enhance editing quality.