Audio-sync Video Instance Editing with Granularity-Aware Mask Refiner

The paper presents AVI-Edit, a novel framework that achieves precise audio-sync video instance editing by employing a granularity-aware mask refiner for spatial accuracy, a self-feedback audio agent for temporal control, and a newly constructed large-scale dataset.

The Gist

Imagine you have a home movie of a birthday party. In the video, your friend is telling a funny joke, but you want to change the joke to something else, or maybe you want to swap your friend's outfit while keeping their voice exactly the same. Or perhaps you want to replace a barking dog with a meowing cat, but you need the "meow" to happen at the exact right moment.

Doing this manually is a nightmare. You'd have to edit the video, then record new audio, then try to sync them up perfectly. If you miss by a fraction of a second, it looks and sounds fake.

AVI-Edit is a new AI tool that acts like a "magic editor" for these specific tasks. It doesn't just edit the picture; it edits the picture and the sound together, keeping them perfectly in sync, like a professional film editor who never makes a mistake.

Here is how it works, broken down into three simple "superpowers":

1. The "Smart Magnifying Glass" (Granularity-Aware Mask Refiner)

The Problem: When you tell a computer "change this person's shirt," you usually just draw a rough box around them. That box is too big; it includes the background, the wall behind them, and maybe the person next to them. If the computer edits everything inside that box, the background gets ruined.

The Solution: Think of the Mask Refiner as a super-smart assistant who looks at your rough box and says, "Wait, you only want the shirt, not the wall."

• It takes your "rough sketch" and uses a special "precision dial" to zoom in and trace the exact outline of the person or object.
• It does this iteratively, getting more and more precise with every step, ensuring that only the specific thing you want to change gets touched, leaving the rest of the world untouched.

2. The "Audio Conductor" (Self-Feedback Audio Agent)

The Problem: In real life, when a person speaks, their mouth moves in perfect time with the sound. When a car drives by, the engine noise matches the speed of the car. If you just swap the audio track, the lips won't move right, or the sound will feel disconnected from the action.

The Solution: The Audio Agent is like a conductor in an orchestra who listens to the video and directs the music.

• Separation: It first listens to the original video and says, "Okay, I need to keep the applause in the background, but I need to remove the person's voice."
• Generation: It then writes a new script for the new sound (e.g., "A man says 'Hello'") and generates that audio.
• Remixing & Checking: It mixes the old background sounds with the new voice. But here's the cool part: it has a "quality control" brain (an AI judge) that listens to the mix. If it sounds weird or out of sync, the agent says, "No, that doesn't sound right," and rewrites the instructions to try again. It keeps looping until the audio is perfect and perfectly synced with the video.

3. The "Time-Traveling Editor" (The Backbone)

The Problem: Most video editors are great at changing one frame but terrible at keeping the movement smooth from one second to the next.

The Solution: AVI-Edit is built on a massive, pre-trained "brain" (a video generation model) that already understands how the world moves. It treats the video and the audio as a single package. When it edits the video, it simultaneously adjusts the audio to ensure that if a car speeds up, the engine noise speeds up too. It's like editing a movie where the picture and the sound are glued together, so you can't accidentally pull them apart.

What Can You Do With It?

The paper shows four cool examples of what this tool can do:

1. Change the Script: A woman is speaking in a video. You can change what she says, and the AI will rewrite her mouth movements and voice to match the new words perfectly.
2. Change the Look: A man is walking down the street. You can change his coat from blue to red, but keep his original voice and the background noise exactly the same.
3. Swap the Character: A dog is barking. You can turn the dog into a cat, and the AI will automatically change the "bark" into a "meow" at the exact right moment.
4. Control with Sound: You can just type "make the water flow faster," and the AI will speed up the water in the video and make the splashing sound louder and faster, all without you touching a single video frame.

In a Nutshell

Think of AVI-Edit as a magical pair of scissors and a recording studio combined into one. It lets you cut out a specific part of a video, change it however you want, and seamlessly paste it back in with a brand-new sound that fits perfectly, all while keeping the rest of the movie looking and sounding exactly as it did before. It solves the biggest headache of video editing: making sure the picture and the sound stay best friends.

Technical Summary

1. Problem Statement

While recent video generation models (e.g., Sora, Veo) have achieved high-quality audio-visual synchronization, existing video editing methods face significant limitations:

• Lack of Audio-Visual Synchronization: Most editing models focus solely on visual features, breaking the natural synchronization between audio and video, which is crucial for immersive content.
• Coarse Spatial Control: Existing methods often struggle with precise instance-level editing. They typically rely on coarse user inputs (like bounding boxes) and fail to refine these into precise instance contours.
• Lack of Temporal Control: Current approaches lack fine-grained temporal control, making it difficult to specify the exact timing of events or synchronize edits with specific audio cues.
• Data Scarcity: There is a lack of large-scale datasets specifically designed for instance-centric audio-sync video editing with comprehensive annotations.

2. Methodology

The authors propose AVI-Edit, a framework designed to edit specific instances in a video while preserving audio-visual synchronization. The framework consists of three core components:

A. Audio-Sync Video Backbone

• Base Architecture: Built upon the pretrained Wan2.2-5B video diffusion transformer to leverage strong generative priors for temporal consistency.
• Input Conditioning: The model accepts a text description ($y$), an instance mask ($m$), and curated audio tokens ($a$).
• Mechanism:
  • Background Preservation: Uses a downsampled instance mask to interpolate the noisy latent code with the clean input video, ensuring non-target regions remain unchanged.
  • Frame-wise Cross-Attention: Introduces a specific cross-attention layer within the Diffusion Transformer (DiT) blocks to process audio tokens. This allows the model to align video generation with audio cues frame-by-frame.
  • Unified Interface: Supports optional control contexts (scribbles, poses, reference images) via element-wise addition or concatenation.

B. Granularity-Aware Mask Refiner (GAMR)

• Problem: User-provided masks are often coarse (e.g., bounding boxes).
• Solution: A dedicated module with an architecture similar to the video backbone that iteratively refines coarse masks into precise instance contours.
• Key Innovation:
  • Precision Factor ($p$): A parameter indicating the degree of mask degradation (from a bounding box to a precise contour). This factor is linearly encoded and injected into the GAMR via AdaLN and gating mechanisms.
  • Visual Semantics: Unlike standard text-to-mask models, GAMR replaces text tokens with video tokens in cross-attention layers, allowing it to reason about instance boundaries based on visual semantics.
  • Audio Guidance: Incorporates audio tokens via frame-wise cross-attention to align mask refinement with event timing.
  • Iterative Process: During inference, the mask is refined step-by-step. The precision factor follows a "degradation schedule" (specifically, an Instant Degradation strategy where the factor drops immediately after the first step) to ensure rapid convergence to the target region followed by precise boundary refinement.

C. Self-Feedback Audio Agent

• Goal: To curate high-quality, synchronized audio guidance for the video generation process.
• Pipeline: A Separate–Generate–Remix–Rework cycle:
  1. Separation: Uses a VLM to analyze the scene and generate prompts to separate the original audio into "to-keep" (e.g., background applause) and "to-remove" (e.g., target speech) components.
  2. Generation: Generates new audio for the target instance based on the text description using specialized models (TTS, TTM, TTSound).
  3. Remix: Combines the separated and generated audio.
  4. Rework (Self-Feedback): A Multimodal Large Language Model (MLLM) acts as a judge, evaluating the remixed audio on five dimensions (separation accuracy, generation accuracy, acoustic harmony, instruction adherence, and fidelity). If the score is below a threshold, the MLLM generates improvement prompts to refine the separation/generation steps iteratively.

3. Key Contributions

1. AVI-Edit Framework: The first framework to achieve fine-grained spatial and temporal control for audio-sync video instance editing, handling both human speech and non-speech events.
2. Granularity-Aware Mask Refiner (GAMR): A novel module that uses a precision factor and visual semantics to transform coarse user inputs into precise instance masks, solving the "coarse-to-fine" editing problem.
3. Self-Feedback Audio Agent: A robust pipeline that uses an iterative MLLM feedback loop to ensure high-quality, realistic, and synchronized audio generation.
4. AVISET Dataset: A new large-scale dataset (71k training, 1k validation, 1k testing clips) featuring instance-centric correspondence, instance masks, and paired original-edited text instructions.

4. Experimental Results

The authors evaluated AVI-Edit on AVISET and the AvED-Bench, comparing it against state-of-the-art methods (AvED, Ovi, and a sequential VACE-Foley baseline).

• Quantitative Performance: AVI-Edit outperformed baselines across all metrics:
  • Visual Quality: Lower FVD (312.89 vs. 364.69 for AvED) and higher IS.
  • Consistency: Higher Frame Consistency (96.65%) and Text-Video/Audio-Video alignment scores.
  • Synchronization: Superior lip-sync metrics (Sync-C/Sync-D) and audio-video alignment.
• User Preference: In a user study, AVI-Edit received the highest preference scores (approx. 49% for Audio-Visual Synchronization and 45% for Overall Preference) compared to baselines.
• Ablation Studies:
  • Removing the Precision Factor led to imprecise mask estimation.
  • Removing the Mask Refiner caused background artifacts.
  • Removing the Audio Agent resulted in noisy audio and poor synchronization.
• Efficiency: AVI-Edit is significantly faster (311s) than AvED (1140s) and requires less VRAM than Ovi.

5. Significance and Applications

• Bridging the Gap: AVI-Edit successfully bridges the gap between high-fidelity video generation and precise instance-level editing, a task previously hindered by the lack of audio-visual synchronization.
• Versatile Applications: The framework supports diverse scenarios including:
  • Speech Modification: Changing what a specific person says while keeping their appearance.
  • Appearance Alteration: Changing a person's clothing or look while preserving their original speech.
  • Semantic Transformation: Changing an object (e.g., dog to cat) and its corresponding sound.
  • Dynamics Adjustment: Modifying physical dynamics (e.g., water flow) purely via audio cues.
  • Instance Insertion/Removal: Seamlessly adding or removing objects and their sounds.
• Future Impact: By providing a robust method for audio-sync editing, this work paves the way for more engaging and realistic content creation tools for film, advertising, and game development.