Technical Summary: AVE-Compass and AVE-Agent

1. Problem Statement

While instruction-based video editing has advanced rapidly, existing systems and benchmarks primarily focus on visual transformations of silent clips or isolated audio editing. Real-world videos, however, consist of tightly coupled audio and visual signals where semantics emerge from the interplay of visual dynamics, foreground sounds, background ambience, and speech. Current models struggle to execute complex, cross-modal instructions that require synchronized modifications (e.g., changing sound when an action changes, or removing acoustic traces when a visual target is removed) while preserving non-target content.

Existing benchmarks are insufficient for evaluating these capabilities because they:

• Focus on visual transformations of silent videos or isolated audio generation.
• Lack evaluation of cross-modal consistency and implicit synchronization constraints.
• Rely on coarse-grained metrics that fail to diagnose specific failures in audio-visual editing, such as the distortion of background sounds or the loss of lip-sync.

2. Methodology

A. AVE-Compass: A Comprehensive Benchmark

The authors introduce AVE-Compass, a benchmark designed to evaluate free-form audio-visual editing.

• Dataset Composition: It comprises 145 curated source videos (spanning diverse domains, shot counts, and styles) and 196 audio-visually coupled editing instructions.
• Instruction Types: Instructions cover four branches: joint audio-visual, speech, video-only, and audio-only editing. They are categorized into 28 fine-grained operation types and annotated with difficulty labels regarding object localization, audio complexity, and cross-modal linkage.
• Evaluation Protocol: The benchmark decouples performance into four dimensions:
  1. Instruction Following (IF): Assesses if the requested modification is correctly applied to the intended target using atomic Yes/No checklist questions.
  2. Fidelity Preserving (FP): Evaluates whether non-edited content (visual and auditory) remains consistent with the source and whether unintended content is introduced.
  3. Realism (REAL): Uses a dedicated MLLM rubric to judge the perceptual plausibility and physical consistency of the edited clip.
  4. Editing Intent (EI): A primary metric defined as the product of IF and FP, rewarding complete execution of the edit while preserving non-target content.
• Metrics: Evaluation combines MLLM-as-Judge (using 2,688 fine-grained checklist items) with automated metrics for cross-modal synchronization (Lip Sync, AV Sync), video quality (Aesthetic, Subject Consistency, Motion Smoothness), and audio quality (Aesthetic, Speech Quality).

B. AVE-Agent: A Modular Editing Framework

To address the limitations of current models, the authors propose AVE-Agent, a planning-based agent framework.

• Planner Agent: Converts abstract user instructions into an executable, dependency-aware subtask DAG (Directed Acyclic Graph). It analyzes the input clip, identifies explicit requirements and implicit cross-modal consequences, and decomposes the task into modality-specific subtasks with actionable evaluation criteria.
• Executor Agent: Grounds subtasks in concrete operations (video, audio, or speech) using a self-check reflection loop. It compiles high-level intents into tool-specific prompts, executes the task, and evaluates the output against planner criteria. If the output fails, an "improver" rewrites the guidance to target specific errors rather than blindly retrying.
• Mixed Evaluator Agent: Inspects the assembled clip for global coherence, instruction following, and fidelity. It determines the least costly corrective action (e.g., remixing, regenerating a subtask, or full replanning) to address detected issues without unnecessary recomputation.

3. Key Contributions

1. AVE-Compass Benchmark: A comprehensive dataset with 145 videos, 196 coupled instructions, and 2,688 checklist items, specifically targeting realistic scenarios requiring synchronized cross-modal editing and strict preservation of non-target content.
2. Decoupled Evaluation Protocol: A framework covering Instruction Following, Fidelity Preserving, Realism, and Editing Intent, enabling the diagnosis of incomplete edits, content drift, low realism, and misalignment.
3. AVE-Agent Framework: A modular, planning-based agent that decomposes instructions into subtasks and utilizes self-reflection to improve editing quality and alignment in complex joint editing scenarios.

4. Experimental Results

The authors evaluated six systems (including Wan2.7, HappyHorse, Gemini-Omni, Seedance, LTX2, and AVE-Agent) on AVE-Compass.

• Performance of AVE-Agent: AVE-Agent achieved the highest scores in Editing Intent (59.8) and Instruction Following (77.3), significantly outperforming baselines. It showed particular gains in audio-side instruction following and audio-visual synchronization.
• Trade-offs in Baselines: Current models struggle to balance Instruction Following and Fidelity Preserving.
  • Models like LTX2 often follow instructions by regenerating the entire video, resulting in poor Fidelity Preserving.
  • Models like Gemini-Omni and Seedance sometimes return the original video/audio to achieve high preservation scores, failing to execute the instruction (low Instruction Following).
• Realism Gap: While some models achieved high automated quality scores (e.g., Video/Audio Aesthetic), their Realism scores were notably lower, indicating a struggle with physical-world understanding and logical validity in edited results.
• Robustness: AVE-Agent maintained more stable performance across varying source video durations, object localization hardness, audio complexity, and cross-modal linkage degrees compared to baselines.
• Metric Validity: The study confirmed that automated metrics and subjective MLLM judgments are largely orthogonal, capturing distinct aspects of quality. Human-LLM agreement on the evaluation metrics averaged close to 90%.

5. Significance

The paper claims that AVE-Compass and AVE-Agent represent a step forward in moving the field from isolated pixel-level modifications toward physically and perceptually coherent multimodal editing. By systematically evaluating the preservation of non-target content alongside instruction execution, the benchmark reveals that state-of-the-art models still lack explicit mechanisms for planning cross-modal operations and enforcing synchronization constraints. The proposed agent framework demonstrates that decomposing complex instructions into dependent subtasks with iterative self-reflection can effectively improve editing quality and alignment in joint audio-visual editing tasks.