Step-by-Step Video-to-Audio Synthesis via Negative Audio Guidance

Technical Summary: Step-by-Step Video-to-Audio Synthesis via Negative Audio Guidance

Problem Statement
Current video-to-audio (V2A) generation models typically produce a single, complete audio track in one pass. While these models achieve high semantic and temporal alignment with input videos, they lack mechanisms for incremental refinement. In traditional Foley workflows, artists layer complementary sounds (e.g., footsteps, fabric rustling) to build a composite audio track. Existing V2A models cannot replicate this interactive process; if a generated track misses specific events, the entire track must be regenerated. Furthermore, achieving step-by-step generation via standard conditional learning would require costly multi-reference video-audio datasets (where a single video is paired with multiple distinct, non-overlapping audio tracks), which are difficult to obtain at scale.

Methodology
The authors propose a Step-by-Step Video-to-Audio (V2A) synthesis framework driven by Negative Audio Guidance (NAG). This approach enables the iterative generation of complementary audio tracks that fill missing sound events without duplicating sounds already present in previously generated tracks.

1. Concept Negation Formulation:
   The core challenge is generating a target audio $x^{(2)}$ conditioned on a video $V$, a text prompt $C_2$, and a previously generated audio track $x^{(1)}$, such that $x^{(2)}$ aligns with $V$ and $C_2$ but excludes concepts present in $x^{(1)}$. The authors formulate this as a concept negation problem, approximating the target distribution $p(x^{(2)} | V, C_2, x^{(1)})$ by treating $x^{(1)}$ as a negative condition ($\neg x^{(1)}$).

2. Guided Flow Derivation:
   Building on flow-matching models and classifier-free guidance, the authors derive a new guidance formulation. They decompose the target probability distribution to include a term that repels the generation from the conditioning audio $x^{(1)}$. The resulting guided flow $\tilde{u}_{\theta, \psi}(x_t)$ consists of:

   • An unconditional flow term.
   • A standard guidance term strengthening fidelity to the video $V$ and text $C_2$.
   • A Negative Audio Guidance (NAG) term: $\beta (u_\theta(x_t, t, V, C_2) - u_{\theta, \psi}(x_t, t, V, \emptyset, x^{(1)}))$. This term pushes the sampling trajectory away from the acoustic content of $x^{(1)}$.

3. Training Strategy (No Specialized Data):
   To avoid the need for multi-reference datasets, the authors train the audio-conditioned branch (parameterized by $\psi$) using non-overlapping audio segments from the same video.

   • Architecture: They utilize a ControlNet-style architecture adapted for the MMAudio (a pre-trained flow-matching V2A model). The audio conditioning is injected via single-modal transformer blocks that extract features from the reference audio $x^{(1)}$ and add them to the main branch.
   • Data Construction: From a standard single-reference dataset (e.g., VGGSound), they sample two non-overlapping 4-second segments: one as the target ($x_{tgt}$) and one as the condition ($x_{cond}$). This forces the model to learn shared acoustic context (environment, timbre, recording conditions) rather than copying the waveform, enabling it to act as a "repulsive prior" during inference.

4. Inference Process:
   The generation proceeds iteratively. For the $k$-th step, the composite audio of all previously generated tracks ($\sum_{l=1}^{k-1} x^{(l)}$) is used as the negative conditioning signal $x^{(1)}$ to generate the next complementary track $x^{(k)}$.

Key Contributions

• Step-by-Step Synthesis Formulation: The paper formulates V2A generation as an iterative process of creating multiple complementary tracks for a single video, enabling Foley-style interactive authoring.
• Negative Audio Guidance (NAG): A novel guidance mechanism that uses an audio-conditioned branch negatively during sampling to prevent duplication of sounds across sequential tracks. Crucially, this is trainable using standard single-reference audiovisual datasets via non-overlapping segment sampling.
• Improved Separability and Quality: The method demonstrates that it is possible to generate distinct, high-quality audio tracks that do not overlap in content, outperforming baselines that rely on independent generation or text-based negative prompting.

Results
The authors evaluated their method on a newly constructed dataset, Multi-Caps VGGSound, which contains videos paired with multiple distinct captions describing different sound events.

• Objective Metrics:
  • Composite Audio: The proposed method achieved the best scores in Fréchet Distance (FD), KL divergence, and temporal alignment (DeSync) compared to baselines like Seeing-and-Hearing, FoleyCrafter, and standard MMAudio. It outperformed "Independent Generation" (generating tracks without context) and "Step-by-Step with Negative Prompting" (using text to exclude previous sounds).
  • Individual Tracks: The method significantly improved audio separability (measured by CLAP audio-audio similarity) compared to the vanilla MMAudio. While negative prompting improved separability, it degraded audio quality and text alignment. NAG maintained high audio quality and video-text alignment while achieving superior separability.
• Subjective Evaluation:
  • User studies showed the proposed method was preferred for audio quality, semantic alignment, and temporal alignment in the final composite audio.
  • For individual tracks, the method received significantly higher ratings for separability and marginally higher ratings for audio quality and text fidelity.
• Scalability: Analysis varying the number of generation steps (2 to 5 captions) showed that the benefits of NAG (specifically in separability) become more pronounced as the number of required audio events increases, without degrading overall fidelity.

Significance and Claims
The paper claims that this work resolves a critical bottleneck in automated Foley: the inability to interactively refine audio tracks without regenerating the entire sequence. By introducing NAG, the authors enable a workflow where users can author multiple sound events step-by-step, similar to traditional Foley artists, but without the labor-intensive requirement of multi-reference training data. The method bridges the gap between high-quality generative models and practical, controllable sound design, ensuring that generated sounds are not only semantically aligned with the video but also distinct from one another in a composite mix. The authors emphasize that their approach is practical because it leverages existing pre-trained models and standard datasets, avoiding the need for specialized, hard-to-acquire multi-track audio-video pairs.