⚡ electrical engineering

Conditional Flow Matching for Visually-Guided Acoustic Highlighting

This paper introduces a Conditional Flow Matching framework with a novel rollout loss and cross-modal conditioning module to address the ambiguity in visually-guided acoustic highlighting, demonstrating that generative modeling outperforms existing discriminative approaches in rebalancing audio to align with visual focus.

Original authors: Hugo Malard, Gael Le Lan, Daniel Wong, David Lou Alon, Yi-Chiao Wu, Sanjeel Parekh

Published 2026-06-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hugo Malard, Gael Le Lan, Daniel Wong, David Lou Alon, Yi-Chiao Wu, Sanjeel Parekh

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a movie scene where a character is giving an important speech. You can see their lips moving and their face is the center of attention, but the audio is a mess: the background music is drowning out their voice, and the sound of a passing car is louder than the dialogue. This is the problem this paper tries to solve. It's called Visually-Guided Acoustic Highlighting. The goal is to automatically "remix" the audio so that what you hear matches what you see.

Here is a simple breakdown of how the authors solved this, using some everyday analogies.

The Old Way: The "One-Size-Fits-All" Translator

Previously, computers tried to fix this audio using a method called a discriminative model. Think of this like a strict translator who believes there is only one correct way to translate a sentence from one language to another.

The problem is that audio remixing isn't like that. If a video shows a person speaking, there isn't just one perfect volume level for their voice versus the background noise. There are many "good" versions. The old computer models got confused because they were trying to find a single, perfect answer in a situation where many answers are possible. They often made mistakes, like turning the music down too much or not boosting the voice enough.

The New Way: The "Flowing River" (Flow Matching)

The authors decided to stop looking for a single answer and instead treat the problem like generative modeling. They used a technique called Conditional Flow Matching.

Imagine the bad audio (the messy mix) is a boat stuck in a swamp, and the good audio (the clear mix) is a boat in a calm, open ocean.

The Goal: The computer needs to learn a path to guide the boat from the swamp to the ocean.
The Method: Instead of jumping straight to the destination, the computer learns a "current" (a vector field) that gently pushes the audio step-by-step from bad to good. It's like a river flowing from a muddy source to a clear lake.

The Two Big Problems They Solved

Even with this "river" idea, the computer faced two major hurdles, which the authors fixed with clever tricks.

1. The "Wrong Turn" Problem (Rollout Loss)

The Issue: In a step-by-step journey, if you make a tiny mistake at the very first step (like turning the boat slightly left instead of right), that error gets bigger and bigger with every step. By the time you reach the end, you might be miles off course. In audio terms, if the computer guesses wrong about which sound to boost in the first second, the final result sounds terrible.

The Fix: They introduced a Rollout Loss.

The Analogy: Imagine a coach training a runner. In the old method, the coach only checked the runner's form at the finish line. In this new method, the coach makes the runner run the entire race during practice, but then forces them to run it again from the start, correcting their own mistakes along the way.
How it works: The computer simulates the whole journey from bad audio to good audio. Then, it looks at the final result and says, "Wait, that's not quite right." It uses that feedback to adjust its path for the entire journey, not just the last step. This stops the small errors from piling up and keeps the audio on the right track.

2. The "Blindfolded" Problem (Conditioning Module)

The Issue: To fix the audio, the computer needs to know what to fix. It looks at the video to decide. However, in the old systems, the "vision" part of the computer and the "audio" part were separate. The vision part would just say, "I see a person talking," and hand that note to the audio part. The audio part then had to figure out, "Okay, does 'person talking' mean I should boost the voice or the music?" It was a bit like a blindfolded chef trying to guess the ingredients of a dish just by reading a menu description.

The Fix: They built a Cross-Modal Adapter.

The Analogy: Instead of giving the chef a menu, they let the chef taste a tiny bit of the dish while looking at the menu.
How it works: They connected the "vision" brain directly to the "audio" brain early on. Now, when the computer looks at the video, it simultaneously "hears" the audio context. This allows the vision part to say, "I see a person talking, and I hear a voice, so I know exactly which sound to boost." This makes the decision much sharper and more accurate.

The Results

The authors tested their new system (which they call VisAH-FM) against the old methods.

The Score: It won significantly more often in tests where humans judged which audio sounded better.
The Feeling: The new system creates audio that feels much more natural and aligned with the video. It doesn't just make things louder; it understands the scene.
The Control: Because the system works in steps, users can actually control how much highlighting they want. It's like a volume knob that lets you decide how much to boost the speech, stopping at any point in the process.

Summary

In short, the paper says: "Don't try to force audio to have one perfect answer. Instead, teach the computer to flow from bad audio to good audio like a river. But to keep the river from getting lost, make the computer practice the whole trip to correct its own mistakes, and let its eyes and ears talk to each other right from the start."

The result is a smarter, more reliable way to fix messy video audio automatically.

Technical Summary: Conditional Flow Matching for Visually-Guided Acoustic Highlighting

Problem Definition
Visually-guided acoustic highlighting (VisAH) aims to rebalance audio tracks in alignment with accompanying video content to create a coherent audio-visual experience. In raw daily recordings, audio is often poorly balanced due to recording limitations, where background noise or music overwhelms primary sources like speech. The goal is to automatically transform these poorly balanced mixes into well-balanced ones by leveraging visual cues to determine which audio sources (speech, music, sound effects) should be enhanced or suppressed.

Previous approaches, such as the discriminative VisAH model, treated this as a one-to-one mapping problem using sound source separation architectures. However, the authors argue that audio remixing is inherently a many-to-many problem: a single video scene can admit multiple plausible "well-balanced" mixes, and conversely, multiple poorly balanced mixes can correspond to the same target. Discriminative models struggle with this ambiguity, often leading to misalignment between visual and auditory focus.

Methodology
The paper reframes VisAH as a generative task using Conditional Flow Matching (CFM). Instead of learning a direct point-wise mapping, the model learns a continuous vector field that transports samples from a distribution of poorly balanced audio ( $x_0$ ) to a distribution of well-balanced audio ( $x_1$ ), conditioned on visual cues ( $c$ ).

The proposed framework, VisAH-FM, introduces two critical innovations to address specific challenges in iterative flow-based generation:

Rollout Loss for Trajectory Stabilization:
In iterative flow matching, small prediction errors in early steps (specifically in selecting which sources to enhance) can compound, causing the generation trajectory to drift off the data manifold. To mitigate this, the authors introduce a rollout loss. Unlike standard CFM where steps are trained independently, this loss performs a full rollout of the flow operator during training (using $T=4$ steps) and penalizes the mean-squared error between the final predicted output and the ground-truth target. This encourages the model to self-correct intermediate errors and ensures the trajectory converges to the correct endpoint, effectively mitigating exposure bias.
Audio-Aware Cross-Modal Conditioning:
Prior work conditioned the generation model solely on visual features (e.g., CLIP embeddings), forcing the denoising network to implicitly learn the correspondence between visual scenes and audio sources. The authors propose an adapter module that fuses audio and visual cues before the vector field regression stage. Specifically, they inject features from a CLAP audio encoder into the intermediate layers of a CLIP vision encoder via a lightweight cross-attention mechanism. This allows the conditioning pathway to perform explicit cross-modal source selection, providing the main model with semantically aligned features that focus purely on accurate audio regression.

Key Contributions

Generative Reformulation: The authors demonstrate that formulating VisAH as a generative problem (via Flow Matching) is superior to the discriminative paradigm, better capturing the inherent variability and many-to-many nature of audio remixing.
Rollout Loss: A novel loss function that supervises the entire generated trajectory, stabilizing long-range flow integration and preventing error accumulation in iterative steps.
Improved Conditioning: An architecture that integrates audio features directly into the visual encoder, enabling early cross-modal filtering and improving the model's ability to identify target sources.

Experimental Results
The method was evaluated on the Muddy Mix Dataset. Quantitative results show that VisAH-FM consistently outperforms the state-of-the-art discriminative VisAH baseline and its variants across multiple metrics:

Semantic Alignment: Improved ImageBind (IB) scores and lower KL divergence (KLD) between predicted and ground-truth audio distributions.
Remixing Accuracy: Significant reduction in Loudness Difference (LDif) for speech, music, and sound effects.
Signal Quality: Better performance in magnitude (Mag) and envelope (Env) distances.

Ablation studies confirm that both the rollout loss and the audio-aware conditioning module are essential. Removing the rollout loss leads to a marked performance drop due to trajectory drift. Furthermore, integrating audio features into the conditioning module yields greater gains than using text captions, as audio features directly encode the auditory scene necessary for source selection.

Subjective tests involving human participants also favored VisAH-FM, with a win rate of ~45.5% compared to ~17.2% for the baseline, indicating better alignment with visual scenes in terms of loudness balance and overall quality.

Significance
The paper establishes that visually-guided audio remixing is best addressed through generative modeling rather than discriminative mapping. By introducing mechanisms to stabilize iterative generation (rollout loss) and explicitly aligning modalities during conditioning, the proposed approach achieves state-of-the-art results. The work highlights that allowing models to iteratively refine predictions and self-correct errors is crucial for tasks where the input-output relationship is ambiguous and non-deterministic.