🤖 machine learning

SALSA-V: Shortcut-Augmented Long-form Synchronized Audio from Videos

SALSA-V is a multimodal video-to-audio generation model that leverages a masked diffusion objective and a novel shortcut loss to synthesize highly synchronized, high-fidelity long-form audio from silent videos in as few as eight steps, significantly outperforming existing state-of-the-art methods in both alignment and efficiency.

Original authors: Amir Dellali, Luca A. Lanzendörfer, Florian Grötschla, Roger Wattenhofer

Published 2026-08-17

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Amir Dellali, Luca A. Lanzendörfer, Florian Grötschla, Roger Wattenhofer

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where every silent movie you've ever seen suddenly comes to life with its own soundtrack, not just a generic piano score, but the specific thump of a falling box, the crunch of gravel under tires, or the rustle of leaves in a breeze. This is the dream of "video-to-audio" generation, a field of computer science where machines try to listen to what they see. For a long time, computers were terrible at this; they could guess the general vibe, but they missed the timing, making sounds that felt like they were arriving late to the party. The core challenge is twofold: the computer needs to understand what is happening (semantic understanding) and exactly when it is happening (temporal alignment). If a ball hits a wall, the sound must happen at that precise millisecond, or our brains get confused. Until now, making these sounds took a long time, like waiting for a slow oven to bake a cake, and the results often fell apart if you tried to make a long movie instead of a short clip.

Enter SALSA-V, a new digital wizard that aims to fix these problems. Think of it as a master Foley artist (a person who creates sound effects for movies) who has been given a superpower: the ability to work incredibly fast and never lose the beat. The researchers behind SALSA-V have built a model that can take a silent video and generate high-quality, perfectly synchronized audio in a flash. Unlike previous methods that struggled with long videos or required hours of waiting, SALSA-V can create minutes of sound in seconds, and it can even "listen" to a sample of audio you provide to mimic its style, like a musical chameleon.

Here is how this new wizard works and what it discovered:

The Magic of the "Shortcut"
Most AI models that create sound work like a sculptor chipping away at a block of stone, step by step, removing noise until the sound appears. Usually, this takes dozens of steps, which is slow. SALSA-V, however, learned a "shortcut." Imagine trying to walk from your house to the park. A normal model takes small, cautious steps, checking the ground every inch. SALSA-V learned to look ahead and take giant, confident strides. By training the model to understand that one big step is the same as two small steps combined, the researchers taught it to skip the boring parts. The result? SALSA-V can generate high-quality audio in as few as eight sampling steps. That's fast enough to feel near real-time, turning a process that used to take minutes into something that happens almost instantly.

The Masked Puzzle
To handle long videos without the audio turning into a muddy mess, SALSA-V uses a clever trick called "masked flow matching." Imagine you are filling in a crossword puzzle, but instead of starting from scratch, you are given a few words already filled in, and you have to guess the rest. SALSA-V does this with sound. During its training, the model is shown a video and a sound clip, but a random chunk of the sound is hidden (masked). The model has to figure out what the missing sound should be based on the video and the parts of the sound it can still hear. This teaches the model to be flexible. It means you can give the model a short video and a snippet of audio, and it can "outpaint" the sound, extending it seamlessly to match a longer video, or fill in gaps to create a continuous soundscape.

The Rhythm Section
The most critical part of video-to-audio is timing. If a dog barks in the video, the bark must happen exactly when the dog's mouth opens. Previous models often got the timing slightly off, which feels unnatural to human ears. SALSA-V solved this by training a special "synchronization coach." This coach is a separate AI that learns to recognize the exact moment an event happens in a video and matches it to the sound. The researchers found that to make this coach really good, they needed to be careful about how they trained it; using too many examples at once actually made it worse at spotting the tiny differences between similar sounds. By tuning this carefully, SALSA-V achieved a level of sync that humans rated as superior to other top models. In a listening test, people preferred SALSA-V's timing and overall quality over other state-of-the-art models.

The Long-Form Challenge
Many AI models are great at short clips (around 10 seconds) but fall apart when asked to make a 30-second or longer video. They either run out of memory or the sound starts to drift. SALSA-V tackles this by using a "progressive" approach. Instead of trying to guess the sound for a whole minute at once, it generates the audio in small chunks, using the end of the previous chunk as a guide for the next one. This allows it to keep the rhythm and style consistent for much longer durations. When tested on 30-second videos, SALSA-V maintained its high quality and synchronization, while other models started to lose their way.

What It Can't Do (Yet)
The authors are honest about the limits. Because the model builds sound by extending the previous chunk, if a video has a sudden cut to a completely different scene (like jumping from a quiet library to a loud construction site), the model might try to carry over the quiet library sounds into the construction site unless the user intervenes. It works best on continuous scenes without abrupt changes.

The Bottom Line
SALSA-V suggests that we can have our cake and eat it too: high-fidelity, perfectly synchronized audio that is generated in seconds rather than minutes. It doesn't just make sounds; it makes them feel right, matching the visual rhythm with a precision that previous models struggled to achieve. By combining a "shortcut" training method with a smart way of handling long sequences, this model paves the way for near-real-time sound design, potentially changing how we create content for films, games, and even silent archival footage. While it's not a perfect solution for every possible video scenario, it represents a significant leap forward in making machines that can truly "hear" what they see.

Technical Summary: SALSA-V

Problem Statement

Video-to-audio (V2A) generation, or "computational Foley," aims to synthesize diegetic sounds (e.g., rain, footsteps, impacts) that are semantically and temporally aligned with silent video content. While recent advances in diffusion and flow-matching models have improved audio quality, existing state-of-the-art methods face three critical limitations:

Synchronization and Controllability: Models often struggle with precise frame-level alignment and offer limited control over generated sounds, typically relying on text conditioning which lacks native editability and does not support reference audio matching.
Length Constraints: Most models are restricted to short snippets (often $\le$ 10 seconds) due to memory constraints or architectural limitations. Extending these models to longer sequences often results in performance degradation or stitching artifacts.
Inference Efficiency: High-quality generation in diffusion-based models typically requires dozens of sampling steps, making real-time or near-real-time applications impractical without dedicated fine-tuning or distillation.

Methodology

SALSA-V is a multimodal latent flow-matching model designed to address these issues through a unified architecture and novel training objectives.

Architecture

The model utilizes a latent flow-matching approach with an audio encoder based on a fine-tuned Stable-Audio VAE (increased to 43 Hz frame rate). The core generator employs a mixture of transformer blocks:

MMDiT-X Blocks: Modified blocks that operate jointly on concatenated audio and coarse semantic video sequences. These utilize joint attention and an optional additional self-attention layer applied solely to the audio stream for the first $n$ blocks.
DiT Blocks: Standard single-sequence DiT blocks process the latent representation.
Conditioning: The model integrates three conditioning sources:
1. Semantic Visual Features ( $v_s$ ): Extracted from a frozen SigLIP 2 encoder (8 FPS) to provide high-level semantic context.
2. Synchronization Features ( $v_{syn}$ ): High-resolution features (24 FPS) derived from a dedicated contrastive audio-visual alignment model trained on large-scale data (VideoPrism backbone). These are upsampled and used in sequence-level adaptive LayerNorm (adaLN) layers.
3. Global Text Features ( $t_c$ ): Derived from a text encoder.

Key Training Innovations

Contrastive Pre-training for Synchronization:
Unlike previous works that rely on smaller pre-trained models (e.g., Synchformer), SALSA-V trains a dedicated synchronization model using a large-scale VideoPrism backbone and an Audio Spectrogram Transformer (AST). The model is trained using a SigLIP loss on 0.667-second snippets. Crucially, the authors found that smaller batch sizes are optimal for learning precise temporal alignment, as larger batches dilute the "hard negatives" necessary for distinguishing fine-grained temporal differences.
Masked Flow Matching:
To enable audio conditioning and long-form generation, the authors introduce a masked training objective. During training, a contiguous subsequence (5 ms to 2.5 s) is randomly masked. The model receives the clean audio for the masked region but is not penalized for it; instead, it learns to predict the velocity for the noised (unmasked) tokens. This allows the model to perform "inpainting" (generating missing audio based on reference) and "outpainting" (extending audio), enabling seamless long-form generation via iterative extension.
Shortcut Loss for Few-Step Sampling:
To achieve high-fidelity generation in as few as eight steps without post-training distillation, SALSA-V adopts a "shortcut" formulation. The model is augmented with a desired step size $d$ as an input. The training objective enforces a self-consistency property: one large step must equal the sum of two smaller steps. By training a fraction of the batch (e.g., 25%) on this consistency target alongside the standard flow-matching loss, the model learns to anticipate the curvature of the denoising path, enabling rapid convergence.

Key Contributions

SALSA-V Model: A video-to-audio model capable of high-fidelity, few-step sampling without requiring additional fine-tuning or distillation.
Audio Conditioning & Outpainting: A masked training objective that allows the model to generate audio based on reference audio sections and perform iterative long-form extensions.
High-Resolution Synchronization: A contrastively trained alignment model using a large-scale vision backbone, yielding superior temporal synchronization features compared to prior methods.

Experimental Results

The authors evaluated SALSA-V against state-of-the-art models (including Frieren, MMAudio, LoVA, and FoleyCrafter) on mixed-length video clips (up to 30 seconds).

Quantitative Performance: SALSA-V achieves state-of-the-art results in synchronization (DeSync: 0.497s vs. 0.521s for MMAudio) and competitive performance in distribution matching metrics (FADVGG: 1.07, KLPaSST: 1.63). It outperforms all baselines in human evaluation for synchronization (3.52 MOS) and overall quality (3.43 MOS).
Few-Step Efficiency: In the few-step regime (4–8 steps), SALSA-V maintains high performance with minimal degradation. At 4 steps, it significantly outperforms Frieren (reflow) and MMAudio across all metrics, whereas other models suffer drastic quality drops.
Long-Form Generation: Unlike models that process full sequences at once (which degrade after ~20 seconds due to unseen positional embeddings), SALSA-V uses an iterative extension strategy. It maintains high synchronization and audio quality on 30-second samples, outperforming LoVA in all objective metrics.
Efficiency: SALSA-V generates a 10-second sample in 0.71 seconds using 4 steps on an NVIDIA A100, offering a favorable balance of speed and quality compared to competitors.

Significance and Claims

The paper claims that SALSA-V represents a significant step forward in making V2A generation practical for real-time and professional workflows. By integrating a shortcut loss, the model overcomes the latency bottleneck inherent in diffusion models, enabling near-real-time applications without sacrificing fidelity. The masked training objective broadens the utility of V2A models beyond simple generation to include professional sound design tasks like Foley generation and spectral matching.

The authors emphasize that their approach addresses the "long-form" gap, allowing for the synthesis of extended audio sequences that remain synchronized with video content, a capability previously limited by architectural constraints. The work suggests that these advancements can reduce the cost and iteration time for sound design in film, games, and user-generated content, while also acknowledging the potential risks regarding media manipulation and the need for safeguards in deployment.