Technical Summary: Efficient Audio-Visual Generation via Synchrony-Aware Cross-Modal Sparse Attention

1. Problem Statement

Recent unified audio-visual generation models (e.g., Ovi, Harmony, LTX-2) synthesize synchronized video and sound using bidirectional cross-modal diffusion. However, these models suffer from high inference costs due to the repeated evaluation of dense attention over long spatio-temporal token sequences.

Existing acceleration techniques for video generation—such as low-bit quantization, attention sparsification, and feature caching—are primarily designed for single-modality (video-only) tasks. They rely on identifying redundancy within a single modality (e.g., spatial or temporal redundancy in video frames). Directly applying these methods to audio-visual models is problematic because:

• Cross-Modal Dependence: A token or feature that appears redundant within the video branch may still provide critical evidence for the audio branch (and vice versa) through bidirectional interaction.
• Synchronization Risk: Modality-agnostic acceleration strategies may inadvertently skip or approximate "synchronization-critical" computations, leading to a degradation in audio-video alignment, audio quality, and visual fidelity.

The core challenge is to distinguish between computation that can be safely approximated and computation that carries essential cross-modal evidence, without relying on independent analysis of either stream.

2. Methodology

The authors propose a training-free, synchronization-aware acceleration framework that leverages the model's native bidirectional cross-attention as an internal saliency signal. The framework operates on the observation that cross-modal interactions are highly structured and non-uniform: high responses are concentrated on specific sound-related visual regions and temporal cues.

The framework consists of three main components:

A. Bidirectional Cross-Modal Saliency Estimation

The method aggregates attention masses from bidirectional cross-attention layers to generate modality-specific saliency maps:

• Video Saliency ($s^v$): Derived from audio queries attending to video keys, identifying video tokens associated with sound-producing objects or events.
• Audio Saliency ($s^a$): Derived from video queries attending to audio keys, identifying audio tokens characterizing speech or acoustic events.
• Lagged Saliency Reuse: Since cross-modal attention is computed after modality-specific self-attention within a block, the current saliency map cannot guide the self-attention just executed. To avoid a costly dense profiling pass, the method employs a lagged saliency cache. It updates a cached saliency map ($\bar{s}$) using a moving average of the current and previous steps, exploiting the empirical stability of cross-modal interactions across neighboring denoising steps.

B. Synchronization-Aware Protected Sparse Attention

This component modifies standard sparse attention to preserve high-fidelity computation for critical tokens:

• Protected Query Selection: Based on the cached saliency maps, a small subset of queries (e.g., top 5%) in both video and audio branches are selected as "protected."
• Hybrid Attention Mechanism:
  • Protected Queries: These retain dense attention connectivity (full interaction with all keys) to ensure high-fidelity representation updates for synchronization-critical content.
  • Ordinary Queries: These follow the standard sparse attention pattern (restricted connectivity) to reduce redundant computation.
• This ensures that sound-related visual tokens can aggregate complete visual context before interacting with the audio branch, and vice versa.

C. Synchronization-Aware Cache Reuse

Standard feature caching reuses intermediate representations when global feature changes are small. This method introduces a stricter criterion to prevent unsafe reuse that could disrupt alignment:

• Multi-Factor Indicator: The cache reuse decision ($\Delta^{sync}$) is the maximum of four factors:
  1. Global video feature change ($\Delta^v$).
  2. Global audio feature change ($\Delta^a$).
  3. Protected-region change ($\Delta^p$): Specifically measuring changes in the high-saliency regions identified by the saliency map.
  4. Saliency drift ($\Delta^s$): Measuring the cosine similarity drift between current and cached saliency maps.
• Reuse Condition: A block is only reused if $\Delta^{sync}$ is below a threshold $\tau$. This prevents caching when synchronization-critical regions or cross-modal dependencies shift significantly, even if the global feature change is small.

3. Key Contributions

1. Analysis of Cross-Modal Interactions: The paper demonstrates that bidirectional audio-video attention reveals structured, non-uniform interactions where high responses concentrate on sound-relevant visual/temporal regions. This serves as an intrinsic saliency signal for identifying synchronization-critical content.
2. Protected Sparse Attention: A training-free mechanism that uses modality-specific saliency maps to preserve dense attention for critical tokens while sparsifying low-saliency interactions, effectively balancing efficiency and synchronization.
3. Synchronization-Aware Cache Reuse: A novel caching criterion that jointly considers modality-specific feature variations, changes in salient regions, and the drift of cross-modal dependencies to prevent unsafe feature reuse.
4. Empirical Validation: The method is validated on open-source models (Ovi and LTX-2.3), showing significant speedups while maintaining video fidelity, audio quality, and synchronization.

4. Experimental Results

The method was evaluated on Ovi (141-frame, 5s videos) and LTX-2.3 (10s videos) against dense baselines and state-of-the-art video acceleration methods (SparseVideoGen, TeaCache, BWCache, SeaCache, FasterCache).

• Efficiency:
  • Ovi: Achieved 1.992× end-to-end speedup (latency reduced from 448.75s to 225.30s).
  • LTX-2.3: Achieved 1.281× speedup (latency reduced from 255.41s to 199.38s).
• Quality & Synchronization:
  • The proposed method outperformed all other accelerated baselines in PSNR, SSIM, LPIPS, and Q-Align (perceptual quality).
  • Crucially, it maintained superior audio-video synchronization (Sync-C and Sync-D metrics) and audio similarity (VISQOL), whereas other methods often degraded synchronization significantly.
  • For example, on Ovi, the proposed method achieved a PSNR of 25.69 and Sync-C of 4.61, compared to 21.76/3.66 for SparseVideoGen and 24.78/3.74 for TeaCache.
• Error Analysis: Feature error analysis showed that vanilla sparse attention introduces high errors on salient tokens, while the protected approach significantly reduces error in these critical regions without increasing error in less salient areas.

5. Significance and Claims

The paper claims that accelerating audio-visual generation requires more than just identifying computational redundancy; it requires distinguishing computation that carries cross-modal evidence. By explicitly accounting for cross-modal dependence during acceleration, the proposed framework:

• Preserves Synchronization: It prevents the "weakening" of audio-video correspondence that occurs when modality-agnostic methods skip critical interactions.
• Maintains Quality: It achieves high-fidelity generation comparable to dense inference while offering substantial speedups.
• Is Training-Free: The method does not require retraining the generative backbone, making it applicable to existing open-source dual-branch models.

The authors conclude that aligning acceleration policies with audio-video interaction patterns is essential for efficient, high-quality unified audio-visual generation.