Technical Summary: Unison – Harmonizing Motion, Speech, and Sound for Human-Centric Audio-Video Generation

1. Problem Statement

The paper identifies two fundamental challenges in current human-centric audio-video generation models, which hinder the creation of realistic, immersive content:

1. Intra-Audio Interference (Speech Dominance): Existing models often fail to balance heterogeneous acoustic elements within the audio stream. Structured speech components tend to dominate and obscure transient environmental sounds (e.g., footsteps, instrument strums, or ambient noise), leading to flattened auditory scenes. This is particularly problematic in scenarios like "singing while playing an instrument," where the model lacks mechanisms to disentangle and balance speech and sound effects.
2. Cross-Modal Misalignment: Motion and audio generation are typically linked only through implicit architectural cross-attention without explicit synchronization objectives. This results in desynchronized gestures, lip motions, and acoustic onsets. While some recent methods achieve frame-level synchronization, they often lack explicit consistency constraints, leading to temporal drift between visual actions and their corresponding sounds.

2. Methodology: The Unison Framework

Unison is a unified framework designed to generate motion, speech, and sound effects in a coherent, synchronized manner. It employs a dual-branch architecture (video and audio) coupled via bidirectional cross-attention, built upon a Flow Matching (FM) foundation with Diffusion Forcing (DF) principles.

2.1 Architecture Overview

• Dual-Branch Structure: The framework utilizes a video branch based on Wan2.2-5B and an audio branch built upon MMAudio, augmented with Zipformer to enable robust speech generation.
• Integration: The branches are connected via frame-level bidirectional cross-attention, where video and audio latents serve as mutual queries to facilitate seamless information exchange.
• Input/Output: The system takes a caption ($\kappa$) and a transcription ($\tau$) as inputs to generate synchronized video ($\nu$) and audio ($\alpha$).

2.2 Solution 1: Semantic-Guided Harmonization Strategy (Audio Stream)

To resolve intra-audio interference, Unison decouples the generation of speech and sound effects (SFX) into separate streams while maintaining their interaction.

• Bidirectional Audio Cross-Attention (Bi-ACA): The audio latent is represented as a dual-stream tensor comprising speech ($h^{sp}$) and sound-effect ($h^{sfx}$) components. These streams undergo an "interact-merge-split" cycle within Transformer blocks. They are concatenated, processed through shared self-attention with modality-specific biases to prevent semantic confusion, and then split. Crucially, Bi-ACA performs bidirectional cross-attention between the two streams, allowing them to mutually refine features and ensure acoustic consistency without relying solely on visual cues.
• Semantic-Conditioned Gating (SCG): To prevent unconstrained context injection, SCG adaptively regulates cross-stream updates. Global semantic vectors derived from the caption ($c_a$) and transcription ($c_s$) are used to predict gating coefficients ($g_{sp}, g_{sfx}$).
  • In narration-dominant scenes, SCG attenuates SFX features to preserve phonetic purity.
  • In complex soundscapes (e.g., musical performances), it amplifies cross-stream influence to enrich non-speech acoustics.
• Supervision: The model uses Mel-Roformer to disentangle mixed audio into ground-truth speech and SFX latents, optimizing the dual-stream decoder via independent flow-matching losses ($L_{sp}^{CFM} + L_{sfx}^{CFM}$).

2.3 Solution 2: Bidirectional Cross-Modal Forcing Strategy (Motion-Audio Alignment)

To address motion-audio desynchronization, Unison introduces a training strategy that explicitly aligns denoising progress between modalities.

• Decoupled Timesteps: Unlike prior works using identical timesteps, Unison independently samples timesteps ($t_v$ for video, $t_a$ for audio) for each modality. This allows the "cleaner" modality (lower noise level) to guide the "noisier" one.
• Directional Guidance: A direction indicator ($d$) identifies which modality is leading. The loss function is dynamically reweighted to prioritize the noisier branch, compelling it to extract stronger supervisory signals from the cleaner counterpart via cross-modal conditioning.
• Progressive Training Curriculum: To ensure stability, training follows a three-stage curriculum:
  1. Synchronous Warmup: Enforces $t_v = t_a$ to establish foundational alignment.
  2. Incremental Decoupling: Activates independent sampling with constraints on the noise gap ($|t_v - t_a| \leq 0.25$).
  3. Full Independence: Employs unconstrained decoupled updates.
     This progression prevents optimization instability caused by extreme noise disparities.

3. Key Contributions

1. Unified Framework: Proposes Unison, a framework that explicitly enforces cross-modal consistency to generate motion, speech, and sound in a harmonized manner, moving beyond the loose coupling of previous audio-visual models.
2. Semantic-Guided Harmonization: Introduces a strategy to decouple speech and sound-effect generation using Bidirectional Audio Cross-Attention (Bi-ACA) and Semantic-Conditioned Gating (SCG). This resolves acoustic interference, preventing speech dominance and enabling controllable, layered audio generation.
3. Bidirectional Cross-Modal Forcing: Develops a strategy that strengthens temporal dependencies by allowing the cleaner modality to guide the noisier one through decoupled denoising schedules, effectively bridging the gap between motion and acoustics.

4. Experimental Results

The authors evaluated Unison on a curated test set of 1,000 samples against state-of-the-art baselines (including Universe-1, UniAVGen, MOVA, and LTX-2).

• Quantitative Performance:
  • Audio Fidelity: Unison achieved the highest scores in Perceptual Quality (PQ: 6.34), Content Usefulness (CU: 5.61), and Word Error Rate (WER: 0.22), outperforming models with significantly larger parameter counts (e.g., LTX-2's 19B vs. Unison's 5B video backbone).
  • Synchronization: The model demonstrated superior cross-modal consistency, achieving the best DeSync (DS: 0.08) and competitive Lip-Speech Synchronization (LSE-C: 3.30) scores.
• Qualitative Improvements:
  • Visual comparisons showed Unison successfully synthesizing harmonious speech and sound effects (e.g., balancing singing with guitar strums) where baselines suffered from speech dominance or missing sound effects.
  • Ablation studies confirmed that removing the Semantic-Guided Harmonization Strategy led to significant audio quality degradation, while removing the Cross-Modal Forcing Strategy resulted in sharp increases in temporal desynchronization (DS rising to 0.19).
• User Study: In a study with 25 participants, Unison received the dominant overall preference, ranking highest in speech-sound harmony and motion-audio alignment.

5. Significance and Claims

The paper claims that Unison establishes a principled approach to human-centric video generation by explicitly addressing the heterogeneity of temporal characteristics across modalities.

• Balanced Auditory Scenes: By decoupling speech and sound effects, the framework ensures that environmental sounds are not suppressed by human voices, enhancing perceptual realism.
• Robust Synchronization: The bidirectional cross-modal forcing strategy creates a stable training environment where audio and video mutually reinforce each other, leading to tighter temporal correspondence without relying solely on massive parameter scaling.
• Flexibility: The symmetric design enables bidirectional generation (Audio-to-Video and Video-to-Audio), demonstrating the framework's adaptability as a unified model for multimodal content creation.

The authors conclude that explicit multimodal harmonization is critical for the next generation of generative AI, and Unison's architectural innovations effectively capture high-fidelity audiovisual correlations.