Technical Summary: Baton – Explicit Semantic Blueprints for Joint Video-Audio Generation

1. Problem Statement

Current open-source joint video-audio generation models, while capable of producing synchronized content for simple prompts, struggle significantly with scenarios requiring complex semantic reasoning. These scenarios include multi-stage actions, intricate human-object interactions, and multi-speaker dialogues.

The root cause of this failure lies in the prevailing methodology: existing models rely on coarse global text embeddings derived from off-the-shelf encoders (e.g., T5, Qwen) to guide the denoising trajectories of both video and audio diffusion branches. This approach suffers from two critical limitations:

1. Loss of Fine-Grained Semantics: Global embeddings fail to decompose complex instructions into modality-specific, temporally grounded details.
2. Lack of Shared Long-Horizon Planning: Without a coordinated plan, the video and audio denoising trajectories drift apart, leading to adversarial dynamics. This results in severe cross-modal misalignment, body distortion, and the inability to maintain subject consistency or synchronize specific actions (e.g., a punch landing with a sound effect) over long durations.

While some recent works attempt to enrich prompts using Large Language Models (LLMs), they still inject the resulting representations as global embeddings, failing to address the structural misalignment between semantic intent and the spatio-temporal generation process.

2. Methodology: Baton

Baton proposes a framework that explicitly disentangles semantic reasoning from synthesis. Instead of relying solely on global embeddings, Baton introduces a "planning" phase that generates explicit, modality-aware blueprints before the diffusion process begins. The architecture consists of three main components:

2.1 VA-Planner (Video-Audio Planner)

The core innovation is the VA-Planner, a trainable Multimodal Language Model (MLLM) equipped with dual semantic alignment towers.

• Input: The planner receives a structured prompt containing system instructions, video descriptions, and audio descriptions.
• Reasoning: It performs autoregressive reasoning to predict a pair of planned tokens: one set for video ($H_v$) and one for audio ($H_a$). These tokens serve as keyframe-level blueprints.
• Dual Alignment Towers: To ensure cross-modal consistency, the planner uses two towers with learnable queries.
  • The video tower extracts video-specific semantics from the MLLM's hidden states and then performs cross-modal attention to absorb complementary auditory cues.
  • The audio tower does the symmetric operation.
  • This bidirectional attention ensures that the video plan is aware of audio context and vice versa, producing mutually consistent temporal blueprints.
• Feature Projection: The planned tokens are projected into the continuous feature domains of pretrained perceptual encoders (SigLip2 for video, WavTokenizer for audio) to serve as high-fidelity semantic guides.

2.2 Relative Semantic RoPE (RS-RoPE)

A significant challenge arises because the planned tokens (generated at a sparse keyframe resolution) and the diffusion latents (generated at a dense spatio-temporal grid) do not share a one-to-one correspondence. Standard positional encodings fail here, causing the diffusion model to treat the structured plan as a vague global signal.

Baton introduces Relative Semantic RoPE, a relative positional encoding mechanism that:

• Maps the planned tokens and diffusion latents into a shared spatial-temporal coordinate frame.
• Rescales position indices based on the physical time and spatial location they represent.
• Enables each diffusion latent to accurately attend to its positionally corresponding semantic cue, preserving the keyframe-level structure of the blueprint rather than flattening it.

2.3 Training Strategy

Baton employs a three-stage training curriculum to ensure robustness:

1. VA-Planner Pretraining: The MLLM and alignment towers are trained to predict continuous perceptual features (from frozen SigLip2 and WavTokenizer) based on text prompts. This teaches the planner to generate accurate semantic blueprints.
2. DiT Adaptation: The Diffusion Transformer (DiT) is trained using ground-truth perceptual features (bypassing the planner) to learn the semantic feature distribution without being confounded by planner prediction noise.
3. Joint Fine-tuning: The VA-Planner is frozen, and the DiT is fine-tuned using the planner's predicted tokens. This stage bridges the gap between ideal ground-truth features and realistic planner predictions, mitigating exposure bias.

3. Key Contributions

1. VA-Planner: The first framework to introduce explicit semantic planning for joint video-audio generation. It generates semantically aligned video and audio planned tokens that act as keyframe-level blueprints, anchoring both denoising trajectories to a shared roadmap.
2. Relative Semantic RoPE: A novel positional encoding that resolves the spatio-temporal mismatch between sparse semantic plans and dense diffusion latents, enabling precise cross-modal attention.
3. Disentanglement of Reasoning and Synthesis: By separating the semantic planning phase from the generation phase, Baton prevents the adversarial drift seen in models that rely solely on global embeddings.

4. Experimental Results

The authors evaluated Baton on Verse-Bench (standard open-source benchmarks) and Sem100 (a custom dataset of 100 unseen videos with complex, multi-stage prompts requiring deep semantic reasoning).

• Quantitative Performance:

  • On Verse-Bench, Baton achieves results comparable to the state-of-the-art (SOTA) model LTX-2.
  • On Sem100, Baton significantly outperforms all open-source competitors. It shows a 32% improvement in Prompt Following Accuracy (P-Acc), a 76% reduction in Multi-Speaker Word Error Rate (M-WER), and a 30% reduction in Desynchronization (DeSync) compared to LTX-2.
  • The improvements in M-WER and P-Acc highlight the model's superior ability to handle multi-speaker dialogues and complex sequential actions.

• Qualitative Performance:

  • Visual comparisons demonstrate that Baton produces stable video-audio content with correct subject consistency and synchronized actions, whereas competitors (Ovi, UniAVGen, LTX-2, MOVA) suffer from body distortion, object disappearance, and audio-video desynchronization in complex scenarios.
  • In user studies, Baton was preferred over competitors for video/audio alignment, identity consistency, and synchronization.

• Comparison with Commercial Models:

  • While commercial models (e.g., Veo 3, Seedance 2.0) may exhibit higher visual fidelity or audio aesthetics, Baton achieves comparable prompt-following capabilities, particularly in scenarios demanding complex semantic reasoning where commercial models sometimes fail to follow multi-stage instructions accurately.

5. Significance and Claims

The paper claims that Baton is the first framework to explicitly disentangle semantic planning from synthesis for joint video-audio generation. Its significance lies in addressing the fundamental limitation of current diffusion models: the inability to maintain cross-modal coherence over long horizons without a shared, pre-aligned semantic roadmap.

By introducing explicit semantic blueprints and a mechanism to align them with diffusion latents (RS-RoPE), Baton transforms generation from "blindly denoising under vague global signals" to "following a semantically grounded and cross-modally coordinated plan." This allows the model to synthesize stable, synchronized video-audio content even for prompts involving complex causal relationships, multi-stage actions, and multi-speaker interactions, setting a new standard for open-source joint video-audio generation.