💻 computer science

Taming Text-to-Sounding Video Generation via Advanced Modality Condition and Interaction

This paper addresses the challenges of text-to-sounding-video generation by proposing the Hierarchical Visual-Grounded Captioning (HVGC) framework to disentangle text conditions and the BridgeDiT model with a Dual CrossAttention mechanism to achieve robust semantic and temporal synchronization, resulting in state-of-the-art performance across multiple benchmarks.

Original authors: Kaisi Guan, Xihua Wang, Zhengfeng Lai, Xin Cheng, Peng Zhang, XiaoJiang Liu, Ruihua Song, Meng Cao

Published 2026-06-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Kaisi Guan, Xihua Wang, Zhengfeng Lai, Xin Cheng, Peng Zhang, XiaoJiang Liu, Ruihua Song, Meng Cao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to create a movie scene just by typing a sentence, like "A blacksmith hits a hot piece of iron." You want the video to show the hammer striking, and you want the audio to be the loud clang of metal hitting metal, perfectly timed so the sound happens exactly when the hammer hits.

This paper introduces a new system called BridgeDiT that does exactly that. It turns text into a video with perfectly synchronized sound. The authors argue that previous attempts to do this were like trying to build a house by building the roof and the foundation separately and then hoping they fit together. Often, the roof would be late, or the sound would be wrong.

Here is how they fixed it, explained in simple terms:

1. The Problem: Two Different Languages

The researchers identified two main headaches that stopped other systems from working well:

The "One Script for Two Actors" Problem: Imagine a director giving the exact same script to an actor playing a visual role and an actor playing an audio role. The visual actor needs to know about colors and shapes, while the audio actor needs to know about volume and rhythm. If you give them the same text, they get confused. The audio actor might try to "hear" a color, and the visual actor might try to "see" a sound. This causes interference.
The "Short Note vs. Long Story" Problem: When you ask the AI, you usually give a short note like "A man hits iron." But the AI was trained on long, detailed stories like "A muscular blacksmith with soot on his face strikes a glowing orange hammer against a hot anvil, sending sparks flying." If you feed the AI the short note, it gets lost because it's used to the long stories.

2. The Solution: The "Translator and Editor" Team (CRR)

To fix the script problem, they built a smart pipeline called the Cross-Referential Rewriter (CRR). Think of this as a team of two editors working together:

The Fact-Checker (Semantic Checker): First, they take the short user note and imagine what the scene looks and sounds like. But here's the trick: they don't just guess. They cross-reference the visual and audio ideas. If the audio editor suggests a "bird chirping" but the visual editor sees a "blacksmith," the Fact-Checker says, "Nope, birds don't belong in a blacksmith shop." It filters out hallucinations (fake sounds) and keeps only the things that make sense together.
The Specialist Writers (Cross-Modal Rewriter): Once the facts are checked, this team splits the story into two separate, perfect scripts.
- Script A (Video): Describes only what you see (the hammer, the sparks, the muscles). It strictly avoids words like "loud" or "clang."
- Script B (Audio): Describes only what you hear (the metallic ring, the rhythm). It strictly avoids words like "red" or "hammer."
- The Magic: They also take your short note ("man hits iron") and expand it into the long, detailed story the AI loves, ensuring the short note gets the same treatment as the long training stories.

3. The Solution: The "Bridge" (BridgeDiT)

Once the two actors have their separate, perfect scripts, they need to perform together. In the past, systems either tried to force them to share one brain (which was messy) or built a wall between them (which meant they couldn't sync up).

The authors built a Bridge called Dual Cross-Attention (DCA).

Imagine the Video AI and the Audio AI are two people in separate rooms.
Instead of merging them into one room, they built a special two-way walkie-talkie system between the rooms.
Every few seconds, the Video person whispers, "I'm hitting the hammer now!" to the Audio person.
The Audio person whispers back, "Okay, I'm making the clang sound right now!"
This happens constantly and in both directions. This ensures that when the video shows the hammer moving, the audio knows exactly when to start the sound, without mixing up the visual details with the sound details.

4. The Result

The paper tested this system on three different datasets. The results showed that:

The video and audio were much better synchronized than any previous method.
The sounds matched the text descriptions better.
Human testers preferred these videos over others because the timing felt natural, like a real movie, rather than a video with a slightly off-beat soundtrack.

In summary: The paper didn't just build a better video generator; it built a better conductor. It uses a smart editor to write two separate, perfect scripts (one for eyes, one for ears) and a special bridge to ensure the two musicians play in perfect time, creating a seamless experience from a simple text prompt.

Technical Summary: Taming Text-to-Sounding Video Generation via Advanced Modality Condition and Interaction

Problem Statement

The paper addresses Text-to-Sounding-Video (T2SV) generation, the task of synthesizing a video with synchronized audio from a single text prompt. While Text-to-Video (T2V) and Text-to-Audio (T2A) generation have matured, joint generation remains challenging due to two primary bottlenecks:

The Conditioning Problem: Existing dual-tower methods often rely on a shared caption ( $T_V = T_A$ ) for both modalities. This causes semantic interference, as visual attributes (e.g., color) confuse audio models and auditory descriptors (e.g., timbre) distract video models. Furthermore, a significant training-inference gap exists: models are trained on dense, detailed captions, but users provide concise prompts at inference, leading to distribution shifts and degraded performance.
The Interaction Problem: In dual-tower architectures, the optimal mechanism for exchanging information between the video and audio streams to achieve both semantic and temporal synchronization remains unclear. Previous strategies, such as full attention or unidirectional conditioning, often fail to balance feature preservation with effective cross-modal fusion.

Methodology

The authors propose BridgeDiT, a framework comprising a novel captioning pipeline and a dual-tower diffusion architecture.

1. Cross-Referential Rewriter (CRR) Caption Framework

To resolve the conditioning problem, the authors introduce a dual-agent pipeline that generates disentangled, modality-pure captions ( $T_V$ and $T_A$ ) from a single input.

Semantic Checker ( $\mathcal{F}_{sc}$ ): This agent cross-references raw captions generated by separate Video and Audio Large Language Models (LLMs). It extracts Semantic Anchors—a structured set of grounded attributes (entities, environment, actions, and corresponding sounds). Crucially, it filters out hallucinations and semantic conflicts (e.g., an audio LLM hallucinating a sound with no visual source) by using the visual description as a grounding reference.
Cross-Modal Rewriter ( $\mathcal{F}_{cr}$ ): This agent generates dense, disentangled captions strictly from the verified Semantic Anchors.
- Training: It converts grounded anchors into detailed, modality-specific descriptions.
- Inference: It expands concise user prompts into dense descriptions that match the training distribution, effectively bridging the training-inference gap without complex user-side prompt engineering.

2. BridgeDiT Architecture with Dual Cross-Attention (DCA)

The core generation model is a dual-tower diffusion transformer consisting of a video tower ( $G^V_\theta$ ) and an audio tower ( $G^A_\theta$ ), connected by a lightweight interaction module.

Dual Cross-Attention (DCA): The authors propose DCA as the interaction mechanism. Unlike Full-Attention (which concatenates latents and forces a shared feature space) or unidirectional conditioning, DCA employs symmetric, bidirectional cross-attention streams.
- In the Audio-to-Video (A2V) stream, the video latent acts as the Query, while the audio latent provides Keys and Values.
- In the Video-to-Audio (V2A) stream, the roles are reversed.
Rationale: This design preserves the distinct feature manifolds of the pre-trained backbones (which are largely frozen) while learning a lightweight "bridge" for targeted information exchange. This avoids the optimization difficulty of merging disjoint manifolds into a single space.

Key Contributions

CRR Caption Framework: A dual-agent pipeline that eliminates modal interference through grounded Semantic Anchors and bridges the training-inference gap via prompt expansion.
BridgeDiT Architecture: A dual-tower diffusion model utilizing Dual Cross-Attention (DCA), identified through systematic comparison as the optimal fusion strategy for pre-trained backbone-based T2SV systems.
Comprehensive Evaluation: Extensive experiments on three benchmarks (AVSync15, VGGSound-SS, Landscape) demonstrating state-of-the-art performance, supported by human evaluations and ablation studies validating each component.

Experimental Results

The model was evaluated on three datasets using metrics for generation quality (FVD, KVD, FAD, KL), text alignment (CLIPSIM, CLAP), and synchronization (VA-IB, AV-Align).

Performance: BridgeDiT achieved state-of-the-art results on most metrics across all three benchmarks. It significantly outperformed both retrained baselines (using the same CRR captions) and pre-trained baselines.
Ablation Studies:
- Conditioning: Disentangled captions generated by CRR significantly outperformed shared captions. Removing the Semantic Anchor bottleneck or prompt expansion led to severe degradation in audio quality and synchronization.
- Fusion: DCA outperformed Full-Attention, Additive Fusion, and unidirectional Cross-Attention. The authors attribute this to DCA's ability to maintain separate feature spaces while enabling symmetric interaction, which is critical when using frozen pre-trained backbones.
Human Evaluation: In a user study with 20 raters, BridgeDiT received the highest scores across Video Quality, Audio Quality, Text Alignment, Synchronization, and Overall Quality, outperforming all baselines including pipelined approaches.

Significance and Claims

The paper claims that T2SV generation requires a fundamental shift in how text conditions are processed and how modalities interact.

On Conditioning: The authors argue that "modality purity" is essential; feeding shared captions to specialized backbones introduces noise that degrades generation. The CRR framework provides a robust solution by grounding text generation in verified semantic anchors.
On Interaction: The study challenges the assumption that Full-Attention is universally optimal for multimodal fusion. It demonstrates that for systems leveraging pre-trained, frozen backbones, Dual Cross-Attention is superior because it respects the disjoint nature of the pre-trained feature manifolds, allowing for stable training and precise synchronization without compromising the quality of individual modalities.
Limitations: The authors note that current evaluation relies on relatively small-scale benchmarks and that the system does not yet support speech or music generation due to the scarcity of open-source pre-trained models for those specific modalities. They suggest future work in scaling datasets and exploring post-training refinement via Reinforcement Learning from Human Feedback (RLHF).