💻 computer science

Dialogue-Aware Video-to-Music Generation Using Public Domain Film Collections

To address reproducibility and copyright challenges in video-to-music generation, this paper introduces the reproducible Open Screen Soundtrack Library version 2 (OSSL-v2) dataset of public-domain films and proposes a dialogue-aware model that enhances music generation by modulating video cross-attention with on-screen speech.

Original authors: Haven Kim, Zachary Novack, Julian McAuley, Hao-Wen Dong

Published 2026-08-13

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Haven Kim, Zachary Novack, Julian McAuley, Hao-Wen Dong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a movie. You see a hero running through a storm, and suddenly, thunder rumbles while a dramatic orchestra swells. That music isn't just background noise; it's the emotional glue holding the scene together, telling you how to feel. For a long time, scientists have been trying to teach computers to be the "composer" for videos, automatically creating music that matches the mood of what's happening on screen. This field is called video-to-music generation. But there's a huge problem: to teach a computer this skill, you need thousands of examples of movies and their perfect soundtracks. Most researchers have been using lists of links to videos on the internet to find these examples. The trouble is, internet links are like sandcastles; the tide comes in, and the videos get deleted or locked away. This means the "textbooks" the computers learn from keep disappearing, making it impossible to check if one computer is actually smarter than another. To fix this, we need a library of movies that can't vanish, and a way to teach the computer not just what the characters look like, but also what they are saying.

Enter the researchers from UC San Diego and the University of Michigan, who have built a brand-new, unshakeable library called OSSL-v2. Think of this as a massive, self-contained movie theater that no one can shut down. Instead of pointing to links on the internet that might rot away, they downloaded 1,886 films that belong to the public (meaning anyone can use them) and chopped them into 34,343 little clips. This adds up to about 246.4 hours of video and music, all safely stored in one place. Because it's self-hosted, it won't disappear, and it gives every researcher a fair, identical set of data to train their AI models on.

But the team didn't just build a library; they realized the computers were missing a crucial clue. In movies, the music often changes the moment a character starts speaking. If someone is shouting, the music might get quieter or stop to let the voice be heard. The researchers noticed that in their new library, there is a slight but real connection between how loud the dialogue is and how loud the music is. To use this, they invented a "dialogue-aware" system. Imagine the computer is a conductor. Before, the conductor only looked at the actors' faces and movements to decide when to speed up or slow down the orchestra. Now, the conductor has a new pair of glasses that lets them hear the actors' voices in real-time. The system takes the sound of the dialogue and uses it to gently nudge the music generation frame-by-frame, ensuring the music breathes and pauses exactly when the characters do.

When they tested this new approach on their massive library, the results were promising. They compared their "dialogue-aware" models against the best existing systems. The new method didn't necessarily make the music sound "better" in a general sense (like making it more complex or diverse), but it did make the music fit the video much more tightly. Specifically, the generated soundtracks became more faithful to the original movie scenes, especially when the computer had to handle tricky moments where dialogue and music overlap. The researchers found that this improvement held true even when they tested the models on commercial movies they had never seen before, suggesting that listening to the dialogue helps the AI understand the rhythm of a scene better than just watching it.

The paper also carefully ruled out a few potential tricks. They worried that maybe the computer was just "relying on" leftover music leaking into the dialogue track (since separating audio isn't perfect). To check this, they ran a special filter to find clips where the dialogue was almost completely free of any music noise. Even with these "clean" clips, the dialogue-aware system still performed better. This suggests the improvement really comes from understanding the speech, not from accidentally hearing the music. While the system isn't a perfect replacement for human composers yet, the study suggests that adding the voice of the characters is a powerful way to help computers write music that feels truly connected to the story.

Technical Summary: Dialogue-Aware Video-to-Music Generation Using Public Domain Film Collections

Problem Statement
The field of video-to-music generation lacks a common, durable training dataset to serve as a reproducible benchmark. Current progress is hindered by a "reproducibility gap": most models are trained on web-crawled corpora referenced via URLs (e.g., YouTube, TikTok). These datasets suffer from "link rot," where videos are deleted or made private over time, and are subject to rate-limiting that makes re-acquisition time-consuming. Furthermore, existing self-hosted datasets are often too small (e.g., ~36 hours) to train competitive models. Additionally, existing approaches often overlook dialogue as a conditioning signal, despite the close temporal coupling between on-screen speech and film music in professional productions.

Methodology
The authors propose a two-pronged approach: the creation of a new dataset and a novel architectural modification for dialogue-aware generation.

Dataset Construction (OSSL-v2):
The authors introduce the Open Screen Soundtrack Library version 2 (OSSL-v2), a self-hosted corpus derived from 1,886 public-domain films. The dataset comprises 34,343 video clips totaling 246.4 hours.
- Source Separation: An open-source cinematic source separation model is applied to extract music tracks from the original audio.
- Event Detection: To ensure high quality, an automatic event detection model estimates the probability of musical vs. non-musical events. Segments are selected where the music probability exceeds the non-music probability for at least 10 consecutive seconds, with an additional filter to exclude segments where non-music probability is too high (threshold > 0.05).
- Result: A reproducible, copyright-conscious dataset free from link rot, suitable for training and fair comparison.
Dialogue-Aware Generation Architecture:
The authors propose a lightweight dialogue-conditioned module to be inserted into existing open-source video-to-music backbones (specifically VidMuse, GVMGen, and Diff-V2M).
- Signal Processing: The dialogue track is isolated using source separation and encoded via a 1-D convolutional layer to produce per-frame acoustic representations (e.g., loudness, energy).
- Modulation: A two-layer MLP maps these representations to Feature-wise Linear Modulation (FiLM) parameters ( $\gamma_i, \beta_i$ ). These parameters modulate the video conditioning vectors frame-by-frame: $\tilde{v}_i = (1 + \gamma_i) \odot v_i + \beta_i$ .
- Temporal Alignment: To ensure the model respects the timing of dialogue relative to visual events, the authors modify the cross-attention mechanisms in the backbones. For models like GVMGen that compress frames into global vectors, they restore a time axis by adding learned per-frame positional embeddings to the Q-Former outputs and enabling fixed sinusoidal positional embeddings on cross-attention keys and values. This allows the dialogue to modulate the conditioning signal on a frame-by-frame basis.

Key Contributions

OSSL-v2 Dataset: A large-scale (246.4 hours), self-hosted video-music corpus sourced from public-domain films. It addresses the impermanence and scalability issues of previous web-scraped datasets, enabling reproducible research.
Dialogue Conditioning: A demonstration that dialogue serves as an effective conditioning signal for film music generation. The proposed adapter integrates acoustic dialogue features into the video conditioning pathway without requiring a complete model overhaul.
Architectural Adaptation: Specific modifications to cross-attention mechanisms in autoregressive and diffusion-based models to preserve temporal identity, allowing for frame-aligned modulation by dialogue.

Experimental Results
The models were trained and evaluated on OSSL-v2 (9:1 split) and an out-of-distribution commercial film set (OES-Com). Evaluation metrics included distributional fidelity (FAD, Precision, Recall) and paired fidelity (CLAP similarity, KL divergence).

Baseline Performance: Among the baselines, GVMGen achieved the strongest overall performance on the public-domain test set. Diff-V2M showed high paired fidelity on out-of-distribution data but failed to generate samples conforming to the ground-truth distribution (Precision = 0.00).
Impact of Dialogue Adapter:
- Paired Fidelity: The dialogue adapter consistently improved paired fidelity (semantic faithfulness to ground truth), particularly in terms of reducing KL divergence and increasing CLAP similarity.
- Generalization: The improvements were often more pronounced on the out-of-distribution OES-Com set than on the in-distribution OSSL-v2 set, suggesting the dialogue signal acts as a regularizer that aids generalization.
- Distributional Fidelity: The adapter did not systematically reshape the generated distribution (FAD and Precision varied inconsistently).
- Robustness: These gains persisted on Low-Residual-Music (LRM) subsets, where the separated dialogue stems contained negligible leaked music. This confirms that the improvements are driven by the dialogue signal itself rather than artifacts of imperfect source separation.

Significance and Claims
The paper claims that OSSL-v2 lowers the barrier to reproducible research and fair comparison in video-to-music generation by providing a durable, self-hosted benchmark. Regarding the methodology, the authors modestly claim that their dialogue-aware formulation improves upon state-of-the-art baselines, particularly in temporal fidelity (similarity to ground truth) and generalization to commercial films. They posit that incorporating dialogue as a time-aligned conditioning signal addresses a gap in current film-domain generation, where global video-level information alone is insufficient to capture the local, temporal interactions between speech and music. The work does not claim to solve all challenges in the field but identifies dialogue conditioning as a promising direction for future development.

Technical Summary: Dialogue-Aware Video-to-Music Generation Using Public Domain Film Collections

More like this