FoleyDirector: Fine-Grained Temporal Steering for Video-to-Audio Generation via Structured Scripts

This paper introduces FoleyDirector, a framework that leverages Structured Temporal Scripts and a novel Temporal Fusion Module to achieve precise, fine-grained temporal control in DiT-based Video-to-Audio generation, effectively addressing challenges in multi-event scenarios and off-screen sound synthesis while maintaining high audio fidelity.

The Gist

Imagine you are watching a silent movie. In the old days, a human "Foley artist" would sit in a studio, watching the screen, and manually create sounds: crunching celery for footsteps, shaking a sheet of metal for thunder, or honking a horn when a car appears.

Now, imagine you have a robot that can do this automatically. But here's the problem: most current robots are a bit clumsy. If you tell them, "Make a car honk," they might honk it for the whole 10 seconds of the video, or they might miss it entirely if the car is hidden behind a tree. They struggle to say, "Honk only between seconds 5 and 6, and be silent otherwise."

FoleyDirector is like giving that robot a precise script and a director's baton. It's a new AI system that lets you control exactly when and what sounds happen in a video, even if the sound isn't visible on screen.

Here is how it works, broken down with simple analogies:

1. The Problem: The "Blurry" Robot

Current video-to-audio AI is like a musician who hears a song but can't read sheet music. They know the general vibe (e.g., "it's a busy street"), but they can't tell you exactly when the siren goes off or when the dog barks. If the visual cue is weak (like a tiny object or something off-screen), the robot gets confused and either stays silent or makes random noise.

2. The Solution: The "Structured Script" (STS)

The authors realized that instead of just giving the robot a vague description like "a busy street," we should give it a minute-by-minute script, just like a movie director gives to actors.

• The Analogy: Imagine you are directing a play. Instead of saying, "Make noise during the play," you hand the sound guy a script that says:
  • 0:00–0:05: Silence.
  • 0:05–0:06: Car horn (loud).
  • 0:06–0:10: Silence.
  • 0:10–0:12: Cat meowing.
• How the AI uses it: FoleyDirector breaks the video into tiny 1-second chunks. For each chunk, it creates a specific "script" describing exactly what should happen. This gives the AI a roadmap, so it knows exactly when to start and stop making noise.

3. The Magic Glue: The "Script-Guided Fusion"

Now, the AI has the video, the general text description, and this new detailed script. But how do you mix them without the AI getting a headache?

• The Analogy: Think of the AI as a chef. The video is the main ingredient (the steak). The script is the seasoning. If you just throw the seasoning on top, it might not mix well.
• The Innovation: FoleyDirector uses a special "fusion module" (a smart mixing bowl) that blends the script into the cooking process. It uses a technique called Interleaved RoPE, which is like a zipper. It zips the "script instructions" right into the "video timeline" so they stay perfectly aligned. The AI doesn't just hear the script; it feels the timing of the script as it creates the sound.

4. Handling the "Invisible" Sounds: The "Bi-Frame" Trick

Sometimes, you want a sound that isn't on the screen. Maybe a character is talking, but you want to hear a dog barking in the distance (off-screen). Or maybe you want a character to laugh, but the video shows them crying (a counterfactual).

• The Analogy: Imagine a stage with two actors.
  • Actor A (In-Frame): Follows the video perfectly. If the video shows a dog, Actor A barks.
  • Actor B (Out-of-Frame): Ignores the video and listens only to your script. If your script says "dog barking off-screen," Actor B barks, even if the video shows a cat.
• The Innovation: FoleyDirector runs these two "actors" in parallel. It lets the video guide the visible sounds, but lets your script guide the invisible sounds. Then, it stitches them together seamlessly. This allows for complex storytelling, like a horror movie where you hear a monster approaching even though you can't see it yet.

5. The Result: You Become the Director

Before this, AI was like a radio station playing whatever it felt like. With FoleyDirector, you are the director.

• You can tell the AI: "Silence for the first 3 seconds, then a train horn, then silence again."
• You can tell it: "The video shows a tiger, but I want it to meow like a cat."

Why This Matters

This isn't just about making cool sound effects. It's about control.

• For Filmmakers: You can fix bad audio in post-production without re-recording everything.
• For Storytellers: You can create immersive experiences where sound drives the emotion, not just the visuals.
• For Accessibility: It can help generate accurate sound descriptions for people who are visually impaired, describing exactly what is happening in a scene with precise timing.

In short, FoleyDirector takes the "black box" of AI sound generation and opens it up, handing you the remote control so you can conduct the symphony of sound exactly how you want it to play.

Technical Summary

1. Problem Statement

Current Video-to-Audio (V2A) generation models, particularly those based on Diffusion Transformers (DiT), have achieved high audio fidelity. However, they suffer from three critical limitations regarding controllability:

• Lack of Fine-Grained Temporal Control: Existing models treat captions as coarse semantic cues. They struggle to generate sounds at specific, user-defined time intervals (e.g., "car horn at 5–6s, silence otherwise").
• Insufficient Visual Cues: Models fail when visual information is ambiguous, such as for off-screen sounds, occluded objects, or small regions where visual cues are weak or missing.
• Complex Multi-Event Scenarios: Current methods struggle to handle complex scenarios involving simultaneous in-frame and out-of-frame events, or counterfactual scenarios (e.g., a tiger yawning but making a meow sound).

2. Methodology: FoleyDirector

FoleyDirector is a framework built upon the pre-trained MMAudio generator. It introduces fine-grained temporal steering without compromising the base model's audio quality. The architecture consists of three core components:

A. Structured Temporal Scripts (STS)

Instead of a single global caption, the input is decomposed into Structured Temporal Scripts.

• Concept: The audio timeline is divided into short segments (e.g., 1-second intervals). Each segment has a specific text caption describing the sound events, loudness, and timbre for that specific window.
• Extraction Pipeline: A data annotation pipeline using Qwen-Omni (a Multimodal LLM) generates these scripts. It performs:
  1. Perception & Recognition: Generates a global semantic description and identifies sound types.
  2. Segment-level Classification: Localizes events into 1-second segments, annotating presence/absence, loudness, and timbre.
• Result: This creates the DirectorSound dataset, providing rich, temporally aligned semantic cues.

B. Script-Guided Temporal Fusion Module (SG-TFM)

To integrate STS into the DiT-based generator, the authors propose an adapter module:

• Temporal Script Attention (TSA): An additional attention layer that fuses STS features with audio features.
• Interleaved RoPE: To ensure temporal alignment between the STS tokens and audio tokens, the authors employ an Interleaved Rotary Positional Embedding (RoPE). This mechanism interleaves audio and STS tokens along the temporal dimension before applying RoPE, ensuring the model understands the temporal relationship between the script and the generated audio.
• Flexibility: The SG-TFM is an add-on adapter. Users can disable it to revert to standard V2A generation, allowing seamless switching between controlled and uncontrolled modes.

C. Bi-Frame Sound Synthesis Framework

To handle complex scenarios involving both visible (in-frame) and invisible (out-of-frame) sounds:

• Dual-Path Synthesis: The audio latent is split into two parallel streams:
  1. In-Frame Stream: Conditioned on Video ($F_v$), Text ($F_t$), and STS ($F_{tsr}$).
  2. Out-of-Frame Stream: Conditioned on a null visual embedding ($F_\emptyset$), Text, and STS. This allows the generation of sounds that are not visually present (e.g., off-screen dialogue).
• Fusion: The two streams are fused temporally according to the script segments, enabling the model to render on-screen and off-screen sounds simultaneously with precise control.

3. Key Contributions

1. First Temporal Control in DiT-based V2A: FoleyDirector is the first framework to enable precise, segment-level temporal control in DiT-based V2A generation while maintaining high audio fidelity.
2. Structured Temporal Scripts (STS): A novel representation that decomposes complex captions into short, temporally aligned segments, providing richer cues than global text.
3. Bi-Frame Synthesis: A framework that disentangles in-frame and out-of-frame sound generation, significantly improving controllability in complex, multi-event scenarios.
4. New Datasets and Benchmarks:
   • DirectorSound: A dataset constructed via an automated annotation pipeline for training.
   • DirectorBench: A benchmark for evaluating temporal and counterfactual controllability (100 videos, ~400 samples).
   • VGGSound-Director: A subset of VGGSound annotated with STS for evaluating generation quality.

4. Experimental Results

The method was evaluated against state-of-the-art baselines (MMAudio, HunyuanVideo-Foley, ThinkSound, Video-Foley).

• Controllability (DirectorBench):
  • F1-Score Improvement: The overall F1-score for temporal control increased from 0.2451 (Hunyuan-Foley) to 0.4819 (Ours).
  • Counterfactual Control: F1-score improved from 0.2331 to 0.5284.
  • Precision/Recall: Significant gains in both precision and recall for triggering sounds at exact user-specified times.
• Audio Quality (VGGSound-Director):
  • Fidelity: The method preserves the base model's quality. It achieved lower Fréchet Distance (FD) and KL-divergence compared to baselines, indicating generated audio is closer to real distributions.
  • Metrics: Improved Inception Score (ISC) from 14.38 to 14.84 and better semantic alignment (IB-score).
• Ablation Studies:
  • Removing STS drops performance significantly.
  • Interleaved RoPE improves temporal alignment.
  • Bi-Frame synthesis is crucial for handling off-screen and counterfactual events, boosting F1 on complex subsets.
• User Study: Users rated FoleyDirector significantly higher (4.53/5) on controllability and alignment compared to baselines (2.60–3.10).

5. Significance

FoleyDirector represents a paradigm shift in Video-to-Audio generation by empowering users to act as "Foley Directors."

• Practical Application: It solves the labor-intensive problem of manually syncing sound effects to video by allowing users to script sound events with precise timing and semantic control.
• Technical Advancement: It demonstrates that DiT-based models can be effectively steered with structured temporal inputs without retraining the entire backbone, preserving the high fidelity of pre-trained generators.
• Future Impact: The framework opens new possibilities for interactive storytelling, automated post-production, and generating audio for scenarios where visual cues are insufficient or intentionally misleading (counterfactuals).