💻 computer science

EchoFoley: Event-Centric Hierarchical Control for Video Grounded Creative Sound Generation

To overcome key limitations in existing video-text-to-audio models, the paper introduces EchoFoley, a new event-centric task with hierarchical control, supported by the EchoFoley-6k benchmark and the EchoVidia framework, which significantly improves both controllability and perceptual quality in video-grounded sound generation.

Original authors: Bingxuan Li, Yiming Cui, Yicheng He, Yiwei Wang, Shu Zhang, Longyin Wen, Yulei Niu

Published 2026-06-24

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Bingxuan Li, Yiming Cui, Yicheng He, Yiwei Wang, Shu Zhang, Longyin Wen, Yulei Niu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a silent movie playing on a screen. You can see a cat walking, a door slamming, and a car driving by. Now, imagine you want to add sound effects, but not just any sound. You want the cat to meow gently at first, then suddenly roar like a lion when a wizard casts a spell, and you want that specific roar to happen exactly at the 7-second mark, while making all the sounds before it louder than the sounds after it.

Current AI tools are like a clumsy sound engineer who hears "cat" and just dumps a generic "meow" sound file over the whole video. They struggle to listen to your specific, detailed instructions.

EchoFoley is a new project designed to fix this. Here is how it works, broken down into simple concepts:

1. The Problem: The "Visual Dominance" Trap

Right now, if you tell an AI, "Make the second meow sound like a lion," the AI often gets confused. It sees the cat (the visual) and thinks, "Okay, I'll just make a cat sound." It ignores your specific text instructions because it relies too heavily on what it sees rather than what you say. It's like a chef who only cooks what they see on the plate, ignoring your request to "add more salt."

2. The Solution: A "Sound Script" (Symbolic Representation)

The researchers created a new way to talk to the AI. Instead of just giving a vague command, they teach the AI to write a "Sound Script."

Think of this script like a musical conductor's score. It doesn't just say "play music"; it breaks the sound down into tiny, specific notes:

When: Exactly what second does the sound happen?
What: Is it a cat meow or a lion roar?
How: Is it loud? Is it high-pitched? Is it coming from the left or right?

By forcing the AI to write this script first, it can handle complex requests like, "Change the second meow to a lion roar, but keep the first one normal."

3. The New Playground: EchoFoley-6k

To teach the AI this new skill, the team built a massive training library called EchoFoley-6k.

Imagine a library with 6,000 silent videos.
For each video, they didn't just write one sentence; they wrote 6,000 detailed instructions and 42,000 tiny sound notes.
They hired experts to label exactly when a sound starts and stops, and what properties it should have. This is the "textbook" the AI learns from.

4. The New Brain: EchoVidia (The "Slow-Fast" Thinker)

The team built a new AI system called EchoVidia to use this library. It uses a clever trick called "Slow-Fast Thinking," inspired by how humans think:

Fast Thinking (System 1): The AI glances at the video quickly (1 frame per second) to get the general vibe. "Oh, it's a cat video."
Slow Thinking (System 2): The AI then slows the video down to a crawl (watching it in slow motion) to look closely. "Wait, I see the cat's mouth open at 00:04. That's when the meow happens. And at 00:07, the wizard wave happens."

By combining a quick overview with a slow, detailed inspection, the AI can pinpoint exactly when to put a sound and what that sound should be, rather than just guessing based on the general scene.

5. The Results: A Masterful Sound Engineer

When they tested EchoVidia against other top AI models:

Control: It was 40% better at following specific instructions. If you asked for a sound at a specific time, it actually did it.
Quality: It sounded 12% more natural and realistic to human listeners.
Balance: Unlike other models that ignored your text instructions to focus on the video, EchoVidia successfully listened to both the video and your specific commands.

In Summary

The paper introduces a new way to make AI generate sound for videos. Instead of letting the AI guess based on the picture, they gave it a detailed script and a slow-motion thinking process to ensure every sound happens at the right time, with the right tone, exactly as the user requested. It turns a clumsy, guess-and-check process into a precise, creative tool for storytelling.

Technical Summary: EchoFoley and EchoVidia

1. Problem Statement

Current Video-to-Audio (VT2A) generation models, despite recent advancements, suffer from three critical limitations that hinder fine-grained creative control:

Visual Dominance: There is an imbalance between visual and textual conditioning, where models rely heavily on visual cues and often ignore specific textual instructions.
Lack of Fine-Grained Definition: Existing approaches lack a concrete definition for controlling specific sound events. Instructions are often coarse (e.g., categorical tags like "cat meowing") rather than specifying which event to modify, when it occurs, or how its attributes (pitch, volume, timbre) should change.
Weak Instruction Following: Current datasets rely on brief tags, leading to poor adherence to complex, multi-step instructions (e.g., "change the second meow to a lion roar" or "make all preceding sounds louder").

The authors argue that creative storytelling requires the ability to manipulate sound at the event level (disentangling specific sounds) rather than just the video level, enabling operations like insertion, editing, and attribute modification of specific sounding events.

2. Methodology

A. Task Formulation: EchoFoley

The paper introduces EchoFoley (Event-Centric Hierarchical cOntrol), a new task for video-grounded sound generation.

Symbolic Representation: The core innovation is a structured tuple representation for sounding events: $e = (t, d, p)$ $e = (t, d, p)$ .
- $t = (t_{start}, t_{end})$ : Temporal location.
- $d$ : Semantic description (subject, action, object).
- $p$ : Controllable audio properties (timbre, pitch, intensity, spatialization).
Hierarchical Control: The task is organized into three levels:
- Instance Level: Controlling single events (e.g., "change the second meow").
- Group Level: Coordinating related events (e.g., "transform all meows").
- Video Level: Shaping the overall acoustic profile.
Control Types: Temporal (when/duration), Timbre (what/identity), and Volume (how strong/distant).

B. Benchmark: EchoFoley-6k

To support this task, the authors constructed EchoFoley-6k, a large-scale, expert-curated benchmark.

Scale: 6,018 video–instruction–annotation triplets and 42,000 fine-grained sounding event annotations.
Curation Pipeline: Videos are sampled from VGGSound and PE Video Dataset. The process involves motion filtering, frame captioning, LLM-based story proposal, and rigorous human modification to refine temporal boundaries and auditory attributes.
Evaluation Suite: Includes automatic metrics (Temporal IoU, CLAP similarity for timbre, relative loudness for volume) and human evaluation (Instruction Adherence, Audio-Visual Coherence, Perceptual Quality).

C. Proposed Framework: EchoVidia

To address the limitations of existing models, the authors propose EchoVidia, a training-free agentic framework featuring a Slow-Fast Thinking Strategy.

Slow-Fast Thinking: Inspired by dual-process cognition:
- Fast Thinking: Analyzes the video at 1 fps to capture high-level structure.
- Slow Thinking: Analyzes the video at 16 fps (temporally stretched) to perform detailed reasoning for precise event localization and attribute inference.
Agentic Architecture: The framework consists of three stages:
1. Reasoning: A VideoLLM-based agent identifies sounding events and estimates timing.
2. Design: The agent constructs and iteratively refines a symbolic event plan using an "action pool" of 12 atomic operations (visual reasoning, sound design, generation).
3. Synthesis: The finalized symbolic representation is passed to a sound generation module to render audio conditioned on both visual and textual contexts.

3. Key Contributions

EchoFoley Task: A new paradigm for event-centric hierarchical control in video-grounded sound generation, defined by a symbolic representation specifying when, what, and how sounds are produced.
EchoFoley-6k Benchmark: A densely annotated dataset of 6,000+ triplets and 42,000+ events, accompanied by a systematic evaluation suite for controllability and quality.
EchoVidia Framework: A training-free, slow-fast thinking agentic framework that significantly improves controllability, semantic alignment, and perceptual quality over recent VT2A baselines.

4. Experimental Results

Evaluated on EchoFoley-6k against 8 recent VT2A models (including MMAudio, ThinkSound, and HunyuanVideo-Foley):

Controllability: EchoVidia outperforms the strongest baseline by 40.7% in controllability.
- Temporal Control: 0.72 (vs. ~0.43 baseline).
- Timbre Control: 0.78 (vs. ~0.48 baseline).
- Volume Control: 0.75 (vs. ~0.69 baseline).
Perceptual Quality: EchoVidia achieves a 12.5% improvement in perceptual quality scores.
Instruction Adherence: Human evaluation shows EchoVidia attains 3.80/5.0 on Instruction Adherence, significantly higher than baselines (which often score <2.60), effectively eliminating the "Visual Dominance Bias" where models ignore text instructions.
Event Awareness: The Slow-Fast (SF) strategy significantly boosts sounding event detection and localization. For example, on the Gemini-2.5 Pro model, SF increased Recall from 0.66 to 0.83 and IoU for localization from 0.51 to 0.84.

5. Significance and Claims

The paper claims that EchoFoley and EchoVidia represent a shift from coarse, video-level prompting to fine-grained, event-level control. By introducing a symbolic representation and a slow-fast reasoning strategy, the work enables:

Disentangled Control: The ability to isolate and modify specific sounds within a complex scene without affecting others.
Faithful Instruction Following: A reduction in visual dominance, allowing models to execute complex, multi-attribute edits (e.g., changing pitch, volume, and timing simultaneously) as requested by the user.
Creative Storytelling: The framework supports the synthesis of "story-shaped imagination," where sound is not just a background layer but an active narrative element that can be edited to alter the perceived story (e.g., turning a cat's meow into a lion's roar to create a magical narrative).

The authors position this work as a step toward "omni-modal generative intelligence" capable of understanding and recreating the multimodal richness of the real world with high fidelity and controllability.