Echoes Over Time: Unlocking Length Generalization in Video-to-Audio Generation Models

This paper introduces MMHNet, a multimodal hierarchical network incorporating non-causal Mamba that enables video-to-audio models trained on short clips to effectively generalize and generate high-quality audio sequences exceeding five minutes in duration.

The Gist

Imagine you are trying to teach a robot to be a movie sound designer. Your goal is to show the robot a silent video clip and have it generate the perfect background noise, dialogue, and sound effects to match what's happening on screen.

This is the challenge of Video-to-Audio (V2A) generation. While we have gotten really good at making short soundtracks (like 8 seconds of a dog barking), making a continuous, high-quality soundtrack for a 5-minute movie scene has been incredibly difficult. The robot usually gets confused, the audio sounds robotic, or the sounds stop matching the video halfway through.

The paper "Echoes Over Time" introduces a new system called MMHNet that solves this problem. Here is how it works, explained through simple analogies.

The Problem: The "Short-Term Memory" Robot

Most current AI models are like students who only study for 8-second pop quizzes.

• The Issue: If you ask them to write a 5-minute essay (generate 5 minutes of audio), they panic. They forget what happened at the beginning, they start repeating themselves, or they lose track of the plot.
• The Old Way (Transformers): Traditional models use "positional embeddings." Think of this like giving the robot a numbered list of instructions (1, 2, 3...). If the list goes up to 100, the robot knows exactly where it is. But if you suddenly ask it to write a list up to 1,000, it gets lost because it was never taught numbers that high. It tries to guess, and the result is messy.

The Solution: MMHNet (The "Smart Conductor")

The authors built a new system, MMHNet, which acts like a smart orchestra conductor rather than a rigid robot. It uses three main tricks to handle long videos:

1. The "Non-Causal" Ear (Listening to the Whole Room)

Most audio models are "causal," meaning they can only hear what happened before the current moment, like listening to a radio show live.

• The Analogy: Imagine trying to understand a conversation in a noisy room. If you can only hear what people said 5 seconds ago, you might miss the context of what they are saying right now.
• The Fix: MMHNet uses a technology called Non-Causal Mamba. This is like giving the conductor a surveillance camera that shows the entire room at once. It can see the whole video scene simultaneously. It doesn't have to guess what comes next; it knows the whole context, so the audio stays consistent from start to finish, even for 5 minutes.

2. The "Hierarchical" Filter (The VIP Pass)

Long videos are full of boring, repetitive moments (like a car driving down a straight road for 30 seconds). Processing every single frame and sound is a waste of energy.

• The Analogy: Imagine a bouncer at a club. Instead of letting every single person in the crowd (every single data point) into the VIP area (the main brain of the AI), the bouncer checks their ID.
• The Fix: MMHNet uses Token Routing. It acts as a smart filter.
  • Temporal Routing: It asks, "Is there a loud sound happening right now?" If it's just silence or background hum, it skips it.
  • Multimodal Routing: It asks, "Does this sound match the video?" If a car crashes on screen, it prioritizes the crash sound. If the video is just a static landscape, it ignores the audio data that doesn't match.
  • Result: The AI only processes the "VIP" moments that matter, saving energy and keeping the quality high for the long haul.

3. The "Compressed" Brain

Instead of trying to remember every single detail of a 5-hour movie, the AI compresses the information into a summary.

• The Analogy: Think of reading a book. You don't memorize every single letter; you remember the story.
• The Fix: The model processes the video in a "compressed space." It understands the essence of the scene (e.g., "a busy market") rather than getting bogged down in the pixel-by-pixel details. This allows it to stretch a short training memory (8 seconds) into a long, coherent performance (5 minutes) without breaking.

The Result: "Train Short, Test Long"

The most impressive part of this paper is the magic trick:

• They trained the AI only on 8-second clips.
• They tested it on 5-minute videos.
• The Outcome: The AI didn't just "guess." It generated high-quality, synchronized audio that lasted for minutes, beating all previous models that tried to do the same thing.

Why This Matters

Before this, if you wanted to make a 5-minute video with AI sound, you had to chop it into tiny 8-second pieces, generate sound for each, and try to glue them together. The result was usually choppy and sounded like a broken record.

MMHNet is like a musician who, after practicing scales for 8 seconds, can suddenly play a full, complex symphony without missing a beat. It opens the door for AI to generate soundtracks for full movies, long video games, and documentaries, making the digital world feel much more real and immersive.

Technical Summary

1. Problem Statement

The paper addresses the critical challenge of Length Generalization in Video-to-Audio (V2A) generation.

• The Limitation: Existing state-of-the-art V2A models (e.g., diffusion-based or autoregressive transformers) are primarily trained on short clips (typically 8–10 seconds). When tested on long-form videos (e.g., 1 to 5 minutes), these models fail to generate coherent, high-quality audio.
• Root Causes:
  • Positional Embeddings: Transformer-based models rely on explicit positional encodings (e.g., RoPE). These encodings are fixed during training and do not generalize well to sequence lengths unseen during training, leading to degraded performance and temporal drift.
  • Memory & Context: Modeling extended audio sequences requires substantial memory, and standard attention mechanisms struggle to maintain global context over long durations.
  • Fragmentation: Current workarounds, such as generating audio in short segments and stitching them together, result in disjointed transitions, unaligned sound events, and poor audio quality.
• Goal: The authors aim to train models on short, fixed-length data (8 seconds) but enable them to generate high-quality, contextually aligned audio for long videos (up to 5+ minutes) without retraining on long data.

2. Methodology: MMHNet

The authors propose MMHNet (Multimodal Hierarchical Network), a novel framework designed to overcome the limitations of Transformers for long-form generation.

A. Core Architecture: Non-Causal Mamba-2

Instead of Transformers, MMHNet utilizes Mamba-2, a State Space Model (SSM) variant.

• No Positional Embeddings: Mamba-2 processes sequences without explicit positional encodings, allowing it to naturally generalize to variable and longer sequence lengths.
• Non-Causal Design: Unlike standard causal Mamba (which processes tokens sequentially), MMHNet employs Non-Causal Mamba-2.
  • Reasoning: Video conditions are available offline (non-causal). Non-causal processing allows for omnidirectional information flow, enabling the model to capture global dependencies and align modalities simultaneously without the "modulation decay" issues found in causal models over long sequences.
• Flow Matching: The generative process is based on Conditional Flow Matching, operating in a latent space to ensure efficiency.

B. Hierarchical Framework & Routing

To handle the computational cost of long sequences and multimodal alignment, MMHNet introduces a hierarchical structure with dynamic routing:

1. Compressed Space: The model operates in a compressed token space during early layers to reduce redundancy.
2. Temporal Routing: Identifies and selects tokens corresponding to specific timeframes where sound events occur, filtering out redundant silence or static periods.
3. Multimodal (MM) Routing: Selects tokens based on high similarity between modalities (e.g., aligning audio features with visual/text features). This ensures that only the most relevant cross-modal information is passed to the main network.
4. Chunking/Dechunking: The architecture uses downsampling (chunking) to compress inputs and upsampling (dechunking) to reconstruct detailed outputs, leveraging a Straight-Through Estimator (STE) for gradient flow.

C. Multimodal Conditioning

The model integrates diverse inputs via:

• Global Conditioning: Adaptive Layer Normalization (adaLN) injects pooled visual and text features.
• Token-Level Conditioning: Uses semantic video representations (CLIP), motion-audio synchronized features (Synchformer), and text embeddings to ensure precise local alignment.

3. Key Contributions

1. Length Generalization Challenge: The paper formally defines and tackles the "Train Short, Test Long" problem in V2A, demonstrating that models can generalize to 5-minute videos when trained only on 8-second clips.
2. MMHNet Architecture: Introduction of a multimodal hierarchical network combining Non-Causal Mamba-2 with hierarchical token routing. This eliminates the need for positional embeddings and handles long-range dependencies effectively.
3. New Benchmark & Evaluation: The authors introduce a rigorous evaluation protocol using the UnAV100 and LongVale datasets, specifically testing performance on variable-length audio (10s to 500s) where prior methods fail.

4. Experimental Results

The authors evaluated MMHNet against state-of-the-art baselines (MMAudio, LoVA, V-AURA, HunyuanVideo-Foley) on UnAV100 and LongVale.

• Performance Metrics:
  • Distribution Matching: MMHNet achieved significantly lower Fréchet Distance (FD) scores (e.g., 1.80 vs. 3.36 for LoVA on UnAV100), indicating generated audio is statistically closer to real audio.
  • Multimodal Alignment: It achieved the highest IB-Score (36.27 on UnAV100), surpassing the previous best (HunyuanVideo-Foley at 32.90) by a large margin.
  • Temporal Synchronization: MMHNet showed the lowest DeSync scores (0.410), proving superior alignment between video events and generated sound.
• Ablation Studies:
  • Architecture: Non-Causal Mamba-2 significantly outperformed both Transformers (without positional embeddings) and Causal Mamba-2.
  • Hierarchy: The hierarchical routing approach yielded better alignment and lower FD scores compared to non-hierarchical baselines.
  • Duration Robustness: While baselines (like MMAudio) saw performance drop sharply as video duration increased from 10s to 60s, MMHNet maintained consistent performance across all durations.
• Long-Form Capability: The model successfully generated coherent audio for videos exceeding 5 minutes, a capability where prior methods (like LoVA) resulted in noticeable degradation or failure.

5. Significance

This work represents a paradigm shift in multimodal generative AI:

• Breaking the Length Barrier: It proves that high-quality, long-form audio generation is possible without training on massive long-duration datasets, solving a major data scarcity bottleneck.
• Architectural Innovation: By replacing Transformers with Non-Causal Mamba-2, the paper demonstrates a viable path for long-sequence generation that avoids the pitfalls of positional encoding extrapolation.
• Practical Application: The ability to generate synchronized soundscapes for films, games, and long-form content directly from silent video opens new avenues for automated sound design and accessibility.

In summary, MMHNet successfully unlocks length generalization in V2A tasks by leveraging the inherent properties of State Space Models and a hierarchical routing strategy, setting a new standard for long-form audio generation.