TripleSumm: Adaptive Triple-Modality Fusion for Video Summarization

This paper introduces TripleSumm, an adaptive triple-modality fusion architecture that dynamically weights visual, text, and audio features at the frame level to achieve state-of-the-art video summarization, alongside the release of MoSu, the first large-scale benchmark providing all three modalities.

The Gist

Imagine you have a 2-hour movie, a 30-minute cooking tutorial, and a 1-hour sports highlight reel. You want to create a 2-minute "best of" clip for each, but you don't have time to watch them all. This is the problem of Video Summarization.

For a long time, computers tried to solve this by only looking at the pictures (the visual frames). It's like trying to understand a movie by looking at a slideshow of still photos while wearing noise-canceling headphones and blindfolded to the subtitles. You might see a person running, but you won't know why they are running (the audio of a siren) or what they are saying (the text/subtitles).

This paper introduces TripleSumm, a new AI system that solves this by acting like a super-smart film editor who can watch, listen, and read simultaneously, deciding in real-time which sense is most important for every single second of the video.

Here is a breakdown of how it works, using simple analogies:

1. The Problem: The "One-Size-Fits-All" Editor

Previous AI editors were rigid. They treated every video the same way.

• The Flaw: They might prioritize the visuals even when the audio is the most important part.
• The Analogy: Imagine a film editor who is obsessed with the camera angles. If a character is screaming in terror, this editor might cut the scene because the camera angle is "boring," completely missing the fact that the scream (audio) is the most important part of the story. They fail to realize that sometimes the text (subtitles) matters most, sometimes the music, and sometimes the visuals.

2. The Solution: The "Adaptive Triple-Editor" (TripleSumm)

The authors built a new AI that doesn't just watch; it listens and reads too. It uses three "senses":

• Visual: What is happening on screen?
• Text: What are people saying or what do the subtitles say?
• Audio: What sounds are happening? (Music, screams, engines, silence).

The Magic Trick: The "Smart Switch"
The core innovation is that TripleSumm doesn't just mix these three senses together into a smoothie. Instead, it acts like a traffic controller at a busy intersection.

• At Scene A (a judge speaking on a talent show), the traffic controller points all the attention to the Text/Audio lane because the words matter most.
• At Scene B (a robot dancing), the controller points attention to the Visual/Audio lane because the movement and music matter most.
• At Scene C (a chaotic explosion), it uses all three.

It changes its mind frame-by-frame (every 1/30th of a second). It's dynamic, not static.

3. The New "Training Ground" (MoSu Dataset)

To teach this AI, you need a massive library of videos where someone has already marked the "best parts."

• The Old Problem: Previous libraries were tiny (like a shoebox of 50 videos) or only had pictures.
• The New Library (MoSu): The authors created MoSu (Most Replayed Multimodal Video Summarization).
  • Size: It's huge—over 52,000 videos (nearly 4,000 hours of content).
  • The Secret Sauce: They used "Most Replayed" statistics from YouTube. Think of it like this: If thousands of people rewind a specific 5-second clip of a cat falling off a table, that clip is objectively "important." The AI learns from these collective human choices.
  • Completeness: Every video in this library has the picture, the sound, and the text/transcript perfectly synced.

4. How It Works Under the Hood (The "Refine-and-Fuse" Strategy)

The paper describes two main "rooms" in the AI's brain:

1. The Time-Travel Room (Multi-scale Temporal Block): This room looks at the video over different time spans. It looks at the immediate next frame (to catch a quick blink) and also looks at the whole movie (to understand the plot). It's like reading a book: sometimes you focus on a single word, sometimes on a paragraph, and sometimes on the whole chapter.
2. The Fusion Room (Cross-modal Fusion Block): This is where the "Smart Switch" lives. It takes the information from the Time-Travel room and asks: "Right now, is the audio more important than the video? Should I boost the volume or the brightness?" It learns to weigh the three senses dynamically.

5. The Results: Why It Matters

• Better Summaries: TripleSumm creates summaries that humans actually prefer. It captures the "soul" of the video better than previous methods because it doesn't ignore the audio or text.
• Efficiency: Despite being smarter, it is actually lighter and faster than previous heavy-duty models. It's like upgrading from a gas-guzzling truck to a high-performance electric sports car.
• Robustness: Even if you mute the video or remove the subtitles, TripleSumm can still make a good summary because it knows how to rely on the remaining senses.

Summary

TripleSumm is a new AI film editor that stops treating video as just a sequence of pictures. Instead, it acts like a human viewer, dynamically shifting its focus between what it sees, hears, and reads to decide what is truly important in every split second. To teach it, the authors built the MoSu dataset, a massive library of videos that serves as the ultimate training ground for this new generation of smart summarizers.

The Bottom Line: It's the difference between a robot that just watches a video and a robot that actually understands the story.

Technical Summary

1. Problem Statement

Video summarization aims to extract key segments from long videos to represent their content efficiently. However, existing approaches face two critical limitations:

• Static/Modality-Agnostic Fusion: Most current methods rely solely on visual features or employ static fusion strategies (e.g., standard self-attention or fixed cross-attention) for multimodal data. They fail to account for the dynamic, frame-dependent variations in modality saliency. For instance, in a music audition, text (speech) may be most informative at one moment, while visual-audio cues dominate at another.
• Data Scarcity: Research into multimodal video summarization is bottlenecked by the lack of large-scale, comprehensive benchmarks. Existing datasets (e.g., SumMe, TVSum) are small and unimodal, while larger datasets (e.g., Mr. HiSum) often lack audio or text modalities, preventing the training of robust trimodal models.

2. Methodology: TripleSumm

The authors propose TripleSumm, a novel architecture designed to adaptively weight and fuse visual, text, and audio modalities at the frame level. The model is robust to missing modalities and dynamically adjusts to the most informative cue for each frame.

A. Input Representation

• Multimodal Encoding: Raw video streams are synchronized to $N$ time steps.
  • Visual ($V$): Encoded using pre-trained image encoders (e.g., CLIP).
  • Text ($T$): Encoded using text encoders (e.g., RoBERTa) on transcripts.
  • Audio ($A$): Encoded using audio encoders (e.g., AST) on spectrograms.
• Projection: Features are projected into a common embedding space ($D$) via linear layers and Layer Normalization.
• Fusion Tokens: A unique Fusion Token ($E_f$) is introduced for each time step, aggregating the three modalities (via averaging or learnable MLP) to serve as an unbiased anchor for cross-modal attention.
• Positional Encoding: Temporal positional encoding and learnable modality embeddings are added to distinguish time steps and modality origins.

B. Core Architecture

The model employs a hierarchical "refine-and-fuse" strategy with $L$ interleaved layers, each containing:

1. Multi-scale Temporal (MST) Block:
   • Uses Windowed Self-Attention (WSA) with varying window sizes ($w$) to capture temporal dependencies.
   • Multi-scale Design: Lower layers use small windows to capture fine-grained local dependencies, while higher layers expand windows to capture long-range global context.
   • Efficiency: Reduces complexity from $O(N^2)$ to $O(w \cdot N)$.
   • Parameter Sharing: The same MST block parameters are shared across all modalities to learn common temporal patterns efficiently.
2. Cross-modal Fusion (CMF) Block:
   • Operates independently at each time step.
   • Uses the Fusion Token as the Query ($Q$) and the modality-specific tokens ($V, T, A$) as Keys ($K$) and Values ($V$).
   • This mechanism allows the model to dynamically attend to the most relevant modality for a specific frame without bias toward any single modality.

C. Training and Inference

• Loss Function: Minimizes the squared $L_2$ loss between predicted frame importance scores and ground-truth scores.
• Summary Generation: Frames are grouped into temporally coherent shots (via KTS). The model selects shots that maximize total importance scores under a fixed length budget (e.g., 15% of original duration), formulated as a 0/1 knapsack problem.

3. Key Contributions

1. TripleSumm Architecture:

   • The first model to adaptively fuse visual, text, and audio modalities at the frame level.
   • Introduces a Fusion Token mechanism that learns to prioritize modalities dynamically, avoiding the bias of treating modalities uniformly or prioritizing visuals.
   • Demonstrates robustness: The model generates reasonable summaries even when one or more modalities are missing.

2. MoSu Dataset (Most Replayed Multimodal Video Summarization):

   • The first large-scale (52,678 videos, ~4,000 hours) benchmark providing trimodal features (Visual, Text, Audio) and importance annotations.
   • Derived from YouTube-8M using "Most Replayed" statistics as a proxy for human attention, filtered for quality (English transcripts, >50k views, >120s duration).
   • Covers 3,406 categories clustered into 10 thematic groups (e.g., Video Games, Music, Sports).

3. State-of-the-Art Performance:

   • Achieves significant improvements over existing methods on four benchmarks (MoSu, Mr. HiSum, SumMe, TVSum).
   • Demonstrates high parameter efficiency (1.37M parameters) compared to heavy baselines.

4. Experimental Results

• Performance on MoSu: TripleSumm achieves SOTA with a Kendall's $\tau$ of 0.351 and Spearman's $\rho$ of 0.472, significantly outperforming the next best multimodal baseline (CFSum: $\tau=0.277$).
• Generalization:
  • On Mr. HiSum (visual-only ground truth), the full model outperforms visual-only versions, proving that non-visual cues provide crucial context even when not explicitly annotated in the ground truth.
  • On SumMe/TVSum (human-annotated), the model achieves SOTA, validating its ability to learn synergistic relationships between modalities.
• Zero-Shot Long-Form Video: Tested on 50 unseen videos averaging 70.4 minutes in length. TripleSumm significantly outperforms all baselines, demonstrating superior scalability to complex, long-form narratives.
• Efficiency: With only 1.37M parameters and 0.97 GFLOPs, it is significantly lighter than competitors (e.g., CSTA has 10.56M params) while maintaining faster inference times (2.81ms).

5. Significance and Impact

• Paradigm Shift: Moves video summarization from static, visual-centric models to adaptive, dynamic multimodal fusion. It proves that the "best" modality changes frame-by-frame and must be weighted accordingly.
• Benchmarking: The release of MoSu addresses a critical data bottleneck, enabling future research into robust multimodal learning for video understanding.
• Real-World Applicability: The model's ability to handle missing modalities and generalize to extremely long videos makes it suitable for real-world applications like automated content curation for streaming platforms.
• Efficiency: Demonstrates that high performance in video summarization does not require massive computational resources, making the technology more accessible.

In conclusion, TripleSumm establishes a new standard for video summarization by effectively integrating the temporal dynamics of video with the complementary nature of audio and text, supported by a comprehensive new dataset and rigorous evaluation.