💬 NLP

Real-Time Generation of Game Video Commentary with Multimodal LLMs: Pause-Aware Decoding Approaches

This paper proposes two pause-aware, fine-tuning-free decoding strategies for multimodal large language models to generate real-time game video commentary, demonstrating that a novel dynamic interval-based approach outperforms fixed-interval methods in aligning both content and timing with human utterances.

Original authors: Anum Afzal, Yuki Saito, Hiroya Takamura, Katsuhito Sudoh, Shinnosuke Takamichi, Graham Neubig, Florian Matthes, Tatsuya Ishigaki

Published 2026-03-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Anum Afzal, Yuki Saito, Hiroya Takamura, Katsuhito Sudoh, Shinnosuke Takamichi, Graham Neubig, Florian Matthes, Tatsuya Ishigaki

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a thrilling car race or a high-speed video game fight. Usually, a human commentator shouts out what's happening: "Look at that overtake!" or "He's about to get knocked out!"

This paper asks a big question: Can we teach an AI to be that commentator in real-time, without needing a human to teach it every single rule?

Here is the breakdown of their work, explained with some everyday analogies.

The Problem: The "Chatterbox" AI

In the past, if you asked an AI to describe a video, it would often act like a nervous chatterbox.

The Issue: It would talk too much, too fast, or at the wrong times.
The Analogy: Imagine a tour guide who never stops talking. Even when you are just looking at a beautiful view, they are shouting, "Look at the tree! Look at the rock! Look at the tree again!" It's exhausting and hard to follow.
The Technical Term: Existing AI models are great at knowing what to say, but they are terrible at knowing when to say it. They don't know when to take a breath (pause).

The Solution: Teaching the AI to "Wait"

The researchers wanted to see if they could use a "Multimodal Large Language Model" (an AI that can see images and read text) to act like a pro commentator just by giving it instructions (prompts), without retraining it from scratch.

They tested two different ways to tell the AI when to speak:

1. The "Metronome" Approach (Fixed Interval)

How it works: The AI checks the video every 2 seconds, like a clock ticking.
The Analogy: Imagine a drummer tapping a stick on a table every 2 seconds. Every time the stick hits, the AI has to say something.
The Result: If the AI says a long sentence, the next "tap" comes too soon. The AI starts talking over itself, creating a messy wall of text. It's like a drummer who keeps beating the drum even while the singer is still finishing a verse.

2. The "Breath-Based" Approach (Dynamic Interval)

How it works: The AI looks at how long the last sentence it spoke was. If it said a long sentence, it waits longer before checking the video again. If it said a short one, it checks sooner.
The Analogy: This is like a human speaker. When you tell a long story, you naturally pause to let the audience catch up. You don't start the next sentence until you've finished the current one. The AI calculates: "I just said 10 words, so I need to wait 2 seconds before I look at the video again."
The Result: This creates a natural rhythm. The AI knows when to speak and when to be silent, just like a real person.

The Experiment: Racing and Fighting Games

The researchers tested this on two types of games:

Car Racing: Fast, but the action is somewhat predictable.
Fighting Games: Extremely fast, chaotic, and requires quick reactions.

They used both English and Japanese to see if the AI could handle different languages.

The Findings: Timing is Everything

The "Metronome" (Fixed) method was okay at describing what happened, but it was a mess regarding when it happened. It talked too much and too fast.
The "Breath-Based" (Dynamic) method was the winner.
- Human Annotators (people who watched the videos) preferred the Dynamic method. They felt it sounded more natural and knew when to shut up during boring moments.
- The "Pause" Factor: The Dynamic AI was much better at identifying "Key Events" (like a crash or a knockout) and staying silent when nothing important was happening.

Why This Matters

This is a big deal because:

No Heavy Training: They didn't have to spend months teaching the AI specific rules. They just gave it a smart instruction (a prompt) and let it figure it out.
Accessibility: This technology could help blind people "see" a video game or sports match through audio commentary, or help non-native speakers understand what's happening in a foreign language stream.
Natural Flow: It proves that AI doesn't have to be a robot that talks non-stop. It can learn to have a "rhythm" and respect the audience's need for silence.

The Bottom Line

The paper shows that if you want an AI to be a good commentator, you don't just need to teach it what to say; you have to teach it when to say it. By letting the AI decide its own schedule based on how long it just spoke, it becomes a much more natural, human-like narrator.

In short: They taught the AI to stop being a chatterbox and start being a storyteller who knows how to pause for effect.

1. Problem Statement

Real-time video commentary generation aims to provide textual or spoken descriptions of ongoing events in videos (e.g., sports, esports) to enhance accessibility and engagement. While recent Multimodal Large Language Models (MLLMs) have demonstrated strong capabilities in generating content via prompting, they largely ignore the critical aspect of timing.

Existing approaches typically suffer from two main limitations:

Fixed-Input Assumptions: Most prior work assumes fixed-length video clips and generates a single utterance per clip, failing to address the continuous, real-time nature of video streams.
Lack of Temporal Control: Current methods do not explicitly solve the "when to speak" problem. They often generate commentary at arbitrary intervals or rely on heavy fine-tuning of streaming-specific models (e.g., LiveCC, VideoLLM-online), which limits their flexibility and generalizability.

The core research question is: Can general-purpose MLLMs, using prompting alone (without fine-tuning), manage both what to say and when to say it to achieve natural, pause-aware real-time commentary?

2. Methodology

The authors propose a framework that treats real-time commentary as a causal sequence generation task. The model receives a stream of video frames and a history of past utterances, deciding at each step whether to generate text or output a <WAIT> token.

Core Strategies

The paper introduces two prompting-based decoding strategies:

A. Fixed-Interval Decoding (Baseline)

Mechanism: The model is queried at uniform time intervals ( $N$ seconds).
Variants:
- Stateless: Takes only the current video clip.
- Feedback: Includes a list of previously generated utterances in the prompt.
- Feedback (ICL): Uses In-Context Learning with few-shot examples of utterance-video pairs.
Limitation: If the interval is too short, the system struggles with inference latency; if too long, or if generated text is long, it leads to rapid, consecutive updates that are cognitively difficult for users to follow.

B. Dynamic Interval-Based Decoding (Proposed "Realtime" Strategy)

Mechanism: Inspired by "WAIT/WRITE" policies in simultaneous machine translation, this strategy dynamically adjusts the timing of the next prediction based on the estimated duration of the previous utterance.
Process:
1. After generating an utterance, the system estimates its speaking duration ( $\hat{d}$ ) based on word count and a fixed speech rate ( $r$ ).
2. The next prediction time ( $t_{i+1}$ ) is scheduled immediately after this estimated delay.
3. The input video segment length is adjusted dynamically to cover the time elapsed since the last utterance.
Advantage: This creates a feedback loop where the model receives context about what has changed since the last comment, allowing it to decide whether to speak or remain silent. It naturally aligns with human reading/speaking speeds without requiring architectural changes or fine-tuning.

Implementation Details

Models: Evaluated using GPT-4.1 (API) and open-source models (LLaVA-NeXT-Video, Qwen2.5-VL-Instruct).
Prompting: A unified prompt structure includes a role description, a history of past utterances paired with video frames, and an instruction to generate 1–2 lines or output <WAIT> if no significant change occurred.
Datasets: A multilingual benchmark comprising:
- Racing Games: English and Japanese datasets (aligned with recorded car races).
- Fighting Games: Japanese dataset (Super Smash Bros. Ultimate, known for fast-paced commentary).

3. Key Contributions

Pause-Aware Decoding Strategies: Proposed two methods for real-time generation, with a novel dynamic interval-based approach that schedules utterances based on estimated speech duration, enabling natural pacing without fine-tuning.
Demonstration of Prompting Efficacy: Showed that MLLMs can effectively manage both content generation and temporal alignment using prompting alone, offering a lightweight alternative to supervised streaming models.
Multilingual Benchmark: Released a benchmark dataset across three domains (Racing EN, Racing JP, Fighting JP) to support standardized evaluation of pause-aware language generation.
Evaluation Framework: Established a comprehensive evaluation protocol combining automatic metrics (Alignment, ROUGE, BERTScore) with subjective human evaluation focusing on timing and naturalness.

4. Results

Automatic Evaluation

Timing Alignment: Surprisingly, the Fixed-Interval with ICL approach achieved the highest correlation scores with human timestamps in automatic metrics across all datasets.
Content Similarity: ROUGE-L and BERTScore scores were generally low. The authors attribute this to the high verbosity of MLLM outputs compared to human references and significant lexical variation even when the semantic meaning is correct.
Observation: Automatic metrics proved unreliable for capturing the nuance of "timing appropriateness" in this context.

Subjective (Human) Evaluation

Human annotators evaluated the outputs on four criteria: Key Event Identification (KEI), Pause-awareness, Coherence, and Naturalness.

Dynamic Interval Superiority: The Dynamic Interval (Realtime) strategy significantly outperformed fixed-interval methods in Pause-awareness and Naturalness.
- Example: For Japanese racing commentary, Realtime achieved a Pause-awareness score of 3.50 (vs. lower scores for fixed methods).
- Example: For fighting games, Realtime achieved a Naturalness score of 3.93.
Model Performance: GPT-4.1 consistently outperformed open-source models. However, even open-source models (like Qwen2.5) showed significant improvements in timing when using the Dynamic strategy over fixed intervals.
Verdict: While automatic metrics favored fixed intervals, human evaluators strongly preferred the dynamic approach for its ability to avoid unnecessary verbosity and align with natural speech rhythms.

5. Significance and Conclusion

This paper demonstrates that prompting-based MLLMs can achieve effective real-time video commentary without the need for domain-specific fine-tuning or complex streaming architectures.

Key Insight: The "when to speak" problem is best solved by dynamically adjusting the inference schedule based on the output of the previous step (utterance duration), rather than querying at fixed time intervals.
Impact: This approach offers a flexible, "plug-and-play" solution for real-time accessibility in gaming and sports, capable of integrating with downstream systems like text-to-speech or subtitles.
Future Work: The authors suggest improving timing estimation robustness across different languages and genres, and developing evaluation metrics that better correlate with human perception of timing.

In summary, the study validates that dynamic, pause-aware decoding transforms general-purpose MLLMs into effective real-time commentators, bridging the gap between static video captioning and live, engaging narration.