💬 NLP

Low-Latency Real-Time Audio Game Commentary System via LLM-Based Parallel Text Generation

This paper presents a low-latency real-time audio game commentary system that overcomes sequential pipeline bottlenecks by running text generation in parallel with speech playback and buffering candidate utterances, thereby reducing inter-utterance silence from 9.6 to 0.3 seconds and significantly improving the perceived naturalness of the commentary rhythm.

Original authors: Ryota Kawamatsu, Anum Afzal, Yuki Saito, Shinnosuke Takamichi, Graham Neubig, Katsuhito Sudoh, Hiroya Takamura, Tatsuya Ishigaki

Published 2026-06-12

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Ryota Kawamatsu, Anum Afzal, Yuki Saito, Shinnosuke Takamichi, Graham Neubig, Katsuhito Sudoh, Hiroya Takamura, Tatsuya Ishigaki

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a high-speed video game stream, like a frantic soccer match or a fast-paced fighting game. Usually, a human commentator shouts out what's happening: "He jumps! He hits the ball! Goal!" But if you were to replace that human with a robot, the old way of doing it would feel like a broken walkie-talkie.

Here is the problem the paper solves, explained simply:

The Old Way: The "Wait-and-See" Robot

Think of the old system like a very polite, but slow, tour guide.

The guide watches the game for a few seconds.
They stop watching to write a sentence about what they saw.
They stop writing to speak the sentence out loud.
Crucially: They do not start writing the next sentence until they have finished speaking the current one.

In a fast game, this creates awkward, dead air. The guide finishes speaking, then stands there in silence for 10 seconds while they frantically write the next sentence. By the time they speak again, the game has moved on, and the silence feels unnatural and boring.

The New System: The "Assembly Line" Robot

The researchers built a new system that acts more like a well-oiled factory assembly line or a relay race team.

Instead of waiting to finish speaking before thinking, the robot does two things at once:

Speaker: It is currently talking about what just happened.
Thinker: While the speaker is talking, the "thinker" is already watching the next few seconds of the game and writing down multiple possible sentences for the future.

These future sentences are stored in a "buffer" (like a waiting room). As soon as the speaker finishes the current sentence, the system instantly grabs the next pre-written sentence from the waiting room and starts speaking it immediately. There is no gap. No silence. Just a smooth, continuous flow of commentary.

The "Video Delay" Trick

You might ask: "If the robot is thinking ahead, won't it talk about things before they happen on the screen?"

To fix this, the system plays a clever trick with the video itself. It intentionally delays the video stream by a tiny fraction of a second (like holding your breath for a moment). This gives the robot just enough time to finish its "thinking" and "writing" before the video catches up. It's like a conductor slightly slowing down the orchestra so the singer can stay perfectly in sync.

What They Found

The researchers tested this on fast-paced games (specifically Super Smash Bros. Ultimate) and compared their new "Assembly Line" system against the old "Wait-and-See" system.

Silence: The old system had an average gap of 9.6 seconds between sentences. The new system reduced this to just 0.3 seconds. That's the difference between a awkward pause and a natural conversation.
Human Feel: When 120 experienced gamers listened to the results, they rated the new system's rhythm as much more natural and professional. It felt like a real person was talking, not a computer struggling to keep up.
Content: The system didn't just talk faster; it also matched the timing of professional human commentators much better than the old method.

The Bottom Line

This paper presents a system that makes AI game commentary feel alive by letting the AI "think ahead" while it speaks, rather than waiting in silence. By running the thinking and speaking processes in parallel and slightly delaying the video, they eliminated the awkward pauses that make automated commentary feel robotic.

Technical Summary: Low-Latency Real-Time Audio Game Commentary System via LLM-Based Parallel Text Generation

Problem Statement

The paper addresses a critical bottleneck in end-to-end real-time audio game commentary systems: accumulated latency leading to unnatural silence. In conventional pipelines, the process is strictly sequential: the system captures video frames, generates text via a Large Language Model (LLM), synthesizes speech via Text-to-Speech (TTS), and only then initiates the generation of the next utterance after the current speech playback concludes.

This sequential dependency causes significant waiting times between utterances. In fast-paced gaming scenarios, this results in long, unnatural silences (averaging 9.6 seconds in baselines) that degrade the immersive experience and disrupt the natural rhythm of commentary. Furthermore, extending existing systems from structured game data to raw gameplay video using modern multimodal LLMs exacerbates these latency challenges.

Methodology

The authors propose a system design that decouples text generation from speech playback through parallel processing and implements video delay control to maintain temporal consistency.

1. Parallel Text Generation

Unlike traditional systems that wait for speech playback to finish before generating the next segment, the proposed system initiates text generation immediately upon the availability of a new video segment, even while the current utterance is still being spoken.

Buffering: Multiple candidate utterances are generated in the background and stored in a buffer.
Selection Policy: When the current speech playback ends, the system immediately selects a buffered candidate to synthesize, eliminating idle time. The paper evaluates three lightweight selection policies to avoid decision latency:
- Latest: Selects the most recently generated segment.
- Oldest: Selects the earliest buffered segment.
- Random: Selects a segment randomly.

2. Video Delay Control

To mitigate the misalignment between generated speech and visual events caused by inevitable inference latency, the system intentionally delays the outgoing video stream.

In a live streaming setup (e.g., YouTube Live), the server buffers the video stream and begins playback only when the first utterance is ready. This aligns the visual feed with the audio commentary, ensuring that the commentary matches the displayed game state.

3. Experimental Setup

Input: Live gameplay video from Super Smash Bros. Ultimate (captured at 25 fps), grouped into segments of $N=32$ frames.
Models: GPT-4.1-mini for text generation and a standard TTS module.
Baselines:
- After-Audio: Fully sequential (waits for speech to end).
- After-Text: Semi-sequential (waits for text generation to end, but not speech).
Evaluation: Quantitative metrics (silence duration, mean Intersection over Union [mIoU] of speaking/silence patterns, ROUGE scores) and a user study with 120 experienced gamers rating rhythm, alignment, and overall quality.

Key Results

The experiments demonstrate that the parallel architecture significantly outperforms sequential baselines:

Silence Reduction: The mean inter-utterance silence was reduced from 9.6 seconds (After-Audio baseline) to 0.3 seconds (Parallel Oldest/Random policies).
Pattern Similarity: The proposed methods achieved an mIoU of approximately 0.60, closely matching human commentary patterns, compared to near-zero scores (0.01–0.10) for sequential baselines. This indicates a >40% improvement in similarity to professional speaking-silence timing.
User Perception: In a study with 120 participants, the proposed methods received significantly higher Mean Opinion Scores (MOS) across all criteria:
- Commentary Rhythm: Scores rose from ~1.9 (baselines) to ~3.5–3.6 (parallel methods).
- Video-Audio Alignment: Improved from ~2.4 to ~3.1.
- Overall Quality: Improved from ~1.9 to ~3.2.
Token Length: The study found that varying the max new tokens parameter affects speaking duration and text length, with 60 tokens yielding utterance lengths closest to human commentary, while 40 tokens best matched human speaking duration.

Significance and Claims

The paper claims that its primary contribution is a low-latency system design that effectively solves the silence accumulation problem inherent in sequential real-time commentary pipelines. By running text generation in parallel with speech playback, the system enables immediate synthesis at playback boundaries, creating a natural speaking rhythm indistinguishable from human commentary in terms of timing.

The authors emphasize that this approach allows for the use of modern, heavy multimodal LLMs in real-time settings without the prohibitive latency previously associated with them. The work bridges the gap between language generation and speech synthesis research, demonstrating that parallel processing and lightweight video delay control are sufficient to achieve high-quality, immersive automated commentary for fast-paced video games. The system is presented as a practical solution for live streaming platforms, validated by both objective metrics and subjective user feedback.