Proact-VL: A Proactive VideoLLM for Real-Time AI Companions

This paper introduces Proact-VL, a general framework designed to transform multimodal language models into proactive, real-time AI companions that overcome latency and decision-making challenges, validated through the new Live Gaming Benchmark across commentary and guidance scenarios.

The Gist

Imagine you are watching a live video game stream. Usually, you have two choices: either a human commentator talks over the game, or you play alone with no one to talk to.

Proact-VL is a new kind of AI companion designed to be the perfect "third wheel" in this scenario. It's not just a robot that answers questions when you ask; it's a proactive friend who knows when to speak, what to say, and how long to talk, all in real-time.

Here is a simple breakdown of how it works, using some everyday analogies:

1. The Problem: The "Over-talker" vs. The "Silent Ghost"

Current AI video assistants are like two extremes:

• The Over-talker: Some AI models are like a nervous guest at a party who never stops talking. They describe every single frame, even when nothing interesting is happening. This annoys the viewer.
• The Silent Ghost: Other models are like a shy guest who only speaks when you explicitly ask them a question. If you don't ask, they say nothing, even if a massive explosion just happened on screen.

Proact-VL solves this by acting like a skilled sports commentator. They know exactly when to shout "GOAL!" and when to stay silent so you can hear the crowd. They don't just react; they anticipate.

2. The Three Superpowers of Proact-VL

A. The "Chunk" Strategy (The 1-Second Snapshots)

Most AI tries to watch a whole movie at once, which is too slow for live games. Proact-VL treats the video like a flipbook.

• It looks at the screen for just one second (a "chunk").
• It decides instantly: "Is this second exciting? Should I talk?"
• If yes, it spits out a short, punchy sentence. If no, it stays silent and waits for the next second.
• Analogy: Imagine a photographer taking a photo every second. Instead of waiting to develop the whole album, they show you the photo immediately and decide if it's worth printing.

B. The "Traffic Light" (Deciding When to Speak)

This is the most unique part. The AI has a built-in traffic light inside its brain.

• Red Light: Nothing interesting is happening. The AI stays silent.
• Green Light: A boss is defeated, a player is in danger, or a user asks a question. The AI turns green and starts talking.
• The Magic: It learns this timing by watching thousands of hours of human streamers. It learns that humans don't talk constantly; they talk at the right moments.

C. The "Two-Mode" Personality

The paper tested Proact-VL in two specific roles, like an actor changing costumes:

1. The Hype Man (Commentator): In games like League of Legends or Cyberpunk 2077, it acts like a co-commentator. It can chat with other AI commentators, taking turns so they don't talk over each other. It knows when to let the human speak and when to jump in with a joke or a strategy tip.
2. The Coach (Guide): In games like Minecraft or Elden Ring, it acts as a helpful tutor. If you are stuck in a cave, it doesn't just say "You are in a cave." It says, "Hey, look at that lava! Pour water there to make obsidian so you don't burn." It gives advice before you make a mistake.

3. The "Live Gaming" Dataset (The Training Camp)

To teach this AI, the researchers didn't just use textbooks. They built a massive training camp called the "Live Gaming Dataset."

• They collected 561 hours of real gameplay from 12 different popular games (from Minecraft to Street Fighter).
• They used advanced tools to clean up the audio, removing background music and noise, so the AI could learn exactly what the human commentators were saying and when they said it.
• Analogy: Instead of teaching a student by reading a book about swimming, they threw them into the pool with the Olympic champions and said, "Watch how they move, then try to copy them."

4. Why It Matters (The Result)

The tests showed that Proact-VL is the best at balancing speed and quality.

• Speed: It reacts almost instantly (low latency), so it feels like a real-time conversation, not a delayed video call.
• Quality: It doesn't ramble. It speaks only when necessary, making the experience feel natural and human-like.

Summary

Think of Proact-VL as the ultimate AI co-pilot for gamers.

• Old AI was like a GPS that only speaks when you tell it to ("Recalculating...").
• Proact-VL is like a passenger who knows the road, points out the cool scenery, warns you about the potholes, and knows exactly when to shut up so you can focus on driving.

It's a step closer to having an AI friend that doesn't just understand what you see, but understands how you feel about what you see, and knows exactly when to join the conversation.

Technical Summary

1. Problem Statement

The paper addresses the critical gap in creating human-like AI companions for real-time interactive scenarios (specifically gaming commentary and user guidance). While recent Video Large Language Models (VideoLLMs) can understand video, they struggle with three specific challenges required for proactive companionship:

1. Low-Latency Inference: Handling continuous streaming inputs without significant delay.
2. Autonomous Decision Making: Determining when to speak versus when to remain silent (avoiding excessive talking or awkward silence).
3. Content Control: Generating short, high-quality, clip-level responses that align with real-time constraints, rather than long, offline-style answers.

Existing approaches fall into two categories: Proactive models (which decide when to speak but generate long, high-latency answers) and Real-time models (which generate fast but lack control over speaking behavior, often leading to "over-talking"). Proact-VL aims to bridge this gap.

2. Methodology

A. The Live Gaming Dataset and Benchmark

To support training and evaluation, the authors constructed a large-scale dataset and benchmark:

• Data Collection: 561 hours of high-quality English commentary across 12 popular games (e.g., League of Legends, Minecraft, Black Myth: Wukong, Cyberpunk 2077).
• Scenarios: The data covers three interaction modes:
  1. Solo Commentary: Autonomous narrative flow.
  2. Co-Commentary: Multi-agent social coordination.
  3. Real-time User Guidance: Goal-directed engagement (answering player queries).
• Processing Pipeline:
  • Commentary: Uses WhisperX for ASR, Qwen3-Omni for paralinguistic labeling (pauses, laughter), and DeepSeek-V3.2 for domain-specific polishing (correcting game terminology).
  • Guidance: Segments videos into 5-minute clips, uses Qwen3-VL to identify queries and generate visual descriptions, and refines them into coach-style guidance via GPT-4.1.
  • Persona Enrichment: Extracts structured personas (tone, vocabulary, rhythm) to ensure consistent role-playing.

B. Proact-VL Framework

The core architecture integrates chunk-wise processing with a lightweight proactive mechanism:

1. Chunk-Wise Input Schema:

   • Video streams are discretized into fixed-duration chunks (1 second).
   • At each time step $t$, the model receives a triplet: Visual tokens ($V_t$), Optional User Query ($Q_t$), and Environmental Context/History ($B_t$).
   • It uses a persistent KV-cache to maintain temporal context across chunks, enabling continuous dialogue history.

2. Proactive Response Mechanism (Decide-Then-Generate):

   • A special <|FLAG|> token is appended to the end of the user message.
   • The model extracts the hidden state of this token and passes it through a lightweight gated MLP followed by a sigmoid activation to compute a speaking probability ($p_t$).
   • Decision: If $p_t \geq \tau$ (threshold), the model triggers generation; otherwise, it outputs a silence token.
   • Generation: If triggered, it appends an assistant prefix and generates a short, clip-level text response.

3. Training Strategy (Multi-Tier Loss):

   • Causal Language Modeling Loss ($L_{main}$): Supervises the quality of the generated text.
   • Response Loss ($L_{resp}$): Supervises the speaking behavior.
     • Transition-Smoothed Classification Loss ($L_{cls}$): Treats speaking/silence as a sequence learning problem. It applies higher weights to state transitions (switching from silence to speak or vice versa) to handle class imbalance.
     • Stability Regularization ($L_{reg}$): Enforces local temporal consistency (smooth probability transitions) and global speaking-rate constraints (matching the average speaking rate of human commentators).
   • Total Objective: $L = L_{main} + \alpha L_{resp}$.

4. Infinite Inference:

   • To handle unbounded streams, the system uses a dual-cache sliding-window KV-cache. When the context limit is reached, the oldest portion of the streaming cache is evicted, and a reverse-RoPE correction is applied to realign positional encodings, preventing drift.

3. Key Contributions

1. Live Gaming Dataset & Benchmark: A large-scale, multi-scenario dataset (561h) and benchmark specifically designed for evaluating proactive, real-time AI companions in gaming.
2. Proact-VL Framework: A novel architecture combining chunk-wise processing, a lightweight decision head (<|FLAG|>), and specialized training objectives to balance proactivity timing with content quality.
3. Superior Performance: Demonstrated that the model achieves state-of-the-art performance in both response timing (proactivity) and text quality, outperforming both commercial closed-source models and existing open-source proactive/real-time baselines.

4. Experimental Results

The model was evaluated on the Live Gaming Benchmark across Solo Commentary, Co-Commentary, and Guidance scenarios.

• Text Quality:

  • Proact-VL achieved the highest scores in Closed Captions (CC) (win-rate against Gemini 2.5 Pro) and LiveU (LLM-judged streaming quality).
  • It significantly outperformed other proactive models (e.g., VideoLLM-online, MMDuet) and competitive real-time models (e.g., LiveCC).
  • In Co-Commentary, it showed a 51.46% win rate, surpassing GPT-4o (46.72%) and LiveCC-Instruct (37.40%).

• Response Quality (Proactivity):

  • F1 Score: Proact-VL achieved the best overall F1 (64.87%), indicating superior balance between precision and recall in triggering responses.
  • TimeDiff: It maintained low temporal latency (1.71s overall), closely matching human timing.
  • PAUC: It showed strong cumulative performance in capturing key moments.

• Long-Horizon Stability:

  • In the Live Gaming Benchmark-Streaming (30min to 2h videos), Proact-VL maintained stable text quality and response timing, showing only mild degradation over time compared to baselines.

• Generalization:

  • The model performed robustly on out-of-domain tasks (e.g., Black Myth: Wukong and Ego4D), demonstrating strong generalization capabilities beyond the training distribution.

5. Significance and Impact

• Paradigm Shift: Proact-VL moves beyond "reactive" AI (waiting for a prompt) to "proactive" AI that mimics human conversational rhythms, deciding when to speak based on visual and contextual cues.
• Practical Application: The framework is directly applicable to live esports commentary, educational game streaming, and real-time customer support, where low latency and appropriate speaking volume are critical.
• Safety & Ethics: The authors emphasize the cleaning of training data to prevent misinformation and bias, laying a foundation for responsible deployment of AI companions.
• Technical Advancement: The introduction of the <|FLAG|> token and the transition-aware loss function provides a new blueprint for training models to manage temporal dynamics in streaming video tasks.

In conclusion, Proact-VL represents a significant step forward in making AI companions feel truly "present" and interactive in real-time video environments, successfully balancing the trade-off between latency, proactivity, and content quality.