Technical Summary: Don't Pause: Streaming Video-Language Synchrony for Online Video Understanding

1. Problem Statement

Current Online Video Large Language Models (Video-LLMs) face a critical limitation in streaming scenarios: they typically pause video perception while generating full-sentence responses. This "pause-and-decode" approach breaks real-time video-language synchrony, causing stutters and preventing the model from "perceiving while speaking." While existing methods (e.g., response gating, End-of-Sequence token prediction, and verification-based methods) attempt to manage proactive responses, they generally interleave full sentences within visual streams rather than interleaving individual word tokens with incoming frames. This fails to achieve the human-like capability of fluidly integrating continuous sensory input with speech output.

The paper identifies two core challenges:

1. Concurrent Perception and Generation: How to achieve fine-grained, frame-level interleaving of video frames and word tokens without blocking perception for the duration of a full sentence.
2. Dynamic Response Arbitration: How to dynamically decide whether to continue an incomplete utterance, trigger a new response, or remain silent as the visual context evolves, particularly in scenarios with rapidly changing events (e.g., sports) versus sparse events.

2. Methodology: LyraV

The authors propose LyraV, a live streaming assistant built upon a hierarchical control framework wrapped around a frozen online Video-LLM backbone. LyraV does not retrain the backbone but introduces two core modules to establish a new paradigm called Streaming Video-Language Synchrony (SVLS).

A. Streaming Video-Language Synchrony (SVLS) Paradigm

Unlike traditional offline processing (entire video $\to$ caption) or existing online methods (frame $\to$ full sentence), SVLS outputs a small chunk of tokens ($m$) per frame interval. This ensures perception is never blocked for a full sentence, allowing the model to emit sub-budget token chunks within each frame's time budget.

B. Frame-Driven Transition Controller (FDTC)

The FDTC is a training-free, verification-based finite-state machine that governs high-level semantic decisions (when to speak). It operates in three states:

• Triggered State: Initiated when a new response is needed. The model generates the first $m$ tokens of a new utterance.
• Continuing State: The core innovation. If the current frame allows the fluent continuation of the existing utterance buffer (verified via perplexity), the model remains in this state, decoding additional tokens. This prevents the model from restarting sentences unnecessarily.
• Silent State: Entered when an utterance is complete (EOS token) and no new event requires a response.

Mechanism: The controller uses a verification mechanism to compute the Perplexity (PPL) of the current response prefix given the evolving visual context. A stable or decreasing PPL maintains the Continuing State, while a sharp increase (semantic drift) triggers a transition to Triggered State to start a new response.

C. Streaming Token Pacer (SToP)

The SToP is a trainable, lightweight predictive module that governs low-level pacing (how fast to speak).

• Function: It dynamically adapts the number of tokens emitted per frame ($m_i$) to match the semantic density of the visual content and real-time latency constraints.
• Architecture: A lightweight Transformer encoder that takes a multimodal history (visual embeddings and token count categories) to predict an optimal token count distribution.
• Latency Constraint: The predicted token count is capped by a hard latency cutoff to ensure the total processing time (perception + generation) does not exceed the frame interval ($\Delta t$), preventing buffering delays.

3. Key Contributions

1. SVLS Paradigm: The introduction of a new paradigm where Video-LLMs achieve concurrent streaming perception and generation, seamlessly interleaving video frames and word tokens without long pauses.
2. LyraV Framework: A plug-and-play control layer featuring:
   • FDTC: A stateful controller that dynamically arbitrates between utterance continuation and new response triggering based on visual context, enabling the "Continuing State" for smooth narration.
   • SToP: An adaptive pacing module that regulates token emission rates to match visual event density while adhering to strict real-time latency budgets.
3. Empirical Observations: The authors observe an emergent capability of dynamic reasoning over streaming tokens, where the model incrementally refines its narration as visual evidence accumulates, rather than committing to a fixed description from a single trigger frame.

4. Experimental Results

Experiments were conducted on 8 benchmarks (5 online, 3 offline).

• Streaming Synchrony: LyraV achieves a 98.29% Sync Rate (SR) in the untruncated setting, significantly outperforming baselines like LiveStar (78.93%). In the truncated (zero-latency) setting, LyraV achieves the highest Semantic Score (SS: 3.62) and Narrative Fluency (NF: 4.03), demonstrating that its control strategy produces higher-quality content under strict time constraints compared to models that simply truncate generation.
• Real-Time Performance: LyraV processes video at 3.89 FPS (real-time), maintaining high fluidity.
• General Understanding: On offline benchmarks (MVBench, LongVideoBench, VideoMME), LyraV preserves the backbone's general video understanding capabilities, showing no significant degradation compared to the frozen backbone (LiveStar).
• Streaming QA: On benchmarks like OVO-Bench and StreamingBench, LyraV achieves state-of-the-art results among online models (e.g., 50.97 on OVO-Bench), slightly outperforming its backbone and other proactive models.
• Ablation Studies: Removing FDTC causes a drastic drop in performance (SS -34.4%, NF -55.4%), confirming the critical role of the Continuing State. Removing SToP leads to minor declines, indicating that while pacing improves fluidity, the hard latency cutoff alone can maintain basic quality.

5. Significance and Claims

The paper positions LyraV not as a model that improves static, single-answer accuracy, but as a system that fundamentally rethinks video-language interaction for streaming.

• Synchrony over Static Accuracy: The primary contribution is the ability to maintain "speak-while-watching" synchrony and narrative fluency under real-time constraints, a capability absent in current Video-LLMs.
• Incremental Refinement: The authors claim LyraV enables a form of "dynamic reasoning" where the model continuously updates its interpretation of the video stream, offering potential advantages in low-latency inference and error tolerance.
• Modest Scope: The authors explicitly state that their improvements are in the synchrony paradigm and control framework. They do not claim to have improved the backbone's intrinsic visual perception or static reasoning capabilities; rather, they have optimized how these capabilities are deployed in a streaming environment to prevent perceptual blocking and stuttering.

The work concludes that by decoupling high-level semantic decisions (FDTC) from low-level pacing (SToP), it is possible to achieve human-like, seamless video-language interaction where perception is never paused for a full sentence.