🤖 AI

LiveStarPro: Proactive Streaming Video Understanding with Hierarchical Memory for Long-Horizon Streams

This paper introduces LiveStarPro, a proactive live streaming assistant that overcomes the limitations of existing Video-LLMs in handling long-horizon streams through a novel architecture combining Streaming Verification Decoding for autonomous response timing, Streaming Causal Attention Masks for incremental alignment, and Tree-Structured Hierarchical Memory for efficient long-term recall, all validated by the new OmniStarPro benchmark.

Original authors: Zhenyu Yang, Kairui Zhang, Bing Wang, Shengsheng Qian, Changsheng Xu

Published 2026-06-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhenyu Yang, Kairui Zhang, Bing Wang, Shengsheng Qian, Changsheng Xu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to have a conversation with a friend who is watching a live, hour-long video stream with you. You want them to tell you what's happening, but you don't want them to talk constantly. You only want them to speak when something important happens, and you want them to remember what happened an hour ago if you ask about it later.

Current AI assistants are terrible at this. They either talk non-stop (boring and redundant), forget everything after a few minutes, or get confused about when to speak.

LiveStarPro is a new AI assistant designed specifically to solve these problems. Think of it as a "Proactive Streaming Video Understanding" system. Here is how it works, broken down into three simple parts using everyday analogies:

1. The "Silence Detector" (Streaming Verification Decoding)

The Problem: Old AI models tried to learn a special "silence token" (like a secret word meaning "I'm not talking right now"). This was like teaching a dog to bark only when you say "Sit," but the dog kept barking at everything else. It led to a massive imbalance: the AI had to learn to be silent 99% of the time and talk only 1% of the time, which confused the training.

The LiveStarPro Solution: Instead of learning to be silent, LiveStarPro uses a confidence check.

The Analogy: Imagine you are describing a movie scene to a friend. You say, "A man is walking." A second later, the man is still walking. You don't need to say, "A man is still walking" again.
How it works: LiveStarPro checks its own previous description against the new video frame. If the new frame fits perfectly with what it just said (low "perplexity" or confusion), it stays silent. If the scene changes significantly (high confusion), it knows it's time to speak and updates the description. It decides when to talk based on whether the story has actually changed, not by guessing a silence token.

2. The "Smart Training" (Streaming Causal Attention Masks)

The Problem: To teach an AI to do this, you can't just show it a whole movie and ask for a summary. You have to teach it to watch frame-by-frame, just like a human does. Standard training methods often let the AI "cheat" by peeking at the answer for the next second before it actually sees the frame.

The LiveStarPro Solution: They created a special training method called SCAM.

The Analogy: Think of it like a "blindfolded" practice session. The AI is shown a frame and asked to describe it, but it is strictly forbidden from looking at the description it wrote for the previous frame to copy the answer. It has to rely only on the visual evidence it sees right now and the history it has already built.
The Result: This forces the AI to learn a genuine, step-by-step understanding of the video, making it ready for the real-time "confidence check" mentioned above.

3. The "Tree-Like Memory" (Tree-Structured Hierarchical Memory)

The Problem: Humans have short-term memory (what happened 5 minutes ago) and long-term memory (what happened last week). Old AI assistants have a tiny "sticky note" (a sliding window). Once the note is full, they throw away the oldest stuff to make room for the new. If you ask, "What did that person pick up an hour ago?", the AI has no idea because it threw that "sticky note" away.

The LiveStarPro Solution: They built a Tree-Structured Hierarchical Memory (TSHM).

The Analogy: Imagine a library instead of a sticky note.
- Short-Term (The Desk): The AI keeps the most recent, detailed frames on its desk. When the desk gets too full, it doesn't throw things away; it summarizes them. It keeps the "Peak" moments (the most exciting parts) and the "End" moments (the summary of the event), then clears the rest.
- Long-Term (The Bookshelves): The summarized events are filed away on shelves. But instead of a messy pile, they are organized into a Tree. If a new event is similar to an old one, it gets attached as a "child" branch to that old event.
- Retrieval: When you ask a question, the AI doesn't search the whole library. It walks down the tree branches, quickly finding the relevant "event chain." This allows it to remember things from hours ago without getting overwhelmed.

The Result: OmniStarPro

To prove this works, the researchers didn't just test it on short clips. They built a massive new benchmark called OmniStarPro.

It includes 15 different real-world scenarios (like sports, news, and gaming).
It includes videos that are hours long.
It tests if the AI can remember details from 30+ minutes ago.

The Bottom Line:
LiveStarPro is the first AI that can watch a live, hour-long video, decide exactly when to speak (avoiding boring repetition), and remember what happened an hour ago if you ask. In tests, it was 28.9% better at understanding the meaning of the video and 18.2% more accurate in timing its responses compared to previous models, all while running fast enough to keep up with a live stream.

Technical Summary: LiveStarPro

Problem Statement

Current Video Large Language Models (Video-LLMs) have achieved significant success in offline settings but struggle with online, live streaming scenarios. Existing online architectures face three primary challenges:

Proactive Response Timing: Models must process continuous frame-by-frame inputs and autonomously decide when to respond. Current methods often rely on explicit "silence" tokens (End-of-Sequence or EOS) to mark non-response intervals. This approach suffers from severe data imbalance (silence frames vastly outnumber response frames), temporal inconsistency (contradictory targets for adjacent frames), and vocabulary degradation (the EOS token contaminates the semantic space).
Long-Horizon Memory: Continuous streams eventually exceed the fixed context window of LLMs. Standard strategies like sliding windows or First-In-First-Out (FIFO) eviction lead to catastrophic forgetting, where historical events are permanently discarded, making it impossible to answer queries about distant past events (e.g., "What happened an hour ago?").
Evaluation Limitations: Existing benchmarks are often restricted to short-horizon video question answering or rely on first-person perspectives (Ego4D), failing to capture the diversity of real-world streaming applications or the need for long-term memory recall.

Methodology

The authors propose LiveStarPro, a proactive live streaming assistant designed for long-horizon video understanding. The framework integrates three complementary components:

1. Inference: Streaming Verification Decoding (SVeD)

SVeD replaces the traditional "generate-then-compare" or EOS-prediction paradigm with a verify-then-generate approach.

Mechanism: Instead of generating a silence token, the model treats silence as a verification state. It computes the perplexity (PPL) of the current caption given the newly arrived visual frame.
Decision Logic: If the PPL of the existing caption remains stable (below a scaled threshold $\alpha$ ), the model suppresses generation (silence) and logically swaps the caption to the end of the context buffer to reflect time passage. If the PPL spikes (indicating a semantic divergence between the visual input and the current description), the model activates to generate a new caption.
Efficiency: This requires only a single forward pass for verification, eliminating the need for full autoregressive generation for every frame. A Streaming Key-Value (KV) Cache is employed to handle dynamic context updates (swaps and appends) without recomputing historical representations.

2. Training: Streaming Causal Attention Masks (SCAM)

To support the dynamic verification logic of SVeD, the training objective is reformulated to align with incremental video-language processing.

Interleaved Sequences: Data is organized as interleaved frame-caption sequences.
Causal Masking: SCAM enforces a selective visibility policy. Within a semantic clip (a sequence of frames sharing the same event description), the model is masked from attending to previously generated captions for the same clip. This forces the model to rely on the current visual features and accumulated history rather than copying previous text.
Goal: This ensures the model learns to generate captions grounded in visual evidence, providing a robust probability distribution for the SVeD verification stage.

3. Memory: Tree-Structured Hierarchical Memory (TSHM)

To address the unbounded nature of live streams, TSHM organizes memory into two tiers:

Short-Term Working Memory (Peak-End Compression): The active context is compressed using the Peak-End rule. Frames with high semantic divergence (Peaks) and the summary captions of semantic clips (Ends) are retained, while redundant frames are pruned based on a dynamic threshold.
Long-Term Retrieval Memory (Recursive Event Tree): Evicted history is not discarded but stored in a recursive event tree. New events are attached as children to semantically similar existing nodes (if similarity exceeds a threshold $\sigma$ ) or form new branches. Parent node embeddings are updated via momentum aggregation to represent the semantic centroid of their subtrees.
Retrieval: During generation, a hierarchical beam descent retrieves relevant event chains. This allows for sublinear retrieval complexity ( $O(k \cdot b \cdot \log_b n)$ ) compared to the linear scan of flat memory banks, enabling efficient access to long-term context without saturating the active window.

4. Benchmark: OmniStarPro

To facilitate rigorous evaluation, the authors introduce OmniStarPro, a large-scale dataset covering 15 diverse real-world scenarios. It includes:

OmniStarPro-Live: Short-horizon tasks (Real-time Narration, Temporal Grounding, Dense QA, etc.) focusing on instantaneous response timing.
OmniStarPro-Long: A long-form partition (10 mins to >1 hour) with three memory-centric tasks (Long-range Memory Recall, Cross-event Difference Query, Temporal Backtracking) specifically designed to test recall of evicted historical content.

Key Results

Extensive experiments demonstrate that LiveStarPro outperforms existing online Video-LLMs and offline baselines:

Performance Gains: LiveStarPro achieves a 28.9% improvement in semantic correctness and an 18.2% reduction in timing error compared to prior online methods.
Efficiency: The streaming KV cache yields a 1.58× inference speedup over the same model without caching, maintaining a throughput of approximately 3 FPS on hour-long streams.
Long-Term Recall: On the OmniStarPro-Long partition, LiveStarPro significantly outperforms sliding-window baselines and flat memory banks in long-bucket memory spans (>30 minutes), retaining a recall accuracy of 37.2% compared to 21.3% for flat k-NN retrieval.
Offline Capability: The model maintains strong performance on conventional offline benchmarks (MVBench, LongVideoBench, VideoMME), indicating that the streaming-specific training does not severely degrade general video understanding capabilities.

Significance and Claims

The paper claims that LiveStarPro establishes a new paradigm for proactive online video understanding. By shifting the focus from predicting silence tokens to verifying relevance via perplexity, and by organizing long-term memory into a structured, retrievable tree, the work addresses the fundamental limitations of current online assistants regarding timing, memory retention, and context consistency.

The authors assert that their work lays the groundwork for robust, scalable models capable of handling effectively unbounded video streams, bridging the gap between offline video analysis and real-time, interactive streaming assistants. The release of the OmniStarPro benchmark and the LiveStarPro codebase is intended to facilitate further research in long-horizon streaming understanding.