🤖 AI

Response-G1: Explicit Scene Graph Modeling for Proactive Streaming Video Understanding

Response-G1 is a novel, fine-tuning-free framework that enhances proactive streaming video understanding by explicitly modeling the alignment between accumulated video evidence and query conditions through scene graphs, thereby enabling more accurate and interpretable response timing decisions.

Original authors: Ke Ma, Jiaqi Tang, Bin Guo, Xueting Han, Ruonan Xu, Qingfeng He, Ziheng Wang, Xu Wang, Qifeng Chen, Zhiwen Yu, Yunhao Liu

Published 2026-05-11

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Ke Ma, Jiaqi Tang, Bin Guo, Xueting Han, Ruonan Xu, Qingfeng He, Ziheng Wang, Xu Wang, Qifeng Chen, Zhiwen Yu, Yunhao Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a live sports game with a friend who asks, "Who is going to score the next goal?"

In the old way of doing things (which the paper calls "reactive"), your friend asks the question, and you immediately shout out an answer based only on what you've seen up to that exact second. If the goal hasn't happened yet, you might guess wrong or say something silly because you didn't wait for the right moment.

Other newer methods try to be smarter by waiting, but they often do it blindly. They might just look for big changes in the video (like a sudden loud noise or a fast movement) to decide when to speak. This is like a security guard who only sounds the alarm when the door slams, ignoring the fact that someone was quietly picking the lock for ten minutes.

Response-G1 is a new system that changes the game. It acts like a super-organized detective who doesn't just watch the video; they draw a map of what's happening and compare it to the question.

Here is how it works, broken down into three simple steps:

1. The "Sketchbook" (Scene Graph Generation)

Instead of just watching the video frame-by-frame, Response-G1 constantly draws a quick "sketch" of the scene. It doesn't write long paragraphs; it creates a simple list of connections, like:

Woman is holding a mirror.
Man is standing near the woman.

Crucially, it only draws the parts of the sketch that matter to the question. If the question is about a "kiss," it ignores the background scenery and focuses entirely on the people and their actions.

2. The "Memory Bank" (Retrieval)

As the video plays, Response-G1 keeps a stack of these sketches in a memory bank. When the user asks a question, the system doesn't just look at the current second. It flips through its stack of past sketches to find the ones that match the "conditions" needed to answer the question.

Think of it like this: If the question is "When will the woman kiss the man?", the system keeps checking its stack of sketches. It sees sketches of them standing near each other, then holding hands, but it stays silent. It's waiting for the specific sketch that says "Woman kissing Man."

3. The "Trigger" (Decision Time)

This is the magic part. The system compares the "Question Sketch" (what it expects to see to answer the question) with the "Video Sketches" it has collected so far.

If they don't match yet: The system stays silent. It knows it doesn't have enough evidence.
If they match: The system says, "Aha! I have the proof!" and finally gives the answer.

Why is this better?

The paper claims that by using these structured "sketches" (called Scene Graphs) instead of just guessing or waiting for random changes, the system becomes much better at two things:

Timing: It knows exactly when to speak and when to stay quiet, avoiding premature guesses.
Accuracy: Because it is looking at specific connections (like "holding" or "kissing") rather than just blurry video pixels, it understands the story better.

The researchers tested this on video benchmarks and found that Response-G1 is better at answering questions about future events (proactive) and describing current events (reactive) than other top systems, all without needing to be re-trained on new data. It's like giving the AI a better set of tools to organize its thoughts before it speaks.

Technical Summary: Response-G1

Problem Statement

The paper addresses the challenge of Proactive Streaming Video Understanding (SVU). Unlike reactive interaction, where models respond immediately to user queries, proactive interaction requires Video Large Language Models (Video-LLMs) to autonomously determine the optimal moment to respond as a video unfolds. The core difficulty lies in deciding when the currently accumulated visual evidence is sufficient to satisfy the conditions implied by a user's query, particularly for predictive or anticipatory questions where the answer may not be available in the initial frames.

Existing approaches suffer from two primary limitations:

Fine-tuning-based methods: These rely on frame-level annotations to train decision boundaries (e.g., predicting End-Of-Sequence tokens or using auxiliary classifiers). However, they often struggle with inconsistent labels on visually similar consecutive frames, hindering the learning of reliable decision boundaries.
Fine-tuning-free methods: Current approaches often rely on implicit modeling of visual evidence or simple inter-frame difference thresholds. These methods fail to explicitly align the accumulated streaming evidence with the specific semantic conditions of the query, leading to suboptimal response timing.

Methodology: Response-G1

The authors propose Response-G1, a novel, fine-tuning-free framework that establishes explicit, structured alignment between accumulated video evidence and query conditions using scene graphs. The framework operates through three integrated stages:

1. Online Query-Guided Scene Graph Generation

Instead of processing raw frames or implicit features, the system abstracts streaming video clips into structured scene graphs.

Mechanism: A Video-LLM is prompted to generate a scene graph $G_t$ for a video clip centered at time $t$ .
Query Guidance: The generation is conditioned on the user query $Q$ . The prompt steers the model to prioritize triplets (object-predicate-object) that are relevant to the query, suppressing irrelevant visual details.
Representation: The graph consists of nodes (objects and attributes) and edges (spatio-temporal relations), represented as a collection of triplets.

2. Memory-Based Scene Graph Retrieval

To facilitate fine-grained alignment, the system maintains a dynamic memory bank of historical scene graphs.

Linearization & Embedding: Each scene graph is linearized into natural language phrases (e.g., "woman in red") and encoded into dense text embeddings using the Video-LLM's text encoder. Similarly, the user query is parsed into a "Query Condition Graph" and linearized to ensure format consistency.
Similarity Matching: The system computes cosine similarity between the query embedding and the embeddings of all historical scene graphs in the memory bank.
Retrieval: The top- $K$ most semantically relevant historical scene graphs are retrieved to serve as the evidence context ( $G^{ctx}_t$ ).

3. Retrieval-Augmented Streaming Pipeline

The retrieved scene graphs are integrated into the Video-LLM's inference process to drive decision-making.

Trigger Phase (Silence/Response): At each time step, the Video-LLM receives a token sequence containing the current frame embeddings, the retrieved scene graphs (encoded with temporal timestamps), and a trigger prompt (e.g., "Should I answer now?"). The model outputs a binary decision: Silence or Response.
Response Phase: If the decision is to respond, the model generates the final answer using the accumulated visual context up to that moment, augmented by the retrieved scene graphs.
Versatility: The same pipeline can be adapted for reactive tasks by setting the response time to the query arrival time, demonstrating the framework's ability to enhance spatio-temporal grounding even without proactive delays.

Key Contributions

Explicit Scene Graph Modeling: The paper introduces a paradigm shift from implicit evidence modeling to explicit, structured representation using scene graphs. This disentangles underlying semantics and supports principled reasoning over the alignment of evidence and query conditions.
Response-G1 Framework: An end-to-end, fine-tuning-free architecture that seamlessly integrates online query-guided scene graph generation, memory-based graph retrieval, and retrieval-augmented streaming decision-making.
State-of-the-Art Performance: The method achieves superior results on both proactive and reactive streaming video understanding tasks without requiring task-specific fine-tuning.

Experimental Results

The authors evaluated Response-G1 on two established benchmarks: OVO-Bench and StreamingBench.

Proactive Tasks:
- On the Forward Active Responding subtask of OVO-Bench (measuring response timing accuracy), Response-G1 outperformed the second-best open-source streaming Video-LLM by 12.8% on average.
- On the Proactive Output (PO) subtask of StreamingBench, it surpassed the second-best open-source model by 15.1%.
- The performance was comparable to proprietary MLLMs (e.g., GPT-4o, Gemini 1.5 Pro).
Reactive Tasks:
- The framework also improved performance on standard reactive subtasks (Real-Time Visual Perception, Backward Tracing, Real-Time Visual Understanding), with gains ranging from 2.1% to 12.3% over the second-best open-source baselines.
Ablation Studies:
- Removing retrieval augmentation or timestamp encoding significantly degraded performance, particularly on tasks requiring temporal grounding (e.g., Clues Reveal Responding).
- Query-guided generation proved superior to object-guided generation, as the latter led to hallucinations and premature responses.
- Using graph-text embeddings for retrieval yielded better results than using raw query text, highlighting the importance of format consistency.

Significance and Claims

The paper claims that Response-G1 demonstrates the advantage of explicit structural modeling and retrieval augmentation in streaming video understanding. By grounding both visual evidence and query conditions in a shared graph representation, the framework enables Video-LLMs to make more interpretable and accurate decisions regarding response timing.

The authors position this work as a step toward more capable and interpretable multimodal interaction in real-world streaming applications, such as conversational AI assistants and autonomous embodied agents. They acknowledge limitations, noting that while scene graphs improve alignment, they may not fully address complex reasoning (e.g., "why" questions) and that the trade-off between query relevance and hallucination in open-set generation remains a challenge. Future work is suggested to explore richer structural formulations and adaptive generation mechanisms.