💻 computer science

ObjectStream: Latent Objects as Memory Anchors for Streaming Video Understanding

ObjectStream is a training-free framework that enhances streaming video understanding by organizing frozen Video-LLM representations into persistent latent object anchors, enabling efficient, high-performance reasoning on object histories and interactions while significantly reducing memory usage and token counts.

Original authors: Mingkang Dong, Muxin Pu, Jie Li, Bohan Guo, Songruo Chen, Bin Ren, Xu Zheng, Chen Zhao, Tianwen Qian, Mohamed Elhoseiny, Yuqian Fu

Published 2026-07-31

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Mingkang Dong, Muxin Pu, Jie Li, Bohan Guo, Songruo Chen, Bin Ren, Xu Zheng, Chen Zhao, Tianwen Qian, Mohamed Elhoseiny, Yuqian Fu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a live stream of a busy city street. Cars zoom by, people cross the road, and a bird lands on a lamppost. Now, imagine you are a robot trying to answer questions about this street, but you can only remember a tiny slice of what you see at any one moment. If someone asks, "What color was the car that passed by ten minutes ago?" or "Did that person drop their keys?", your memory might be a blank wall. This is the challenge of streaming video understanding. It's a branch of artificial intelligence where computers try to make sense of video that is arriving in real-time, like a live feed, rather than a movie they can pause and rewind. The big problem is that video is huge; it's a flood of visual data. If a computer tries to save every single frame, it runs out of memory (its brain space) and gets too slow to answer questions quickly. So, scientists have to figure out how to throw away the boring parts (like a blank sky) and keep the important parts (like a moving dog) without losing the story.

Enter ObjectStream, a new method that acts like a super-organized librarian for a robot's memory. Instead of trying to remember every single pixel of the video stream, ObjectStream decides to remember objects as the main characters of the story. Think of it this way: if you were describing a chaotic party to a friend, you wouldn't list every single person who walked through the door. Instead, you'd track the "key players"—the DJ, the person dancing on the table, the guy with the red hat. ObjectStream does exactly this for robots. It looks at the frozen video data and says, "Hey, that patch of pixels looks like a cup," and then it keeps a running diary of that cup. It tracks where the cup goes, if it gets knocked over, or if it disappears. It doesn't need to hire a separate team of experts (like external object detectors) to find these items; it figures it out all by itself using the robot's existing brain.

The paper introduces a clever three-part system to manage this memory. First, it creates "Latent Object Anchors." Imagine these as sticky notes you stick on the important things in the video. As the video plays, the robot updates these notes: "The orange mug is now on the table," or "The blue cup is being held." It keeps a history of these objects, even if they move around or change slightly. Second, it has a special section for "Transient Changes." Sometimes, things happen fast—a knife cuts a cucumber, or a ball bounces. These are quick moments that might get smoothed over if you only look at the long-term history. ObjectStream keeps a "snapshot" of these rapid changes so the robot doesn't miss the action. Third, it keeps a "Recent Visual Grounding Window," which is just a clear, high-quality view of the very last few seconds of video. This ensures that if someone asks, "What is happening right now?", the robot has a fresh, unblurred view to look at.

The researchers tested this idea on existing AI models that were already good at understanding videos but struggled with live streams. They found that by adding this "object anchor" system, the models got much better at answering questions about what was happening in the video. For example, on a test called OVO-Bench, which checks how well a robot can perceive things in real-time, the model's score jumped by 10.0 points (going from 63.3 to 73.3). Even more impressively, it did this while using about 50% less memory and answering questions twice as fast as before. In fact, the system was so efficient that it could throw away 82.5% of the raw video data (the "tokens") and still perform better than models that tried to keep everything.

The paper suggests that this approach is a practical way to solve the memory problem without needing to retrain the entire AI model from scratch. It shows that organizing memory around "things" (objects) rather than just "moments" (frames) helps the robot keep track of the story. The authors found that this method works well for both live streams and long, pre-recorded videos, proving that keeping a diary of objects is a smart way to handle the flood of visual information. It's not a magic fix that solves every problem in the world, but it suggests that if we want robots to understand our busy, moving world in real-time, we need to teach them to remember the characters, not just the scenery.

Technical Summary: ObjectStream

Problem Statement

Streaming video understanding requires models to process continuously arriving visual inputs and respond to questions that may be issued at any time, without access to future frames. This capability is critical for applications such as wearable assistants, embodied agents, and autonomous systems. Unlike offline video understanding, where the complete video is available for revisiting, streaming models must decide what information to retain under strict memory and latency constraints before future queries are known.

Existing approaches to streaming Video Large Language Models (Video-LLMs) primarily manage the growing visual context based on token importance, temporal redundancy, or segment-level relevance. However, these methods rarely organize evidence around objects that persist and evolve over time. Consequently, while existing designs preserve coarse-grained contextual and temporal semantics, finer-grained object-level information (such as identity, interactions, and state changes) is often less explicitly organized across time. This limitation hinders the model's ability to reason about questions requiring the tracing of an object's history or state evolution.

Methodology: ObjectStream

To address these limitations, the authors introduce ObjectStream, a training-free, plug-and-play visual memory framework. ObjectStream treats latent objects as memory anchors, organizing the incoming video stream without requiring external object detectors, segmentation models, or modifications to the underlying Video-LLM architecture.

The framework maintains a bounded visual memory $R_\tau$ composed of three complementary modules:

1. Latent Object-Anchored Memory (LOAM)

This module discovers and tracks spatially coherent object candidates directly from the frozen representations of the Video-LLM. It operates in four stages:

Query-Agnostic Token Saliency Estimation: For each frame, visual tokens are normalized and aggregated into a global context vector. Token saliency is estimated based on cosine dissimilarity from this context; tokens deviating from the global representation are identified as potential object candidates.
Spatial Clustering: Active tokens are clustered using a 2D patch-grid structure (4-connected spatial neighbors) to form coherent object-level regions without external segmentation.
Cross-Frame Association: Object candidates are associated across frames to build persistent tracks. A dynamic object set is maintained where each track stores a feature vector, timestamp, temporal consistency score, and saliency score. Matches are updated via Exponential Moving Average (EMA).
Bounded Object-Set Maintenance: To handle continuous video streams, the object set is capped at a budget $M$ . Objects are evicted based on a lexicographic ranking of recency, saliency, and temporal consistency.
Evidence Retention: For retained objects, the framework selects top- $K$ representative tokens from their historical pools, ensuring a compact yet comprehensive representation of persistent visual histories.

2. Object-Conditioned Temporal Residuals (OCTR)

While LOAM smooths object representations over time, this may suppress short-lived but critical changes (e.g., motion, state transitions, or interactions). OCTR detects abrupt object-level feature changes by computing the cosine difference between consecutive observations of a track. When a change exceeds an adaptive threshold, the system retrieves raw visual tokens from a local temporal window around the event. This preserves transient evidence that complements the smoothed long-term memory.

3. Recent Visual Grounding Window (RVGW)

To support real-time questions about the current visual state (e.g., "What is happening now?"), ObjectStream maintains a lightweight window of raw visual tokens from the most recent $L$ frames. This ensures high-fidelity local evidence is available for immediate grounding, complementing the compressed historical context.

Key Contributions

The paper summarizes its contributions as follows:

ObjectStream Framework: A training-free, plug-and-play visual memory mechanism for efficient streaming video understanding that requires no architectural modifications to the base Video-LLM.
Latent Object-Anchored Memory: A novel approach using latent objects as persistent anchors to organize long-term visual evidence, eliminating the need for external detectors or segmenters.
Complementary Memory Design: The integration of persistent object histories with Object-Conditioned Temporal Residuals and a Recent Visual Grounding Window, achieving strong performance and token efficiency across both online streaming and offline video understanding benchmarks.

Experimental Results

The authors evaluated ObjectStream on both online streaming and offline long-video benchmarks using Qwen2.5-VL (3B and 7B) backbones.

Online Streaming Benchmarks:

OVO-Bench: ObjectStream improved Qwen2.5-VL-7B by 10.0 points on Real-Time Visual Perception (from 63.3 to 73.3) and by 4.9 points on Backward Tracing. It outperformed other training-free methods like FluxMem and QueryStream.
StreamingBench: The method improved the average score from 73.9 to 76.8 (7B backbone) and from 68.0 to 72.4 (3B backbone).
Efficiency: In online evaluation, ObjectStream reduced peak GPU memory and Time-To-First-Token (TTFT) by approximately 50% compared to full-token baselines.

Offline Long-Video Benchmarks:

VideoMME-Long & EgoSchema: ObjectStream surpassed the full-token baseline while discarding 82.5% of visual tokens. Specifically, it improved Qwen2.5-VL-7B scores by 2.7 points on VideoMME-Long and 2.3 points on EgoSchema.

Ablation and Efficiency Analysis:

Ablation studies confirmed that LOAM provides structured object memory (boosting OVO-Bench), RVGW preserves recent observations (boosting StreamingBench), and OCTR captures abrupt changes, with the full combination yielding the best results.
Compared to strategies using external segmentation (SAM+DINOv2) or simple patch selection (Patch Top-K), ObjectStream achieved a superior balance between accuracy and efficiency, offering lower latency and memory usage while maintaining high performance.

Significance and Claims

The paper claims that latent objects serve as a practical and effective organizing principle for compact streaming video memory. By shifting from token-centric or segment-centric management to an object-centric approach, ObjectStream enables existing Video-LLMs to reason over object identities, interactions, and state changes without altering the underlying model. The results highlight that organizing evidence around persistent latent anchors allows models to retain useful visual evidence efficiently, addressing the core challenge of streaming video understanding: deciding what to keep and how to organize it before future queries are known.