Visual Agentic Memory: Enabling Online Long Video Understanding via Online Indexing, Hierarchical Memory, and Agentic Retrieval

Technical Summary: Visual Agentic Memory (VAM)

Problem Statement
Long video understanding faces a fundamental challenge: while high-bandwidth video capture is practical, transforming unbounded streams into a stable, queryable, and verifiable memory substrate remains difficult. Existing paradigms suffer from specific limitations. Direct long-context models are vulnerable to attention dilution as video duration increases. Compressed memory schemes improve tractability but often discard fine-grained visual evidence, leading to a "recoverability collapse" where semantic gist is preserved but the raw evidence needed for verification is lost. Recursive agentic approaches can accumulate memory drift when intermediate reasoning cannot be checked against preserved observations. Furthermore, online-tuned designs often compress too aggressively, while long-range-tuned designs rely on costly offline preprocessing. The core problem is how to maintain an archive spanning hours or days that remains searchable, auditable, and useful for downstream reasoning without relying solely on compressed latent states.

Methodology: Visual Agentic Memory (VAM)
The authors propose Visual Agentic Memory (VAM), a training-free framework designed to transform raw video streams into searchable long-horizon memory while preserving original frames as first-class evidence. The system decouples continuous indexing from query-time reasoning and operates through three coupled components:

1. Online Indexing: This component filters incoming frames under streaming constraints to retain visually significant transitions. It employs a three-stage process:

   • Frame Filtering: Removes blurry frames (via Laplacian variance) and visually redundant frames (via pre-embedding similarity checks) before high-cost neural processing.
   • Embedding Extraction: Retained frames are encoded into dense embeddings using a pre-trained multimodal model.
   • Boundary-Preserving Commitment: The system uses an adaptive deduplication gate based on Otsu partitioning of embedding distances to decide when to commit a "moment" (a retained frame with metadata). This ensures transitions are preserved while quasi-static intervals are compacted.

2. Hierarchical Memory: Retained moments are organized into "events" via a parallel temporal-spatial representation.

   • Spatial Representation: Stores retained raw frames and their embeddings for direct visual inspection.
   • Temporal Representation: Stores event summaries generated by an MLLM from the temporally bounded evidence. These summaries capture salient activities, entities, and temporal cues.
   • Age-Aware Storage: The system manages storage tiers (recent, mid, long) where older segments are consolidated into compact cues, while recent windows retain dense payloads. This allows the system to scale to month-long streams (e.g., 51 days) by storing only a fraction of the raw frames.

3. Agentic Retrieval: At query time, an MLLM acts as a retrieval agent operating in a bounded multi-turn loop. It follows an "observe–hypothesise–inspect" paradigm:

   • Hybrid Retrieval: The agent searches both the temporal pathway (event summaries) and the spatial pathway (frame embeddings) to locate candidate evidence.
   • Grounded Inspection: Retrieved candidates are provisional. The agent invokes an inspect action to perform direct visual verification on raw frames, resolving ambiguities and adjudicating conflicts between retrieval signals.
   • Answer Synthesis: The agent terminates only when sufficient grounded evidence is found, producing an answer anchored to specific, recoverable visual intervals and citing the evidence.

Key Contributions

1. Formulation: The paper reframes online long-video understanding as an evidence-recoverable visual memory problem, shifting focus from merely extending context windows to creating searchable and inspectable memory.
2. Framework: VAM is introduced as a training-free, retrieval-first framework that incrementally builds memory from streaming video, decoupling indexing from reasoning.
3. Representation: A hierarchical parallel temporal-spatial representation is proposed, aligning event-level summaries with retained raw frames to support both efficient timeline search and direct visual verification.
4. Empirical Validation: The framework demonstrates improved performance in online and long-horizon settings while storing a minimal fraction of the raw data.

Results
The authors evaluate VAM on two benchmarks:

• OVO-Bench (Online Evaluation): VAM achieves the highest RT+BT (Real-Time + Backward Tracing) average of 68.41 across all reported baselines. This outperforms the end-to-end use of the same underlying MLLM (Gemini 3 Flash), which scores 67.46, indicating that the architectural components (indexing, memory, retrieval) provide a systematic benefit over model capacity alone.
• MM-Lifelong (Long-Horizon Evaluation): On the month-scale split (105.6 hours over 51 days), VAM reaches 17.11% accuracy. This is the second-highest reported result, trailing only ReMA with GPT-5 (17.62%) and significantly outperforming end-to-end MLLM baselines.
• Efficiency: In the month-scale run, VAM stores only 6,876 frames (0.06% of the raw 11.4 million frames), representing a reduction of 27.7x to 55.3x compared to uniform fixed-rate sampling baselines.

Significance and Claims
The paper claims that long-horizon video understanding benefits from treating visual memory as an explicit, inspectable, and queryable substrate rather than relying solely on compressed latent states. By preserving raw frames as first-class evidence, VAM enables grounded reasoning where answers can be verified against original observations. The results suggest that explicit evidence retention and agentic retrieval can narrow the performance gap in month-scale video understanding more effectively than simply increasing context windows or relying on recursive reasoning without evidence preservation. The authors note that while the human-model gap remains large (82.5% vs. 17.11% on MM-Lifelong), VAM's evidence-preserving approach offers a more effective path toward solving this challenge than current alternatives.

The authors also acknowledge limitations, including modality asymmetry (lack of audio preservation), the increasing difficulty of fine-grained disambiguation as archives grow, and the operational costs associated with continuous indexing and iterative inspection. They emphasize that future deployments must address privacy concerns regarding the retention of raw visual evidence.