Technical Summary: ViSAGE

Problem Statement

Multimodal agents operating in long-horizon environments face significant challenges in maintaining entity-consistent and temporally grounded reasoning. Existing agentic memory approaches suffer from three primary limitations:

1. Spatio-Temporal Detail Loss: Aggressive down-sampling and compression required to process extensive video streams erase fine-grained identity cues (e.g., micro-dynamics, occlusion handling), leading to unreliable entity disambiguation.
2. The Segmentation Dilemma: Current pipelines often process inputs in isolated chunks. When identity evidence (such as a name mentioned in dialogue) arrives late, it cannot retroactively correct earlier memory segments, resulting in fragmented narratives.
3. The Similarity–Veracity Gap: Retrieval-based systems relying on vector similarity often conflate semantic relevance with factual correctness. This leads to the retrieval of contextually related but identity-mismatched evidence, causing agents to hallucinate answers or propagate errors based on noisy data.

Methodology: ViSAGE Framework

ViSAGE (Visual Self-correcting AGEntic memory) is a multimodal framework designed to construct self-correcting, entity-centric memories. It operates through a three-stage pipeline:

1. Entity-Anchored Preprocessing

To counter spatial degradation and temporal sparsity, ViSAGE employs Sequence-Level Entity Anchoring rather than per-frame identity assignment.

• Visual Anchoring: The system runs multi-object tracking to generate long-range trajectories. It selects the highest-quality face instance as a "prototype" for each track, mapping all associated face observations (even degraded ones) to a stable entity ID.
• Audio-Visual Binding: Leveraging the McGurk effect, the system performs Dual-Criteria Validation for voice-face binding. It combines Active Speaker Detection (ASD) on high-FPS streams with acoustic voice embeddings. A speaker is only assigned when there is high voice similarity or confirmed audio-visual synchrony, preventing misattribution in crowded scenes.

2. Bidirectional Memory Refinement

ViSAGE maintains a dual-structured memory bank inspired by Tulving's memory systems:

• Incident Logs: An append-only record of sequential events (timestamps, actions, dialogue) where the "what" is immutable but the "who" is refinable.
• Object Cards: Continuously updated profiles for entities that accumulate and revise identity attributes.

The framework introduces Bidirectional Memory Refinement to resolve the segmentation dilemma:

• Forward Grounding: Uses stable entity IDs to retrieve historical context (from Incident Logs and Object Cards) to ground current observations.
• Backward Rectification: When new identity evidence emerges (e.g., a character is named "Mario" later in the video), the system triggers a Global Backward Update. It retroactively traverses historical Incident Logs to replace anonymous IDs with the resolved identity, unifying fragmented narratives and correcting past associations.

3. Epistemic Adjudication

To bridge the similarity-veracity gap, ViSAGE employs Multi-Agent Cross-Verification:

• Identity Resolution Dispatcher: Sanitizes queries by mapping noisy transcriptions (e.g., "Malloy") to correct entity names in the registry before retrieval.
• Specialized Agents: A Scene Agent queries Incident Logs for dynamic, event-driven facts, while a Character Agent queries Object Cards for static attributes. This avoids error-prone multi-hop reasoning on disjointed fragments.
• Judge Agent: Acts as an arbiter to evaluate the sufficiency and consistency of retrieved evidence. Crucially, if valid evidence is missing or conflicting, the system triggers a Verified Epistemic Refusal rather than hallucinating an answer.

Key Contributions

1. Framework Design: Introduction of ViSAGE, which separates memory into sequential Incident Logs and structured Object Cards, enabling retroactive, entity-centric updates that transcend chunk-wise processing.
2. Algorithmic Novelty: Proposal of Bidirectional Memory Refinement, a feedback loop that uses delayed identity evidence to retroactively unify historical records, ensuring long-term entity consistency.
3. Epistemic Safety: Implementation of Multi-Agent Cross-Verification under an identity-evidence alignment constraint. This prioritizes verified refusals over plausible fabrications in high-stakes embodied scenarios.
4. Empirical Impact: Demonstration of state-of-the-art performance on authoritative benchmarks, validating the framework's robustness in long-form video understanding.

Experimental Results

ViSAGE was evaluated on M3-Bench-robot, M3-Bench-web, and Video-MME-long.

• Performance: ViSAGE consistently outperformed strong baselines (including M3-Agent, GPT-4o, and Gemini-3-Pro) by an average of 5.9% in accuracy.
  • M3-Bench-robot: 45.5% (vs. 30.7% for M3-Agent).
  • M3-Bench-web: 58.4% (vs. 48.9% for M3-Agent).
  • Video-MME-long: 79.1% (vs. 74.2% for the ungrounded backbone).
• Cross-Modal Reasoning: The framework showed decisive improvements in binding fragmented visual and auditory cues, outperforming the agentic baseline by +15.9% on robot and +27.1% on web tasks.
• Safety and Reliability: Qualitative analysis showed ViSAGE significantly reduced hazardous hallucinations (Identity Misalignment reduced by 54.3%) and increased Epistemic Refusals by 264.3%, demonstrating a shift toward "knowing what it doesn't know."
• Efficiency: Despite its complexity, ViSAGE achieved superior accuracy with lower computational overhead (26,000 tokens/60s per clip) compared to M3-Agent (30,000 tokens/90s), largely due to aggressive downsampling enabled by robust entity anchoring.

Significance and Limitations

The paper argues that long-form video understanding requires a paradigm shift from strictly sequential processing to self-correcting memory architectures. ViSAGE establishes that post-hoc rectification is structurally necessary for narrative coherence. Its significance lies in establishing a new standard for epistemic safety in embodied AI, prioritizing rigorous evidence verification over plausible fabrication.

The authors acknowledge a primary limitation: the current Entity-Anchored Preprocessing is explicitly tailored for human-centric interactions (relying on Face_ID and Speaker_ID). Consequently, non-human elements (objects, animals) are treated as background context rather than actively tracked profiles. Future work aims to generalize entity registration to include open-vocabulary object tracking.