💬 NLP

VideoARM: Agentic Reasoning over Hierarchical Memory for Long-Form Video Understanding

VideoARM introduces an agentic reasoning paradigm with hierarchical multimodal memory that enables adaptive, on-the-fly coarse-to-fine video analysis, achieving state-of-the-art performance in long-form video understanding while significantly reducing token consumption compared to existing methods.

Original authors: Yufei Yin, Qianke Meng, Minghao Chen, Jiajun Ding, Zhenwei Shao, Zhou Yu

Published 2026-03-31

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yufei Yin, Qianke Meng, Minghao Chen, Jiajun Ding, Zhenwei Shao, Zhou Yu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Library of Alexandria" Dilemma

Imagine you are a detective trying to solve a mystery, but the evidence isn't a few photos; it's a 3-hour movie.

If you ask a standard AI (like a smart assistant) to watch this movie and answer a specific question (e.g., "What was the killer wearing in the scene where the vase broke?"), the AI faces a massive problem:

Memory Overload: It can't remember every single second of a 3-hour movie without getting confused.
Cost Explosion: To "read" the movie, the AI has to process millions of tiny data points (tokens). It's like trying to read every single page of a library of books just to find one sentence. It's incredibly expensive and slow.

Previous methods tried to solve this by either:

The "Pre-Read" Method: Reading the entire movie first, writing a summary of every 10 seconds, and storing it. (Too expensive, wastes time on boring parts).
The "Manual Map" Method: Having a human draw a map of where the important stuff is before the AI starts. (Too rigid, humans can't predict every question).

The Solution: VideoARM (The "Smart Detective")

The authors introduce VideoARM. Instead of a robot that reads the whole book, VideoARM is like a super-smart, adaptive detective who walks into the movie theater with a flashlight and a notebook.

Here is how it works, broken down into three simple concepts:

1. The "Flashlight" Strategy (Adaptive Reasoning)

Imagine you are looking for a specific person in a crowded stadium.

Old Way: You take a photo of the entire stadium, zoom in on every face, and write down a description of everyone. (Wasteful!)
VideoARM Way: You get a general idea of where the person might be. You shine your flashlight on that section. If you see a red hat, you zoom in closer. If you don't see them, you move the flashlight to the next section.

VideoARM does this with video. It doesn't watch the whole movie at once. It starts with a coarse look (wide shot), asks itself, "Is the answer here?" If not, it moves the "flashlight" (the tool) to a new part of the video and zooms in (fine look). It only spends energy on the parts of the video that actually matter to the question.

2. The "Three-Layer Notebook" (Hierarchical Memory)

Detectives need a good memory system. VideoARM uses a special notebook called HM3 (Hierarchical Multimodal Memory) with three layers:

Layer 1: The "Sensory" Scratchpad (Short-term): This is where the detective writes down what they see right now. "I see a man in a blue coat." Once they solve that specific clue, they wipe the board clean to make room for the next clue.
Layer 2: The "Case File" (Result Memory): This is where the detective writes down the facts they found. "The man in the blue coat was holding a key." This stays in the notebook so the detective doesn't forget the clues they've already gathered.
Layer 3: The "Strategy Log" (Working Memory): This is the detective's internal monologue. "I looked at the kitchen, didn't find the key. Now I should check the bedroom." It keeps track of why they are doing what they are doing, so they don't go in circles.

3. The "Toolbelt" (Agentic Tools)

The detective doesn't just "look"; they have a belt of tools to help them:

The "Time-Scoper": A tool that quickly scans a huge chunk of time to say, "The answer is probably between minute 10 and minute 15."
The "Scene Snapper": A tool that takes a quick snapshot of a long scene and summarizes it. "This is a party scene."
The "Clip Analyzer": A magnifying glass. It looks at a tiny 5-second clip in extreme detail to find a specific detail, like a logo on a shirt.
The "Earpiece" (Audio Transcriber): Sometimes the visual clues aren't enough. This tool listens to the audio and writes down what people are saying, which often holds the key to the answer.

The "Observe-Think-Act-Memorize" Loop

VideoARM runs in a continuous cycle, just like a human thinking:

Observe: Look at the current memory and the video.
Think: "I need to find the red car. I should check the parking lot section."
Act: Use a tool to zoom into the parking lot section.
Memorize: Write down what was found (or didn't find) in the notebook.
Repeat: If the answer isn't clear, go back to step 1.

Why is this a Big Deal?

1. It's Cheaper (Token Efficiency):
Because VideoARM only looks at the parts of the video that matter, it uses 50 times less data (tokens) than previous methods. It's like ordering a single slice of pizza instead of buying the whole pie just to eat one slice.

2. It's Smarter:
It doesn't rely on a human to tell it where to look. It figures out the "coarse" location first, then drills down. This makes it much better at finding the needle in the haystack.

3. It's Flexible:
If the question changes, the detective changes their strategy. If you ask about the sound of the movie, the detective grabs the "Earpiece" tool. If you ask about the visuals, they grab the "Magnifying Glass."

The Bottom Line

VideoARM is a new way for AI to understand long videos. Instead of blindly reading the whole thing or relying on a pre-made map, it acts like a curious, strategic detective. It uses a smart notebook to remember clues and a set of tools to zoom in only on the interesting parts. This makes it faster, cheaper, and much more accurate at answering questions about long movies, lectures, or meetings.

1. Problem Statement

Long-form video understanding remains a significant challenge for Multimodal Large Language Models (MLLMs) due to two primary factors:

Extended Temporal Structure: Videos (e.g., documentaries, movies) contain dense information unfolding over long time scales, requiring reasoning about dynamic long-range dependencies.
Inefficiency of Existing Approaches:
- Hand-crafted Pipelines: Methods like VideoTree rely on static, manually designed reasoning steps (clustering, scoring, tree building) that limit autonomy and fail to leverage the full reasoning potential of modern MLLMs.
- Token-Consuming Preprocessing: Methods like DVD (Deep Video Discovery) pre-segment videos into fixed clips, generate captions for all segments, and build a static database. This approach is highly token-intensive, introduces redundancy irrelevant to specific queries, and lacks the ability to update context dynamically during reasoning.

2. Methodology: VideoARM

VideoARM introduces an Agentic Reasoning-over-Hierarchical-Memory paradigm. Instead of static preprocessing, it employs an adaptive, on-the-fly loop of Observe, Think, Act, and Memorize.

A. Hierarchical Multimodal Memory (HM3)

To support the agent's decision-making, VideoARM constructs a dynamic memory system organized into three tiers:

Sensory Memory:
- Long-term Perception Pool ( $P_l$ ): Maintains a snapshot of the current broad temporal focus (base frame intervals).
- Short-term Perception Pool ( $P_s$ ): A transient workspace for fine-grained clips sampled during local exploration, used for rapid hypothesis testing.
Result Memory: Records intermediate tool outputs (timestamps, captions, analysis results) in temporal order. This allows the agent to reflect on past steps, avoid redundancy, and build a fine-grained history of evidence.
Working Memory: Stores the controller's reasoning traces and intended objectives before tool invocation. This externalizes context from the MLLM's immediate input, reducing token overflow and maintaining a clear audit trail of the agent's thought process.

B. Coarse-to-Fine Video Reasoning Agent

The agent is managed by a Controller that adaptively schedules tool invocations based on the evidence stored in HM3. It utilizes two complementary toolsets:

Temporal Scoping Tools:
- Interval Localizer: Identifies relevant frame intervals ( $T_{long}$ ) based on the query and updates the Long-term Perception Pool. It adaptively controls sampling granularity to narrow the search space.
- Clip Explorer: Performs tentative, fine-grained probing within local intervals ( $T_{local}$ ) without altering the global focus, storing results in the Short-term Perception Pool.
Multimodal Understanding Tools:
- Scene Snapper: Generates concise captions for long temporal segments to provide high-level semantic abstraction.
- Audio Transcriber: Extracts and transcribes audio from specific intervals (using Whisper) to resolve visual ambiguities.
- Clip Analyzer: Performs detailed analysis of fine-grained frames to answer sub-questions and provide confidence scores.

C. The Reasoning Loop

The controller operates in an iterative loop:

Observe: Review current HM3 states (sensory inputs, past results, working memory).
Think: Generate a reasoning trace ( $R_t$ ) to hypothesize missing evidence and select the next tool.
Act: Invoke the selected tool with specific parameters (e.g., frame intervals, sub-questions).
Memorize: Append new observations to Result Memory and update the Working Memory with the reasoning trace.
The loop continues until the agent reaches a confidence threshold or hits a step budget ( $N$ ), at which point it generates the final answer.

3. Key Contributions

Paradigm Shift: Moves away from static, exhaustive preprocessing (like DVD) to adaptive, on-the-fly agentic reasoning.
Hierarchical Memory (HM3): Proposes a novel memory architecture that separates sensory perception, result logging, and reasoning traces, enabling the agent to manage long contexts without overwhelming the MLLM's context window.
Coarse-to-Fine Toolset: Designs a complementary set of tools that allow the agent to first localize broad temporal regions and then drill down into fine-grained evidence, significantly reducing token consumption.
Training-Free Efficiency: The framework operates without fine-tuning the base MLLMs, relying instead on prompt engineering and tool orchestration.

4. Experimental Results

The authors evaluated VideoARM on three prevalent benchmarks: Video-MME, LongVideoBench, and EgoSchema.

Performance: VideoARM achieves State-of-the-Art (SOTA) performance across all benchmarks, outperforming commercial VLMs (e.g., Gemini-1.5-Pro, GPT-4o), open-source models, and previous training-free approaches (e.g., DVD, VideoTree).
- Example: On Video-MME (Long), VideoARM (o3+GPT-4o) achieved 82.8% accuracy, surpassing DVD (71.6%) and GPT-4o (71.9%).
Token Efficiency: The most significant contribution is the drastic reduction in token consumption.
- Theoretical: For a 30-minute video, DVD consumes ~3.98M tokens (conservative estimate), while VideoARM consumes ≤80k tokens (approx. 1/50th of DVD).
- Experimental: On Video-MME, processing 10 videos with 30 queries required 1/34th the tokens of DVD.
Ablation Studies:
- Removing any component of HM3 (Sensory, Result, or Working Memory) leads to performance drops, confirming the necessity of the hierarchical structure.
- The adaptive frame sampling strategy outperforms fixed sampling, proving that dynamic granularity control improves efficiency.
- Using OpenAI o3 as the controller and GPT-4o for understanding tools yielded the best results, highlighting the importance of strong reasoning capabilities in the controller.

5. Significance

VideoARM demonstrates that effective long-form video understanding does not require exhaustive preprocessing or massive token budgets. By combining an agentic reasoning loop with a structured, hierarchical memory system, the framework enables MLLMs to:

Reason Dynamically: Adaptively focus on relevant video segments rather than processing the entire video uniformly.
Scale Efficiently: Drastically reduce computational costs (tokens) while improving accuracy.
Generalize: Provide a practical, training-free solution applicable to diverse long-video tasks (movies, lectures, meetings) without the need for expensive model fine-tuning.

This work establishes a new direction for video agents, prioritizing adaptive reasoning and memory management over brute-force data ingestion.