🤖 AI

Watch, Remember, Reason: Human-View Video Understanding with MLLMs

This paper presents a unified "human-view" framework for video understanding with multimodal large language models, organizing the field around three core capabilities—watching, remembering, and reasoning—to systematically analyze challenges, methods, applications, and future directions in processing long and complex video scenarios.

Original authors: Jiahao Meng, Yue Tan, Qi Xu, Kuan Gao, Weisong Liu, Yanwei Li, Jason Li, Lingdong Kong, Haochen Wang, Qianyu Zhou, Jiangning Zhang, Guangliang Cheng, Yunhai Tong, Lu Qi, Minghsuan Yang

Published 2026-06-08

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Jiahao Meng, Yue Tan, Qi Xu, Kuan Gao, Weisong Liu, Yanwei Li, Jason Li, Lingdong Kong, Haochen Wang, Qianyu Zhou, Jiangning Zhang, Guangliang Cheng, Yunhai Tong, Lu Qi, Minghsuan Yang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to understand a very long, complex movie. You can't just glance at it; you have to watch it, remember the plot, and figure out the mystery. This paper argues that teaching computers to do the same thing requires them to master three specific human-like skills: Watching, Remembering, and Reasoning.

Here is a simple breakdown of what the paper says, using everyday analogies.

The Big Picture: The "Human-View" Approach

The authors say that instead of treating video understanding as a bunch of separate math problems, we should look at it like a human does. When you watch a 2-hour movie, you don't stare at every single frame with equal intensity. You watch the important parts, remember the characters and plot points, and reason through the story to answer questions like "Who did it?" or "Why did that happen?"

The paper organizes all the latest AI research into these three buckets.

1. Watching: The "Selective Eye"

The Problem: Videos are huge. A 1-hour movie has thousands of frames. If you show a computer every single frame, it gets overwhelmed (like trying to drink from a firehose). Also, the most important clue might be a tiny detail that only lasts for a split second.

The Solution: The paper reviews methods that teach AI to be a "selective watcher."

Fine-Grained Watching: This is like a detective zooming in on a specific fingerprint. The AI learns to pinpoint exactly when and where something happens (e.g., "The gun was fired at 12:05").
Comprehensive Watching: This is like a movie critic summarizing the whole film. The AI learns to describe the whole video or break it down into scenes.
Audio-Visual Watching: This is like watching a movie with the sound on. The AI learns to listen to dialogue and music while watching the action, understanding that a scream matches a scary face.
Efficient Watching: This is the "skip the boring parts" skill. The AI learns to ignore redundant frames (like a long shot of a wall) and only keep the frames that actually answer the question.

2. Remembering: The "Smart Filing Cabinet"

The Problem: Even if the AI watches the video, it can't hold the whole thing in its "brain" at once. If the video is 2 hours long, the AI might forget what happened in the first 10 minutes by the time it reaches the end.

The Solution: The paper looks at how AI builds a "memory system."

Offline Memory: Imagine you watch a movie and then write a summary in a notebook. Later, you can flip through that notebook to find details. Some AI systems do this by compressing the video into a smaller "summary" or a list of key events before answering a question.
Streaming Memory: This is like a live news anchor. The video is coming in real-time, and the AI has to remember what happened just now while also keeping track of what happened an hour ago, all without running out of space. It uses a "rolling buffer" to keep the most important recent info and a "long-term archive" for the big picture.
Agentic vs. Non-Agent:
- Non-Agent: The AI automatically writes its own summary as it goes (like a student taking notes).
- Agentic: The AI acts like a librarian. It realizes it forgot something, goes back to the "video library," finds the specific clip, and brings it back to its "desk" to look at again.

3. Reasoning: The "Detective's Logic"

The Problem: Just seeing and remembering isn't enough. The AI needs to connect the dots. If the AI says, "The butler did it," but it can't point to the specific moment in the video that proves it, that's a hallucination (a lie).

The Solution: The paper discusses how AI learns to "think" before it speaks.

Text-Only Reasoning: The AI writes a long internal thought process (like a detective's notebook) before giving an answer. It says, "I saw X, then Y happened, so Z must be true."
Thinking with Videos: This is the newest and most advanced idea. Instead of just guessing, the AI actively goes back to the video to check its work. It's like a detective saying, "Wait, I think I saw a clue at minute 10. Let me rewind and look at that specific frame again."
- Agentic: The AI uses tools to zoom in, crop a frame, or search for a specific timestamp.
- Non-Agent: The AI is trained to naturally include timestamps and descriptions of what it saw in its answer, proving it actually looked at the evidence.

Specialized "Movies" (Subfields)

The paper also notes that different types of videos need different skills:

Egocentric (First-person): Videos taken from a person's eyes (like a GoPro). The AI has to understand what the person is doing and intending to do.
Sports: These are fast and full of rules. The AI needs to know the rules of the game to understand why a referee blew a whistle.
Instructional (Lectures): These are long and dense. The AI needs to track the steps of a lesson and match the teacher's voice to the slides.
Medical: These are high-stakes. The AI needs to be very precise about tools and body parts, often looking at long surgical procedures.
Movies/Storytelling: These rely on plot and character relationships. The AI has to remember who the characters are and how they changed over the whole story.

The Toolkit: Data and Tests

The paper lists the "textbooks" (datasets) and "exams" (benchmarks) researchers use to train and test these AI models.

Datasets: These are massive collections of videos with questions and answers, some with detailed notes on why the answer is correct.
Benchmarks: These are tests that check if the AI can actually handle long videos, find specific moments, or reason through complex stories without getting confused.

The Future: What's Next?

The paper concludes that while we are making progress, there are still big hurdles:

Spatial Reasoning: AI is still bad at understanding exactly where objects are in 3D space and how they move relative to each other.
Multi-Video: Most AI can only handle one video at a time. Real life involves switching between different camera angles or related videos.
Hour-Long Videos: Handling videos that are hours long without losing the plot is still very hard.
Efficiency: We need AI that can find the truth without checking every single frame, saving time and energy.
Streaming: Making AI that can watch a live stream and react in real-time, like a proactive assistant, is a major goal.

In short, the paper says: To make AI truly understand video, we need to stop treating it like a static image and start treating it like a dynamic story that requires a human-like mix of observation, memory, and logic.

Technical Summary: Watch, Remember, Reason: Human-View Video Understanding with MLLMs

1. Problem Statement

The field of video understanding is undergoing a paradigm shift driven by Multimodal Large Language Models (MLLMs). While early video models focused on short clips and isolated perception tasks, current research is moving toward long-horizon comprehension, where inputs range from minutes to hours. This shift introduces a fundamental tension between redundancy (long videos contain many repetitive frames) and evidence sparsity (decisive evidence is often brief, distant, or scattered).

Existing approaches often treat video tasks as isolated benchmarks, failing to address the systemic requirements of real-world scenarios such as movies, sports broadcasts, and medical procedures. These scenarios demand models that can:

Selectively perceive useful moments under limited computational budgets.
Preserve salient information beyond finite context windows.
Perform faithful reasoning by connecting answers to explicit spatio-temporal evidence.
Handle asynchronous multimodal signals (visual, audio, text).

Current surveys lack a unified framework that integrates perception, memory, and reasoning for long-video MLLM systems, often focusing instead on specific tasks, techniques, or training paradigms.

2. Methodology: The "Human-View" Taxonomy

This paper proposes a unified framework for analyzing video understanding systems based on a human-view perspective, decomposing the process into three core functional abilities: Watching, Remembering, and Reasoning.

2.1 Unified Formulation

The authors formalize a video understanding system $F_{VU}$ as a sequence of functional modules:

Watching ( $F_{watch}$ ): Extracts task-relevant perceptual evidence ( $Z$ ) from multimodal streams ( $V, A, T$ ) conditioned on a query ( $q$ ). This involves spatio-temporal grounding, query-aware frame selection, and cross-modal alignment.
Remembering ( $F_{remember}$ ): Updates the memory state ( $m_t$ ) over time by accumulating useful evidence and filtering redundancy. This bridges short-term context with long-term memory ( $M$ ).
Reasoning ( $F_{reason}$ ): Operates on perceptual evidence and memory to produce a reasoning trace ( $R$ ), which may include textual steps, grounded timestamps, or tool-use actions.
Output ( $F_{out}$ ): Generates the final prediction ( $O$ ) based on the integrated state.

The system is typically realized within an autoregressive MLLM, trained via Supervised Fine-Tuning (SFT) or Reinforcement Learning (RL) techniques like Group Relative Policy Optimization (GRPO).

2.2 Taxonomy of Methods

The survey organizes representative methods according to these three functional abilities:

Watching (Perception):
- Fine-grained: Focuses on precise spatio-temporal grounding (e.g., TimeChat, LITA) and object tracking.
- Comprehensive: Covers whole-video captioning, dense video captioning, and region-level descriptions.
- Audio-Visual: Integrates auditory signals for omni-modal perception (e.g., Qwen2.5-Omni, LLaMA-Omni).
- Efficient: Addresses redundancy via frame selection, token compression, and sparse attention mechanisms (e.g., AKS, Video-XL-2).
Remembering (Memory):
- Offline Memory: Divided into Agentic (LLMs autonomously invoke tools to construct/retrieve memory, e.g., VideoAgent) and Non-Agent (deterministic pipelines using compression and structured storage, e.g., MovieChat, LongVU).
- Streaming Memory: Handles unbounded video streams via sliding windows, KV cache optimization, and hierarchical memory banks (e.g., Flash-VStream, StreamMem).
Reasoning (Inference):
- Text-only Reasoning: Models generate intermediate thought traces purely in language. Approaches include Agentic (iterative planning and tool use, e.g., DoraemonGPT) and Non-Agent (post-training with RL or CoT supervision, e.g., Video-R1).
- Thinking with Videos: Models interleave reasoning with explicit visual grounding (e.g., re-watching, zooming, bounding boxes). This includes Agentic approaches (using tools to inspect evidence, e.g., VITAL) and Non-Agent approaches (generating structured outputs with timestamps/boxes, e.g., Open-o3-Video).

3. Key Contributions

The paper makes three primary contributions to the field:

Human-View Taxonomy: It introduces a coherent "Watch–Remember–Reason" framework that connects diverse video MLLM methods, clarifying their functional roles rather than grouping them by isolated tasks.
Comprehensive Coverage: It provides a broad review of key techniques in the MLLM era, specifically covering long-video understanding, fine-grained grounding, memory modeling, agentic systems, streaming processing, and grounded reasoning.
Systematic Resource Summary: It consolidates training datasets, evaluation benchmarks, and domain-specific applications (egocentric, sports, instructional, medical, narrative), offering practical guidance for model development and evaluation.

4. Results and Findings

The paper does not present new experimental results but synthesizes the state-of-the-art through its taxonomy. Key findings include:

Shift in Paradigm: The field is moving from simple input compression and answer generation toward structured memory, streaming systems, and evidence-grounded reasoning.
Effectiveness of RL: Reinforcement learning (particularly GRPO and preference optimization) has proven effective for improving temporal grounding, reducing hallucinations, and encouraging faithful reasoning traces.
Memory Challenges: While offline memory methods are maturing, streaming memory remains a critical challenge for handling unbounded inputs with constant latency and memory budgets.
Reasoning Faithfulness: "Thinking with videos" approaches (interleaving visual evidence with reasoning) show superior performance in reducing hallucinations compared to text-only reasoning, though they often incur higher computational costs.
Domain Specificity: Different domains impose unique constraints. For instance, medical videos require global procedural context and local anatomical evidence, while sports videos demand precise temporal localization of fast actions.

5. Significance and Future Directions

The paper argues that the "human-view" perspective is essential for understanding the functional roles of methods and explaining why perception, memory, and reasoning must work together. It identifies several open problems and future directions:

Spatial Reasoning: Bridging the gap between object-level tracking and scene-level 3D understanding.
Multi-Video Grounding: Moving beyond single-clip inputs to handle highlight compilations and multi-segment retrieval.
Hour-Scale Understanding: Developing structured, multi-level memory systems that can handle rare, decisive moments over hours without losing long-range dependencies.
Efficient and Verifiable Reasoning: Balancing computational cost with faithfulness by treating reasoning as a budgeted evidence search.
Streaming Egocentric Video: Creating stateful, goal-driven streaming memory for proactive assistants and embodied agents.

The authors conclude that the future of video intelligence lies in scalable, memory-aware, and evidence-grounded systems, with a continuous effort to trace related works via their curated repository.