🤖 machine learning

Divide, then Ground: Adapting Frame Selection to Query Types for Long-Form Video Understanding

This paper introduces DIG, a training-free framework that enhances long-form video understanding by distinguishing between global and localized queries to apply efficient uniform sampling for the former and specialized query-aware selection for the latter, thereby outperforming existing baselines while reducing computational overhead.

Original authors: Jialuo Li, Bin Li, Jiahao Li, Yan Lu

Published 2026-03-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jialuo Li, Bin Li, Jiahao Li, Yan Lu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to explain a 3-hour movie to a friend, but you only have 5 minutes to do it. You can't show them the whole movie, so you have to pick specific scenes to show them.

The problem is: How do you pick the right scenes?

For a long time, AI researchers tried two main ways:

The "Skip-Track" Method (Uniform Sampling): You just skip ahead every 10 minutes and show a random clip. It's fast and easy, but you might miss the most important plot twist because it happened in minute 14.
The "Detective" Method (Query-Aware Selection): You watch the whole movie, analyze every second, and pick the perfect scenes based on the question. This is very accurate, but it takes forever and burns a lot of energy (computing power).

This paper, titled "Divide, then Ground" (DIG), argues that we don't need to be a detective for every question. Instead, we should split questions into two types and use a different strategy for each.

The Big Idea: Two Types of Questions

The authors realized that questions about videos fall into two buckets:

The "Gist" Questions (Global Queries):
- Example: "What is this video about?" or "What is the main theme?"
- Analogy: Imagine asking, "What's the vibe of this party?" You don't need to see every single person dancing. You just need a few wide shots of the room to get the feeling.
- The Solution: For these, the simple "Skip-Track" method works perfectly! It's fast, efficient, and gets the job done. No need to overthink it.
The "Specific Detail" Questions (Localized Queries):
- Example: "What color was the car the man drove at the 12-minute mark?" or "How did the cartoon sponge walk?"
- Analogy: Imagine asking, "Did the waiter drop the tray?" If you only show random clips, you might miss the exact second the tray falls. You need to find that specific moment.
- The Solution: For these, the "Skip-Track" method fails. You do need the "Detective" method to hunt down that specific scene.

Introducing DIG: The Smart Video Assistant

The authors built a system called DIG (Divide, then Ground) that acts like a smart video editor. Here is how it works, step-by-step:

Step 1: The "Traffic Cop" (Query Identification)

First, DIG looks at the question and asks a super-smart AI (a Large Language Model): "Is this a 'Gist' question or a 'Specific Detail' question?"

If it's a Gist question, the Traffic Cop waves the "Fast Lane" flag.
If it's a Specific Detail question, the Traffic Cop waves the "Detour" flag.

Step 2: The Strategy

If it's a Gist Question (The Fast Lane):
DIG just grabs random, evenly spaced frames from the whole video. It's like flipping through a photo album quickly to get the general idea. It's super fast and cheap.
If it's a Specific Detail Question (The Detour):
DIG goes into "Detective Mode" with a three-step process:
1. Find the Scene Changes (CAFS): It scans the video to find where the scene actually changes (like a cut in a movie). It picks one "representative" frame from each scene. This creates a short, clean summary of the video's visual flow without the boring, repetitive parts.
2. The "Relevance Score" (Reward Assignment): It asks the AI: "Hey, looking at this specific frame, does it help answer the question? Does the next few seconds look like they might have the answer?" It gives the frame a score (0 to 100).
3. The "Refined Cut" (Video Refinement): It takes the high-scoring frames and the video segments around them, stitches them together into a new, shorter "highlight reel," and then feeds that to the main AI to answer the question.

Why is this a Game-Changer?

Think of it like cooking:

Old methods were like trying to cook a gourmet meal for 100 people using only a microwave (too slow) or a campfire (too messy). They tried to do the same complex thing for every single order.
DIG is like a smart kitchen. If you order a simple salad (Global Query), it just tosses the ingredients in a bowl (Uniform Sampling). If you order a complex steak (Localized Query), it fires up the grill and sears it perfectly (Specialized Pipeline).

The Results:

Speed: It saves a ton of computing power because it doesn't waste time doing complex searches for simple questions.
Accuracy: It gets the hard questions right by focusing only on the relevant parts of the video.
Scalability: It works even when you feed it huge videos (up to 256 frames or more), where other methods usually get confused and fail.

In a Nutshell

The paper teaches us that one size does not fit all. By simply asking, "Do I need the whole picture or just a specific detail?" before we start processing, we can make AI video understanding both faster and smarter. It's about working smarter, not harder.

1. Problem Statement

Large Multimodal Models (LMMs) face significant challenges in long-form video understanding due to:

Context Length Limits: LLMs have restricted context windows, making it impossible to process all frames of a long video directly.
Token Overhead: Dense video tokens are computationally expensive.
Inefficiency of Current Methods:
- Uniform Sampling: While efficient, it is query-agnostic. It often selects redundant frames while missing critical, query-specific moments, leading to poor performance on detailed questions.
- Query-Aware Selection: Existing adaptive methods select frames based on query relevance but incur prohibitive computational overhead due to complex search mechanisms. They often apply these heavy mechanisms indiscriminately to all queries, even when unnecessary.

The core question addressed is: Is a complex, query-aware search mechanism strictly necessary for all types of video queries?

2. Key Insight: Query Typology

The authors identify and validate a fundamental distinction between two types of queries, demonstrating that their performance characteristics differ significantly regarding frame sampling:

Global Queries (GQ): Require holistic understanding of the entire video (e.g., "What is the main topic?", "Summarize the video").
- Finding: Uniform sampling is highly effective and efficient for GQs. Adding complex selection mechanisms yields diminishing returns.
Localized Queries (LQ): Target specific temporal segments or details (e.g., "How does the character walk?", "What color is the car at 2:00?").
- Finding: Uniform sampling fails here; as frame counts increase, irrelevant frames introduce noise, degrading performance. These queries strictly require query-aware selection to isolate relevant segments.

3. Methodology: DIG Framework

The authors propose DIG (Divide, then Ground), a training-free frame selection framework that dynamically adapts its strategy based on the query type.

Step 1: Query Type Identification

An LLM (e.g., Qwen3-Next-80B) classifies the input query as either Global or Localized using a Chain-of-Thought (CoT) prompt.
Decision Logic:
- If Global: Proceed directly to Uniform Sampling across the whole video.
- If Localized: Trigger the specialized multi-stage pipeline.

Step 2: Specialized Pipeline for Localized Queries

For localized queries, DIG employs a three-stage process to extract and refine relevant content:

Content-Adaptive Frame Selection (CAFS):
- Instead of static sampling, DIG uses DINOv2 features to compute pairwise frame similarity.
- It identifies "peaks" in the feature distance sequence (indicating scene changes or semantic shifts).
- It selects r-frames (representative frames) as midpoints between these peaks, creating a compact, semantically diverse subset of the video.
Reward Assignment (LMM-based Scoring):
- Existing methods often use CLIPScore, which relies on surface-level feature matching. DIG leverages the LMM itself to score the r-frames.
- The LMM assigns a relevance reward (0–100) to each r-frame based on:
  1. Direct usefulness for answering the query.
  2. Whether the frame suggests adjacent frames contain necessary context.
- This utilizes the LMM's superior reasoning and world knowledge compared to simple embedding similarity.
Video Refinement:
- Iterative Selection: An iterative thresholding process selects r-frames with rewards above the mean, filtering out low-relevance candidates.
- Segment Expansion: For each selected r-frame, the method retrieves the surrounding video segment (defined by a window length $w_{len}$ ) to capture fine-grained temporal details.
- Final Sampling: The refined video (containing only relevant segments) is uniformly sampled to generate the final input frames for the LMM.

4. Key Contributions

Query Typology Discovery: The paper establishes that frame selection efficacy is contingent on query type (Global vs. Localized), challenging the assumption that complex search is universally required.
DIG Framework: A novel, training-free framework that combines efficient uniform sampling for global queries with a targeted, multi-stage pipeline for localized queries.
LMM as Reward Model: Demonstrates that using the LMM itself for relevance scoring outperforms traditional metrics like CLIPScore, providing more semantically rich signals.
Scalability: The method remains robust and effective even when scaling input frame counts up to 256 frames (and tested up to 768 in appendices), a regime where other baselines often degrade.

5. Experimental Results

The method was evaluated on three major benchmarks: MLVU, LongVideoBench, and VideoMME, using various LMMs (Qwen2.5-VL-7B/32B, InternVL3, LLaVA-OneVision).

Performance Gains: DIG consistently outperforms uniform sampling and state-of-the-art baselines (AKS, Q-Frame).
- On MLVU with 32 frames, DIG improved Qwen2.5-VL-7B accuracy by 7.68% over uniform sampling.
- It maintains performance gains even at high frame counts (e.g., 256 frames), whereas baselines like AKS and Q-Frame often suffer performance degradation due to noise or computational limits.
Query-Specific Analysis:
- Global Queries: DIG performs comparably to uniform sampling, validating the efficiency of skipping complex selection for these cases.
- Localized Queries: DIG significantly outperforms uniform sampling, proving the necessity of the specialized pipeline.
Efficiency:
- By bypassing complex selection for global queries, DIG reduces computational overhead significantly compared to applying heavy selection mechanisms universally.
- It achieves a better Accuracy vs. FLOPs trade-off, surpassing the performance ceiling of uniform sampling at higher computational budgets.

6. Significance

Paradigm Shift: Moves away from "one-size-fits-all" frame selection toward adaptive strategies based on query intent.
Practicality: As a training-free solution, DIG can be immediately deployed with existing LMMs without fine-tuning, making it highly accessible.
Scalability: Addresses the critical bottleneck of processing long videos by proving that intelligent, query-dependent sampling allows models to handle dense inputs (256+ frames) effectively, unlocking better reasoning capabilities for long-form content.
Efficiency: Optimizes resource allocation by only paying the "computational tax" of complex search when it is actually needed (i.e., for localized queries).