🤖 AI

HPP: Hierarchical Programmatic Probing for Long Video Understanding by Decoupling Perception and Reasoning

This paper proposes Hierarchical Programmatic Probing (HPP), a framework that decouples semantic perception from higher-order temporal reasoning by leveraging an LLM to iteratively plan and execute programmatic queries on a hierarchically segmented video, thereby overcoming the limitations of latent multi-step planning in standard vision-language models for long video understanding.

Original authors: Awais Rauf, Ahmed Hasssan, Greg Slabaugh

Published 2026-06-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Awais Rauf, Ahmed Hasssan, Greg Slabaugh

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a mystery in a library that contains one million books (representing a long video with thousands of frames).

The Old Way (Traditional AI):
Most current AI models try to read every single page of every single book in the library all at once, in one giant gulp. They try to memorize the whole story, find the clues, and solve the mystery simultaneously.

The Problem: The library is too big. The AI gets overwhelmed, forgets the beginning by the time it reaches the end, and misses tiny details because it's trying to process too much information at once. It's like trying to drink the ocean through a straw.

The New Way (HPP - Hierarchical Programmatic Probing):
The authors of this paper propose a smarter strategy. Instead of one AI trying to do everything, they split the job into two specialized workers who talk to each other:

The Detective (The Coding LLM): This is a smart, logical brain that knows how to write code and plan strategies. It doesn't look at the pictures; it just thinks.
The Librarian (The Small VLM): This is a smaller, faster AI that is very good at looking at pictures and describing what it sees, but it's not great at solving complex puzzles on its own.

How the Detective Solves the Mystery

The Detective doesn't read the whole library. Instead, it uses a three-step process to find the answer efficiently:

1. Organizing the Library (Hierarchical Segmentation)

First, the Detective realizes the library is messy. It uses a special tool to quickly sort the books.

The Metaphor: Imagine the video is a long movie. The Detective uses the "movie file" itself (which already has built-in markers for scene changes) to chop the movie into Scenes, then into Shots, and finally into Key Moments.
Why it helps: Instead of looking at 100,000 individual frames, the Detective now only has to look at 100 "scenes." It ignores the boring, repetitive parts (like a character walking down a hall for 5 minutes) and focuses only on where the action happens.

2. Asking the Right Questions (Semantic Search)

The Detective has a specific question, like "When did the character drop the red apple?"

The Metaphor: Instead of walking down every aisle, the Detective uses a smart search engine. It types in keywords and asks the Librarian to find the specific "scenes" that might contain an apple.
The Trick: The Detective is clever. It doesn't just ask "apple." It asks "red fruit," "fruit falling," "character eating," etc., all at once. It then combines the results to make sure it hasn't missed anything.

3. Zooming In for Proof (Targeted Perception)

Once the search engine gives the Detective a list of 5 likely scenes, it doesn't ask the Librarian to describe the whole movie again.

The Metaphor: The Detective says, "Okay, Librarian, I think the apple dropped in Scene 42. Please look only at the 5 seconds around that moment and tell me exactly what you see."
The Result: The Librarian gives a precise answer. The Detective then uses that tiny piece of evidence to solve the puzzle.

Why This is a Big Deal

The paper claims this method is much better at understanding long videos for three main reasons:

It Decouples Thinking from Seeing: The "Thinking" (planning the search) and the "Seeing" (looking at the pixels) are separated. The smart brain doesn't get tired from looking at millions of pictures; it just directs the eyes where to look.
It Saves Money and Power: Because the AI only looks at the specific parts of the video that matter, it uses far less computer power (tokens) than models that try to watch the whole video at once. For very long videos, this method is up to 16 times more efficient.
It Gets Smarter with Better Tools: The system gets better automatically if you give the "Detective" a smarter brain (a better coding LLM). The paper shows that as the coding ability of the AI improves, its video understanding improves right along with it.

The Bottom Line

The paper introduces HPP, a system that treats a long video not as a giant wall of images, but as a structured document. It uses a "Detective" to write code that navigates the video, finds the relevant scenes, and asks a "Librarian" to look closely only at those specific moments. This allows the AI to solve complex, long-range puzzles in videos without getting overwhelmed.

Technical Summary: Hierarchical Programmatic Probing (HPP) for Long Video Understanding

1. Problem Statement

Long video understanding presents a fundamental challenge for current Vision-Language Models (VLMs). The dominant paradigm involves projecting video frames into an LLM's embedding space and performing perception and reasoning jointly within a single forward pass. The authors argue this "monolithic" formulation is structurally mismatched for long videos due to two primary bottlenecks:

Latent Reasoning Limits: LLMs struggle to discover and execute multi-step planning strategies within their latent representations, particularly when resolving long-range temporal dependencies.
Token Inflation: Long videos generate massive numbers of visual tokens (e.g., 1.08M tokens for a one-hour video), which inflates computational costs and KV-cache requirements while diluting relevant signals across the context window.

Existing agent-based frameworks attempt to mitigate this via dense captioning or rigid search workflows, but these approaches often rely on manually designed heuristics that constrain adaptive exploration and fail to fully exploit the iterative, dynamic problem-solving capabilities of modern coding-capable LLMs.

2. Methodology: Hierarchical Programmatic Probing (HPP)

The proposed HPP framework decouples semantic perception from higher-order temporal reasoning. Instead of a single-pass inference, HPP reformulates long video understanding as an iterative, programmatic exploration of a hierarchically segmented video within an interactive coding environment.

Core Architecture

The system consists of three main components interacting in a Read-Evaluate-Print Loop (REPL):

Coding-Capable LLM: Acts as the "reasoning agent." It is initialized with minimal context (system prompt, metadata, query) and is responsible for strategy formulation, planning, relational inference, and code execution. It does not process raw pixels directly.
Lightweight VLM: Acts as a "perception module." It is invoked on-demand by the LLM to perform targeted spatio-temporal understanding on specific frames or short segments.
Structured Probing Environment: A Python-based REPL that exposes the video to the LLM via a set of efficient functions, preventing naive frame-level processing.

Key Technical Components

To make programmatic probing tractable over long videos, HPP introduces four specific modules:

Information-Density-Aware Hierarchical Segmentation:
- Instead of uniform sampling, the system leverages compressed video bitstream information (I-frames, motion vectors, residuals) to partition raw frames into shots.
- It computes "shot energy" based on motion and residual vectors to identify information density.
- Shots are further grouped into scenes (temporally coherent) and clusters (globally semantically consistent) using a lightweight distilled visual encoder. This creates a multi-scale structure (Shots $\to$ Scenes $\to$ Clusters) for coarse-to-fine navigation.
Motion-Adaptive Late-Interaction Semantic Retrieval:
- To localize query-relevant segments, the system employs a retrieval module that avoids uniform frame sampling.
- It samples frames proportionally to the shot's motion energy.
- It computes similarity using a hybrid late-interaction approach: combining global shot-level embeddings with fine-grained token-frame interactions (MaxSim) to capture both static spatial context and motion-driven action cues.
Structured Probing Functions:
- The LLM interacts with the video via a cost-graded API within the REPL:
  - Cheap: Structure queries (get_shots, get_scenes), subtitle searches.
  - Moderate: Semantic retrieval (search_in_shots), frame extraction.
  - Expensive: VLM queries (vlm_query).
- This cost structure encourages the LLM to adopt a coarse-to-fine strategy: orient using cheap signals, narrow the temporal region, and invoke perception only on targeted segments.
Iterative Reasoning Loop:
- The LLM writes and executes Python code to decompose queries into sub-goals, perform parallel searches, verify candidates with the VLM, and iteratively refine results (e.g., deduplicating search results, re-phrasing queries).

3. Key Contributions

Decoupling Perception and Reasoning: The paper proposes a paradigm shift where a coding-capable LLM handles high-level planning and strategy execution, while a lightweight VLM is restricted to targeted perceptual extraction. This allows for independent scaling of the reasoning and perception components.
Programmatic Video Navigation: HPP treats video understanding as a code-execution task, enabling the LLM to autonomously discover adaptive strategies (e.g., multi-query ensemble search, temporal non-maximum suppression) without explicit instruction.
Efficient Hierarchical Structuring: By leveraging video codec metadata (I-frames, motion vectors) for segmentation, the method avoids expensive optical flow calculations while effectively capturing temporal redundancy and information density.
Emergent Strategies: The framework demonstrates that LLMs can spontaneously develop complex reasoning patterns, such as parallelizing searches across paraphrased queries and using code to merge/deduplicate results, which are difficult to achieve with fixed workflows.

4. Experimental Results

The authors evaluated HPP on four major long-video benchmarks: LongVideoBench (LVB), EgoSchema, VideoMME, and MLVU.

Performance on LongVideoBench: HPP significantly outperforms monolithic VLMs, particularly on long-duration videos (600s–3600s) and relational reasoning tasks (L2). For instance, on the 3600s subset, HPP achieved a 70.5% accuracy compared to 57.4% for a baseline VLM with a similar vision encoder size.
Scaling Laws:
- Perception Scaling: Improving the vision encoder size (from 0.3B to 6B) within the HPP framework yielded consistent gains (+10.9% and +7.8% respectively) over monolithic baselines.
- Reasoning Scaling: Stronger coding LLMs (e.g., Claude Opus 4.6 vs. MiniMax-M2.5) directly translated to better video understanding performance (59.2% $\to$ 76.3%), confirming that the framework effectively leverages LLM coding capabilities.
General Benchmarks: HPP achieved competitive or superior results on EgoSchema, VideoMME, and MLVU compared to models with significantly larger parameter counts (e.g., 72B–78B models) and proprietary systems like GPT-4o and Gemini 1.5 Pro.
Efficiency: While HPP incurs slightly higher token usage on very short videos due to probing overhead, it becomes significantly more efficient for longer videos. On the longest tasks, HPP used up to 16× fewer tokens than standard frame sampling methods, as it avoids processing redundant frames.

5. Significance and Claims

The paper claims that decoupling semantic perception from higher-order temporal reasoning is a critical step forward for long video understanding. By externalizing reasoning into an adaptive, code-driven probing process, HPP:

Overcomes the "depth ceiling" of latent multi-step reasoning in standard VLMs.
Enables the use of smaller, more efficient VLMs without sacrificing performance on complex, long-horizon tasks.
Demonstrates that modern LLMs, when equipped with a structured interface to video data, can autonomously develop sophisticated strategies for temporal localization and evidence aggregation.

The authors acknowledge limitations, noting that performance is bounded by the coding capability of the underlying LLM and that the autonomous nature of the agent can occasionally lead to failure modes like premature termination or context drift. However, the results suggest that programmatic probing offers a scalable and effective alternative to the current monolithic VLM paradigm for long-form video analysis.