🤖 AI

HiMu: Hierarchical Multimodal Frame Selection for Long Video Question Answering

HiMu is a training-free framework that bridges the efficiency-accuracy trade-off in long video question answering by decomposing queries into hierarchical logic trees routed to lightweight multimodal experts, enabling it to outperform both similarity-based and agent-based selectors with significantly lower computational costs.

Original authors: Dan Ben-Ami, Gabriele Serussi, Kobi Cohen, Chaim Baskin

Published 2026-03-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Dan Ben-Ami, Gabriele Serussi, Kobi Cohen, Chaim Baskin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to answer a tricky question about a 30-minute movie, but you only have time to look at 16 specific snapshots before you have to give your answer. If you pick the wrong 16 snapshots, you'll miss the plot entirely. If you pick the right ones, you can solve the mystery.

This is the challenge of Long Video Question Answering. Current AI models are like brilliant detectives with very short attention spans; they can't watch the whole movie at once. They need a "camera operator" to hand them the right frames.

The paper introduces HiMu, a new, super-smart camera operator that solves this problem without needing expensive, slow computers.

Here is how HiMu works, explained through simple analogies:

1. The Problem: The "Two Bad Options"

Currently, AI has two bad ways to pick frames:

The "Gist" Method (Similarity): Imagine asking a friend to pick 16 photos from a movie based on a vague feeling like, "Show me the part where the hero is sad." The friend picks 16 photos that look sad. But they might miss the specific moment the hero says "I'm sad" or the moment a specific object (like a broken watch) appears. It's fast, but it's shallow.
The "Over-Thinker" Method (Agentic): Imagine asking a genius detective to watch the whole movie, pause, think, rewind, and discuss the clues with a team of experts. They will find the perfect 16 frames. But it takes them 10 hours to do it. It's too slow and expensive for real-world use.

HiMu is the "Goldilocks" solution: It's as smart as the Over-Thinker but as fast as the Gist Method.

2. The HiMu Solution: The "Architect and the Specialists"

HiMu treats the question like a complex recipe that needs to be broken down. It uses a Neuro-Symbolic approach (a mix of AI logic and human-like reasoning).

Step 1: The Architect (The Text-Only Brain)

When you ask a question like, "After the narrator mentions the chemical reaction, what happens to the beaker on the left?", HiMu doesn't just scan the video.

It calls a Text-Only AI (the Architect) just once.
The Architect breaks the question down into a Logic Tree, like a flowchart.
- Branch A: Find the moment the narrator says "chemical reaction."
- Branch B: Find the moment the beaker on the left changes color.
- Branch C: Ensure Branch A happens before Branch B.

Step 2: The Specialists (The Experts)

Instead of one giant AI trying to do everything, HiMu hires a team of tiny, specialized experts to check the video for specific clues:

The Eyes (Visual Experts): One looks for objects (a red car), one looks for actions (a person running), and one reads text on signs.
The Ears (Audio Experts): This is HiMu's secret sauce. One expert listens for specific spoken words (ASR), and another listens for sound effects (like a doorbell ringing or glass breaking).
The Magic: These experts scan the entire video very quickly and create a "heat map" of when their specific clues appear.

Step 3: The Conductor (Fuzzy Logic)

Now, HiMu takes the reports from all the specialists and combines them using Fuzzy Logic (a type of math that handles "maybe" and "almost" rather than just "yes" or "no").

It checks: "Did the 'chemical reaction' sound happen? Yes. Did the 'beaker' change color? Yes. Did the sound happen before the color change? Yes."
It creates a Satisfaction Curve: A graph that shows, for every single second of the video, how well that second matches the entire complex question.

Step 4: The Photographer (PASS)

Finally, HiMu looks at the Satisfaction Curve. Instead of just grabbing the 16 highest peaks (which might all be clustered in one scene), it uses a strategy called PASS (Peak-and-Spread).

It finds the most important "peaks" (moments where the answer is most likely).
It spreads the camera out to grab those peaks and the frames immediately before and after them to get the full context.
It hands these 16 perfect frames to the main AI to answer the question.

3. Why is this a Big Deal?

It's Fast: Because the "experts" are lightweight and the "Architect" only speaks once, HiMu is incredibly efficient. It uses about 10 times less computing power than the slow "Over-Thinker" methods.
It's Smart: It understands that "After X, Y happens" requires looking at two different things in a specific order. It can listen to audio and look at video simultaneously.
It's Cheaper: You don't need a supercomputer to run it. It works on standard hardware.

The Bottom Line

Think of HiMu as a super-efficient film editor.

Old methods were like hiring a intern to guess which scenes are important (fast, but often wrong).
Other methods were like hiring a film school professor to analyze every second (perfect, but takes forever).
HiMu is like a director who instantly knows the script, sends a camera crew to film exactly the right shots, and edits them together perfectly in seconds.

The result? The AI gets the right answers, much faster, and much cheaper than ever before.

1. Problem Statement

Long-form Video Question Answering (VideoQA) faces a critical bottleneck: current Large Vision-Language Models (LVLMs) have finite context windows, making it impossible to process entire videos at native frame rates. Consequently, frame selection is required to identify the most relevant visual evidence.

Existing methods suffer from a sharp trade-off between efficiency and reasoning depth:

Similarity-based methods: Fast and computationally cheap, but they collapse complex, compositional queries (e.g., "What happens after the narrator mentions X?") into a single dense vector. This approach fails to capture sub-event ordering, cross-modal dependencies (audio-visual), and logical structures.
Agentic/Iterative methods: Achieve deep compositional understanding by iteratively calling LVLMs to score frames. However, this incurs prohibitive computational costs and latency (often 10–100× higher than similarity methods), making them impractical for real-time or multi-query applications.

The Gap: There is a lack of a method that can perform compositional, multimodal reasoning for frame selection with single-shot efficiency.

2. Methodology: HiMu Framework

HiMu (Hierarchical Multimodal Frame Selection) is a training-free, neuro-symbolic framework that bridges this gap. It decomposes complex queries into structured logic trees and evaluates them using lightweight, modality-specific experts, avoiding iterative LVLM calls during the selection phase.

The pipeline consists of four stages:

A. Neuro-Symbolic Query Decomposition

A single text-only LLM call parses the natural language question $Q$ into a hierarchical logic tree $T$ .

Leaf Nodes: Represent atomic predicates routed to specific experts:
- Visual: CLIP (actions/scenes), Open-Vocabulary Detection (OVD) (objects/people), OCR (on-screen text).
- Audio: ASR (spoken words), CLAP (environmental sounds/music).
Internal Nodes: Apply logical and temporal operators:
- Logical: AND (co-occurrence), OR (disjunction).
- Temporal: SEQ (chronological sequence), RightAfter (tight temporal adjacency).
Strategy: For multiple-choice questions, the tree typically factors out shared context (AND) and branches into options (OR).

B. Multimodal Expert Signal Extraction

Each leaf node is evaluated by a lightweight, modality-specific expert over the video timeline $t \in \{1, \dots, T\}$ .

Caching: Features for CLIP, ASR, OCR, and CLAP are query-independent and cached per video. Only OVD (which is query-conditioned) is re-run per query.
Signal Processing:
1. Normalization: Raw scores (ranging from detection confidences to cosine similarities) are mapped to $(0, 1)$ using Median Absolute Deviation (MAD) and a sigmoid function to ensure robustness against heavy-tailed distributions.
2. Smoothing: Signals are convolved with modality-specific Gaussian kernels. Visual signals use narrow kernels; audio signals (ASR/CLAP) use wider kernels to bridge temporal resolution gaps and align cross-modal events.

C. Fuzzy-Logic Composition

The logic tree is evaluated bottom-up using continuous fuzzy-logic operators to generate a per-frame satisfaction curve $T(t) \in [0, 1]$ .

Logical Operators: AND uses product t-norm; OR uses probabilistic sum.
Temporal Operators:
- SEQ: Ensures events occur in order by tracking "has-occurred" and "yet-to-occur" signals.
- RightAfter: Scores frames highly when cause and effect are temporally close, with exponential decay based on distance.
Result: A continuous curve representing how well the entire logical structure is satisfied at every timestamp.

D. PASS (Peak-And-Spread Selection)

To select the final $K$ frames, HiMu introduces PASS, which avoids the "peak concentration" problem of naive top-K selection.

Peak Detection: Identifies $N_p$ local maxima in $T(t)$ with a minimum temporal separation $\Delta$ .
Neighbor Spread: Augments each peak with its $N_n$ highest-scoring neighbors within a local window $w$ to capture short-term motion context.
Greedy Fill: Fills any remaining budget with the highest-scoring unselected frames.

3. Key Contributions

Neuro-Symbolic Framework: A novel approach that decomposes queries into hierarchical logic trees, routing atomic predicates to specialized experts and composing signals via fuzzy logic. This enables precise temporal grounding without iterative LVLM inference.
Training-Free & Plug-and-Play: HiMu requires no fine-tuning. It acts as a drop-in module for any LVLM, replacing iterative search with a single text-only LLM planning step and cached expert evaluation.
Multimodal Integration: It is the first frame selector to treat audio (ASR and non-speech sounds via CLAP) as a first-class selection modality, critical for resolving cross-modal queries.
Redefining the Pareto Front: HiMu breaks the existing efficiency-accuracy trade-off, achieving high accuracy with significantly lower computational cost than agentic methods.

4. Experimental Results

Evaluated on Video-MME, LongVideoBench, and HERBench-Lite.

Accuracy vs. Efficiency:
- With Qwen3-VL-8B and a strict 16-frame budget, HiMu outperforms all competing similarity-based (BOLT, AKS) and iterative (T*) selectors.
- Comparison with Agentic Systems: HiMu (16 frames) surpasses agentic systems like VSLS (32 frames) and VideoChat-A1 (512 frames) when paired with GPT-4o. Specifically, HiMu achieves 78.18% accuracy on Video-MME with GPT-4o, beating VSLS (67.09%) and VideoChat-A1 (62.99%).
Ablation Studies:
- Compositional Structure: Replacing the logic tree with a flat fusion of scores caused the largest accuracy drop (-5.49%), proving that how signals are composed is more critical than the signals themselves.
- Audio Importance: Removing ASR caused the second-largest drop (-1.99%), validating the necessity of audio for temporal reasoning.
- Budget Efficiency: HiMu at 16 frames outperforms uniform sampling at 64 frames, demonstrating that selecting precise frames is more effective than increasing the frame count.
Computational Cost:
- HiMu requires roughly 10× fewer FLOPs than agentic systems operating at 32–512 frames.
- While the initial preprocessing (caching) takes ~4.3s, the amortized cost per query is ~9.0s (dominated by the LLM parser and OVD), which is significantly lower than the 30–60s latency of iterative agentic loops.

5. Significance

HiMu fundamentally shifts the paradigm of long-video understanding. It demonstrates that compositional reasoning does not require expensive iterative LVLM inference. By factoring reasoning into a structured, one-shot planning step and leveraging lightweight, cached experts, HiMu achieves:

Scalability: It enables high-accuracy VideoQA on long videos within tight computational budgets.
Interpretability: Unlike black-box similarity or agent methods, HiMu provides a transparent "heat map" of which experts and predicates drove the selection of each frame, allowing for debugging and refinement.
Practicality: It offers a viable path for deploying complex VideoQA systems in real-world scenarios where latency and compute resources are constrained.

In summary, HiMu proves that precise retrieval of a small number of compositionally verified frames is superior to expanding the context window with hundreds of densely sampled, unverified frames.