Think, Then Verify: A Hypothesis-Verification Multi-Agent Framework for Long Video Understanding

The paper introduces VideoHV-Agent, a multi-agent framework that improves long video understanding by replacing reactive retrieval with a structured "think-then-verify" process where hypotheses are formulated, clues are derived, and evidence is grounded before generating a final answer, achieving state-of-the-art accuracy with enhanced interpretability and lower computational cost.

The Gist

Imagine you are trying to solve a mystery in a 3-hour movie, but you only have 5 minutes to figure out the answer.

Most current AI "detectives" try to solve this by frantically scanning the entire movie, looking for anything that might be related to the question. They shout, "I saw a red car! Maybe that's the clue!" or "I heard a dog bark! That must be it!" This approach is like searching a haystack for a needle by grabbing random handfuls of hay. It's slow, confusing, and often leads to wrong answers because the AI gets distracted by irrelevant details.

VideoHV-Agent is a new, smarter detective. Instead of frantically searching, it follows a simple rule: "Think before you find."

Here is how it works, broken down into a simple story:

1. The Setup: The "Clue Board"

Imagine the AI has a whiteboard. First, it quickly skims the movie and writes a short summary on the board. It doesn't read every single word of the script; it just gets the "gist" of the story.

2. The Four Specialists (The Agents)

Instead of one confused detective, this system uses a team of four specialists who work together:

• The Thinker (The Strategist):
  The Thinker looks at the question and the possible answers. Instead of guessing, it asks: "If Answer A is true, what must happen in the movie? If Answer B is true, what must happen?"

  • Analogy: It's like a lawyer building a case. "If the suspect is innocent, he must have been at the park. If he is guilty, he must have been at the bank." The Thinker turns vague answers into specific, testable predictions (Hypotheses).

• The Judge (The Filter):
  The Judge looks at all the Thinker's predictions and says, "Okay, we have too many ideas. What is the one single thing we need to look for to tell them apart?"

  • Analogy: If the Thinker says "Look for a red car" and "Look for a blue car," the Judge says, "No, just look for which car is moving." It creates a sharp, focused "Clue" so the team doesn't waste time looking at the wrong things.

• The Verifier (The Investigator):
  This is the only one who actually goes back to the movie. But it doesn't watch the whole thing! It only watches the specific 5-second clip where the "Clue" is likely to appear. It checks the footage closely.

  • Analogy: Instead of searching the whole city, the Verifier goes straight to the specific street corner the Judge pointed to. It checks: "Is the car moving? Yes. Is it red? No." It gathers hard evidence.

• The Answer Agent (The Judge's Verdict):
  Once the Verifier brings back the evidence, the Answer Agent combines it with the summary and the original question to declare the final winner. It doesn't guess; it decides based on proof.

3. The "Self-Correction" Loop

What if the Verifier looks at the clip and says, "I can't see enough to be sure"?
Old systems would just guess and hope for the best. VideoHV-Agent says, "Okay, my hypothesis was too vague. Let's try again." It refines the question, looks at a different part of the movie, and tries again until it finds the truth.

Why is this better?

• No More Hallucinations: Because it demands proof before deciding, it doesn't make up facts.
• Speed: It doesn't waste time watching the whole movie. It only watches the 10 seconds that matter.
• Logic: It follows a clear chain of reasoning (If X, then Y. Let's check Y.) rather than just guessing based on what words sound similar.

In short: VideoHV-Agent stops the AI from being a frantic scavenger and turns it into a methodical detective who plans the investigation, identifies the critical clue, checks the evidence, and only then delivers the verdict.

Technical Summary

1. Problem Statement

Long-form Video Question Answering (VideoQA) faces three primary challenges:

• Dense Visual Redundancy: Long videos contain vast amounts of irrelevant or repetitive content, making it computationally prohibitive to process every frame.
• Semantic Drift & Error Accumulation: Existing Chain-of-Thought (CoT) and retrieval-based agents often rely on reactive, correlation-driven searches. They iteratively search for clips related to the current plan, which leads to early retrieval errors propagating through subsequent reasoning steps.
• Lack of Deliberate Planning: Current methods often search for evidence before defining what specific evidence is needed. They fail to articulate the logical conditions required for an answer to be true, leading to "trial-and-error" cycles that are inefficient and prone to hallucinations.

The authors argue that effective long-video reasoning must shift from correlation-based search to deliberate task formulation ("thinking before finding").

2. Methodology: VideoHV-Agent

The paper proposes VideoHV-Agent, a multi-agent framework that reframes VideoQA as a structured Hypothesis–Verification process. The framework operates through three main stages:

A. Context Summarization

Instead of feeding raw frame captions directly into the reasoning loop, the system first generates a query-conditioned video summary.

• Frame-level captions are generated via a captioning model.
• A compact summary ($P_s$) is derived, conditioned on the question.
• Design Choice: This decouples global reasoning (using the summary) from local grounding (using detailed frame captions), reducing computational load while preserving necessary detail.

B. Two-Step Reasoning Pipeline

The core innovation is a two-stage reasoning loop involving four specialized agents:

1. Hypothesis Generation (Thinker & Judge Agents):

   • Thinker: Takes the video summary and candidate answer options. It rewrites each option into an explicit, testable hypothesis ($h_i$). These hypotheses specify exactly what entities, actions, and temporal/causal constraints must be true in the video for that answer to hold.
   • Judge: Evaluates the set of hypotheses and generates a discriminative clue ($\kappa$). This clue identifies the minimal visual observation required to distinguish between competing hypotheses (e.g., "Check if the tool is a sewing machine or scissors").

2. Hypothesis Verification (Verifier Agent):

   • Temporal Localization: The Verifier uses the clue to locate the specific temporal window in the video where the evidence is likely to exist.
   • Detailed Captioning: It retrieves only the relevant frames within that window and invokes fine-grained captioning (processing up to 5 frames at a time) to extract detailed evidence.
   • Status Determination: The agent outputs a structured status:
     • VERIFIED: Evidence supports the hypothesis.
     • PARTIAL: Partial evidence found; requires more.
     • NOT VERIFIED: Clue is insufficient or contradicted; requires regeneration.

3. Self-Refinement Loop:

   • If the verification is inconclusive (NOT VERIFIED or PARTIAL), the system triggers a self-refinement loop.
   • It can enhance specificity (making hypotheses more concrete) or discriminability (sharpening the contrast between options) and re-run the verification. This mimics human revision of hypotheses.

C. Evidence Integration (Answer Agent)

Once a hypothesis is verified, the Answer Agent integrates the validated evidence with the global summary. It constructs a transparent reasoning chain, explicitly stating what was tested, observed, and how it supports the final decision.

3. Key Contributions

1. Hypothesis-Verification Paradigm: A shift from reactive, correlation-based retrieval to a proactive "think-then-verify" approach where the model defines what must be true before searching for evidence.
2. Multi-Agent Architecture: Implementation of a specialized pipeline (Thinker, Judge, Verifier, Answer) that separates hypothesis formulation, clue distillation, evidence grounding, and final decision-making.
3. Robustness via Self-Refinement: A mechanism to detect uncertainty and iteratively refine hypotheses and clues, preventing error propagation common in linear CoT approaches.
4. Efficiency: By narrowing the search to specific temporal windows based on discriminative clues, the system avoids scanning the entire video repeatedly.

4. Experimental Results

The framework was evaluated on three benchmarks: EgoSchema, NextQA, and IntentQA.

• State-of-the-Art Performance: VideoHV-Agent achieved the highest accuracy among zero-shot methods on all datasets.
  • EgoSchema: 81.0% (vs. 80.6% for VideoAgent2).
  • NextQA: 80.7% overall, with a significant improvement to 71.2% on the difficult "ATP-hard" subset (vs. 68.2% for VideoAgent2).
  • IntentQA: 75.6% (vs. 73.9% for VideoAgent2).
• Efficiency: Despite the multi-step reasoning, VideoHV-Agent demonstrated lower latency (123.66s) compared to other agent-based methods like VideoTree (160.21s) and VideoMultiAgents (134.90s), due to its targeted retrieval strategy.
• Ablation Studies:
  • Removing the Hypothesis module dropped accuracy by 5%.
  • Removing the Clue module dropped accuracy by 2.4%.
  • Removing the Verification Status (self-refinement) caused a 7% drop, proving the necessity of adaptive refinement.
• Question Type Analysis: The model showed superior performance across Causal, Temporal, and Descriptive question types, particularly excelling in complex causal reasoning.

5. Significance

This paper addresses a critical bottleneck in long-video understanding: the tendency of LLMs to hallucinate or drift when reasoning over long contexts without a structured plan.

• Logical Soundness: By forcing the model to articulate what needs to be true before finding it, the framework reduces semantic drift and ensures answers are grounded in explicit visual evidence.
• Interpretability: The framework provides a transparent reasoning chain (Hypothesis $\to$ Clue $\to$ Evidence $\to$ Verification Status), making the decision process explainable.
• Scalability: The "hypothesis-driven" search significantly reduces computational costs by avoiding redundant frame processing, making it a viable solution for very long videos where previous methods fail due to context limits or inference time.

In summary, VideoHV-Agent demonstrates that structured, hypothesis-driven reasoning is superior to reactive retrieval for complex long-video tasks, offering a new paradigm for building robust and efficient video understanding agents.