Technical Summary: VideoSEAL

Problem: Evidence Misalignment in Agentic Long Video Understanding

Long-video understanding (LVU) presents a unique challenge: relevant visual evidence is often sparse and temporally scattered within highly redundant content. While current Multimodal Large Language Models (MLLMs) excel at short-video reasoning, extending them to long videos requires iterative, agentic interaction to navigate timelines and retrieve evidence.

The authors identify a critical failure mode in existing agentic LVU systems termed evidence misalignment. In this regime, agents produce correct answers that are not actually supported by the retrieved or inspected evidence. The paper formalizes this as the "correct but ungrounded" state, where an agent achieves outcome correctness ($C=1$) but lacks trace groundedness ($G=0$). This phenomenon is driven by two systemic pressures:

1. Prompt Pressure (Inference-time): As interaction traces grow longer and noisier, the planner is incentivized to be decisive without reliably locating evidence. This shifts behavior from "evidence seeking" to "evidence fitting," where the agent relies on parametric priors or plausible templates to resolve uncertainty, leading to semantic hallucinations.
2. Reward Pressure (Training-time): Standard reinforcement learning (RL) for LVU often employs outcome-only rewards (correctness of the final answer) due to the scarcity of fine-grained temporal grounding labels. This encourages agents to maximize returns via speculative completion or shortcut behaviors rather than rigorous evidence verification, widening the gap between accuracy and groundedness.

The authors argue that the root cause of these pressures is the coupled agent paradigm, where a single monolithic model is responsible for both long-horizon planning (search) and answer generation (authority) within the same shared context.

Methodology: The Decoupled Planner–Inspector Framework

To address evidence misalignment, VideoSEAL proposes a structural remedy: decoupling long-horizon planning from answer authority. The framework introduces two distinct entities:

1. The Planner: A language model policy responsible solely for evidence seeking. It navigates the video timeline, formulates queries, and retrieves candidate temporal spans using tools (offline indexing, retrieval, and visual inspection). Crucially, the planner does not generate the final answer; it maintains a sparse, structured search memory to avoid context saturation.
2. The Inspector: A frozen MLLM that holds exclusive answer authority. It receives candidate video evidence submitted by the planner and performs pixel-level verification. The inspector outputs a binary sufficiency verdict ($z_t \in \{0, 1\}$):
   • If $z_t=1$, the evidence is deemed sufficient, and the inspector generates the final answer.
   • If $z_t=0$, the inspector returns a "SEARCH MORE" signal with feedback on missing evidence, forcing the planner to continue searching.

Key Technical Components:

• Tool Interfaces: The system utilizes VisualRetrieve (semantic search over pre-computed captions/embeddings) and VisualInspect (pixel-level verification of specific spans).
• Training: The planner is optimized using Group Relative Policy Optimization (GRPO) while the inspector remains frozen. The reward design includes an evidence-gated terminal reward that scales with the temporal overlap between accessed spans and ground-truth evidence, encouraging alignment without requiring full retraining of the visual backbone.
• Modularity: The architecture supports "plug-and-play" upgrades. The inspector can be swapped for a stronger MLLM at inference time to improve verification fidelity without retraining the planner.

Key Contributions

1. Evidence Misalignment Diagnostics: The paper introduces two complementary diagnostics to quantify misalignment:
   • Temporal Groundedness ($G_t$): Measures whether the agent accesses relevant temporal regions (using temporal IoU).
   • Semantic Groundedness ($G_s$): Uses an LLM judge to verify if the final answer is logically supported by the tool outputs in the trace.
     These diagnostics reveal the "correct but ungrounded" regime and quantify the "temporal hallucination rate" ($H_t$) and "semantic hallucination rate" ($H_s$).
2. Decoupled Planner–Inspector Framework: A novel architecture that separates evidence seeking from answer generation. By gating the final answer on pixel-level verification, the framework prevents unsupported answers driven by unverified tool outputs.
3. Empirical Gains and Scaling: The framework demonstrates consistent improvements across four long-video benchmarks (MLVU, LongVideoBench, LVBench, VideoMME). Notably, it scales monotonically with larger search budgets and stronger inspector models, whereas coupled baselines often plateau or degrade due to context saturation.

Results

The authors evaluate VideoSEAL against proprietary models, open-source MLLMs, and existing agentic frameworks (both coupled and decoupled).

• Accuracy: VideoSEAL achieves 55.1% on LVBench and 62.0% on LongVideoBench, outperforming coupled baselines by significant margins (e.g., +20.5% on LVBench over the coupled version of the same backbone).
• Grounding: The framework significantly reduces hallucination. On LVBench, it achieves a semantic groundedness ($G_s$) of 0.808 (compared to ~0.25–0.47 for baselines) and reduces semantic hallucination ($H_s$) to 0.113.
• Scaling Properties:
  • Search Scaling: Increasing the search budget ($K$) improves performance for VideoSEAL, while coupled agents regress after a certain point due to context saturation.
  • Perceptual Scaling: Upgrading the inspector from a 7B to a 72B model yields substantial accuracy gains without retraining the planner, demonstrating the modularity of the approach.
• Refusal Capability: The inspector effectively distinguishes between sufficient and insufficient evidence, refusing to answer (outputting $z=0$) on non-target clips significantly more often than on ground-truth clips, thereby reducing guessing.

Significance and Claims

The paper claims that the primary driver of performance in agentic long-video QA is not merely larger models or better reward functions, but the architectural separation of planning and answer authority. By decoupling these roles, VideoSEAL mitigates the "prompt pressure" and "reward pressure" that lead to evidence misalignment.

The authors position this work as a shift in paradigm: moving away from training a single coupled agent to search and decide, toward a verifiable, modular system where answer generation is strictly gated by visual evidence. This approach enhances interpretability (via search trajectories) and verifiability, offering a structural solution to the "correct but ungrounded" problem that plagues current long-video agents. The work suggests that future improvements in LVU may depend more on the fidelity of pixel-level inspection (perception) than on the sophistication of linguistic planning (reasoning), provided the planning logic is sound.