Technical Summary: TIMEPROVE

Problem Statement

Long Video Question Answering (LVQA) in Activities of Daily Living (ADL) requires identifying sparse, query-relevant evidence within hours-long, untrimmed videos. Existing approaches face a fundamental trade-off:

1. Dense Processing: Applying large Vision-Language Models (VLMs) directly to full videos incurs prohibitive computational costs and latency due to the massive number of visual tokens generated (e.g., ~2.6 million tokens for a 60-minute video at 1 FPS).
2. Sparse Captioning: Converting videos to text captions first is computationally cheaper but often fails to preserve fine-grained, temporally localized, and motion-centric evidence (e.g., subtle hand-object interactions or brief actions like taking medication). Once this evidence is lost in the text, downstream reasoning cannot recover it.

Furthermore, existing LVQA benchmarks often rely on multiple-choice formats that implicitly guide evidence selection, failing to explicitly evaluate a model's ability to ground answers in specific temporal intervals within unstructured, real-world environments.

Methodology: TIMEPROVE

The authors propose TIMEPROVE (Time-aware Proposal and Verification), a cost-efficient hybrid framework that performs lightweight temporal reasoning to generate hypotheses and reserves expensive VLM inference for targeted verification. The framework operates in two primary stages:

1. Action-based Candidate Evidence (ACE) Module

This lightweight component processes the full video in a single pass to generate query-conditioned hypotheses without invoking a large VLM.

• Action Detector: A lightweight temporal action detector (e.g., MS-Temba) processes the video to produce a sparse temporal action timeline, identifying what actions occur and their precise start/end times.
• Query-conditioned Proposal Generator: An edge LLM (e.g., Gemma-2B or Qwen2-7B) reasons over the action timeline and the user query to generate candidate answers paired with supporting evidence windows.
  • Atomic Windows: Direct mapping of detected actions to evidence windows.
  • Merged Windows: The LLM groups adjacent or overlapping actions to form broader temporal contexts when the query requires composite reasoning (e.g., "What did they do while drinking?").
• Scoring and Reranking: A local scoring function ranks candidates based on four criteria:
  • Temporal Compatibility: Alignment with the query's temporal intent (e.g., "first," "last," "between").
  • Semantic Relevance: Strongest action-level match to the query.
  • Coverage: Extent to which the window covers query concepts collectively.
  • Length Penalty: Penalizing unnecessarily long clips to maintain efficiency.
    The top-ranked hypothesis is selected for verification.

2. Temporal Verifier

This component invokes an expensive cloud VLM (e.g., GPT-4o or VideoLLaMA3) only on the short, selected RGB evidence clips.

• The verifier receives the query, the candidate answer, and the specific video clip.
• It determines if the clip contains sufficient visual evidence to support the hypothesis.
• If verified, the system returns the answer with semantic and visual provenance. If rejected, it proceeds to the next candidate in the ranked list.

This design supports hybrid deployment, where ACE runs locally on edge devices and the verifier utilizes remote, powerful VLMs.

Key Contributions

1. TIMEPROVE Framework: A novel hybrid architecture that decouples broad temporal proposal generation (lightweight) from fine-grained visual verification (expensive), avoiding full-video VLM inference.
2. ACE Module: The first module to transform detected action timelines into query-conditioned answer-evidence candidates using lightweight LLM reasoning and structured reranking.
3. OPENTSUBENCH (OTB): An open-ended benchmark for temporally grounded LVQA in real-world ADL scenarios, built on the Toyota Smarthome Untrimmed (TSU) dataset. It evaluates both answer correctness and the precision of temporal evidence localization.
4. Generalization: Demonstration that the framework's intermediate representation (localized evidence windows) is transferable to temporal grounding tasks, achieving competitive results on CHARADES-STA without explicit grounding training.

Experimental Results

Evaluated on the new OPENTSUBENCH benchmark:

• Performance: TIMEPROVE outperforms the strongest baseline (VideoAgent) by 7.3% in overall accuracy.
• Efficiency: The framework reduces VLM invocations by 75% and lowers inference costs by 93% compared to baselines that process longer video durations or rely on dense sampling.
• Ablation: Removing the Action Detector, Edge LLM, or Scoring module significantly degrades performance, validating the necessity of the action-grounded hypothesis generation and structured reranking.
• Temporal Grounding: On the CHARADES-STA dataset, TIMEPROVE (enhanced with stronger grounding backbones like Time-R1) achieves state-of-the-art performance, demonstrating robustness beyond open-ended LVQA.

Significance and Claims

The paper claims that TIMEPROVE addresses the scalability bottleneck of long-video understanding by shifting the paradigm from "process everything" to "propose and verify." By grounding reasoning in a learned action prior, the system preserves fine-grained visual evidence that caption-based methods miss while avoiding the token explosion of full-video VLM processing.

The authors emphasize that their approach is particularly significant for real-world ADL scenarios, where answers often hinge on brief, subtle moments that are easily lost in dense token compression or sparse text summaries. The introduction of OPENTSUBENCH is highlighted as a critical step toward evaluating models on open-ended reasoning where temporal grounding is explicit and essential, rather than implicit through multiple-choice options.

The paper modestly notes limitations: the framework assumes evidence can be captured within localized action windows, which may not suffice for questions requiring diffuse scene understanding over very long intervals. Additionally, the final verification step remains dependent on the visual reasoning capabilities of the chosen VLM.