How Should Video LLMs Output Time? An Analysis of Efficient Temporal Grounding Paradigms

1. Problem Statement

Video Temporal Grounding (VTG) involves localizing specific temporal segments in a video that match a natural language query. While Multimodal Large Language Models (MLLMs) have advanced this field, existing research suffers from two critical limitations:

• Lack of Controlled Comparison: Existing methods use diverse backbones, datasets, and training protocols, making it impossible to isolate the specific impact of the output formulation (how time is represented) on performance.
• Efficiency vs. Accuracy Trade-off: As VTG systems move toward resource-constrained edge deployment (e.g., mobile, real-time analytics), there is no systematic understanding of how different output paradigms affect computational cost, inference latency, and training throughput, particularly when paired with compact models.

The paper addresses the core question: How should temporal predictions be formulated in the model's output space to maximize both accuracy and efficiency?

2. Methodology

The authors conducted a controlled empirical study comparing three dominant VTG output paradigms under a unified framework.

A. Experimental Setup

• Backbones: Five compact VLMs ranging from 0.5B to 8B parameters (SmolVLM2, FastVLM, Molmo2). This covers resource-constrained scenarios up to a standard 8B control.
• Training Protocol: Identical datasets (~1.2M samples from 11 sources), LoRA fine-tuning (rank 16), optimizer settings, and hardware (NVIDIA H200). The only variable was the output head architecture and its associated loss function.
• Datasets: Evaluated on Charades-STA (moment retrieval), QVHighlights (highlight detection), and YouCook2 (dense video captioning).

B. The Three Paradigms Compared

1. Text Numeral Generation (Text):

   • Mechanism: Treats timestamps as standard text tokens (e.g., "52.0 to 63.0 seconds") generated autoregressively using the LLM's native vocabulary.
   • Pros: No architectural modifications; parameter-free.
   • Cons: Entangles temporal semantics with general number tokens; relies on the LLM's intrinsic numerical reasoning.

2. Temporal Token Generation (Gen):

   • Mechanism: Introduces a dedicated vocabulary of temporal tokens and a specialized head (e.g., TRACE-style). It decodes events autoregressively as structured sequences (timestamps, scores, captions) using a restricted vocabulary.
   • Pros: Separates time from general text; supports structured outputs like saliency scores.
   • Cons: Requires sequential, autoregressive decoding of discrete tokens, leading to high inference latency.

3. Continuous Temporal Decoding (Cont):

   • Mechanism: Maps hidden states directly to a continuous probability distribution over time bins (e.g., DisTime-style). A lightweight MLP head predicts a distribution, and the final timestamp is derived via expectation (weighted sum).
   • Pros: Non-autoregressive (single forward pass); explicitly models boundary ambiguity; avoids quantization errors.
   • Cons: Requires a small MLP head; does not natively output discrete tokens for saliency without modification.

3. Key Contributions

• Unified Benchmark: First controlled study isolating output formulation as the sole variable across a spectrum of model sizes (0.5B–8B).
• Efficiency-Accuracy Analysis: Comprehensive evaluation of localization accuracy alongside system-level metrics: inference latency, training throughput, parameter overhead, and memory usage.
• The "Paradigm Dividend": Discovery that optimizing the output formulation yields performance gains that are orthogonal to model scaling. A small 1.5B model with continuous decoding outperforms established 7B+ models using inferior paradigms.

4. Key Results & Findings

A. Accuracy Dominance of Continuous Decoding

• Superior Performance: The Continuous Temporal (Cont.) paradigm consistently achieved the highest accuracy across all tasks and model sizes.
  • Example: On Charades-STA with an 8B backbone, Cont. achieved 57.1% mIoU, whereas the Text paradigm plateaued at 27.5%.
  • Scaling: The Cont. paradigm at 0.5B (43.4% mIoU) outperformed the Text paradigm scaled to 8B (27.5% mIoU).
• Token Generation Limitations: While the Gen. paradigm offered structured outputs (like saliency scores), it lagged behind Cont. in strict boundary precision and suffered from high latency due to autoregressive decoding.

B. Efficiency and Latency

• Latency: The Cont. paradigm was the most efficient for inference.
  • Reason: It uses a non-autoregressive MLP decoder, predicting boundaries in a single forward pass.
  • Comparison: The Gen. paradigm incurred extreme latency penalties (e.g., ~1379 ms/query at 2.2B) due to sequential token generation. The Text paradigm also suffered delays from decoding long numerical strings.
• Pareto Frontier: The Cont. paradigm dominated the Efficiency-Accuracy Pareto frontier, offering the best trade-off for real-time deployment.

C. Robustness and Error Analysis

• Hallucination Reduction: Qualitative analysis revealed that Text and Gen. paradigms frequently suffered from Temporal Hallucinations (predicting disjoint, incorrect timeframes). The Cont. paradigm drastically suppressed this, shifting errors to benign Boundary Jitters (slightly off but covering the correct event).
• Data Efficiency: The Cont. paradigm required significantly less data to converge. Fine-tuned on only 25% of the training corpus, it outperformed Text and Gen. paradigms trained on 100% of the data.
• Visual Compression: The Cont. paradigm remained robust even when visual token counts were aggressively reduced (e.g., FastVLM vs. SmolVLM2), whereas Text performance degraded.

5. Significance and Implications

• Redefining Deployment Strategy: The paper argues that for resource-constrained edge devices, optimizing the output formulation is more effective than simply scaling up model parameters. A compact model with a continuous decoder is superior to a massive model with a text-based decoder.
• Design Guidelines: For practitioners building VTG systems, the study recommends Continuous Temporal Decoding as the default choice for moment retrieval and localization tasks due to its superior accuracy, speed, and robustness.
• Future Directions: The authors suggest that while output optimization is critical, future work must address input-side encoding (e.g., learnable position embeddings) to handle complex causal reasoning, which remains a failure point for all current paradigms. Additionally, integrating a lightweight saliency branch into the continuous decoder could unify moment retrieval and highlight detection.

In conclusion, this paper provides objective empirical evidence that Continuous Temporal Decoding is the optimal paradigm for Video LLMs, offering a "paradigm dividend" that enables highly efficient, accurate, and deployment-ready temporal grounding systems.