OmniVTG: A Large-Scale Dataset and Training Paradigm for Open-World Video Temporal Grounding

1. Problem Statement

Video Temporal Grounding (VTG) involves locating the precise start and end timestamps of an event described by a text query within an untrimmed video. While existing methods perform well on closed-domain datasets, they struggle significantly in open-world scenarios. The primary challenges identified are:

• Limited Semantic Diversity: Existing datasets (e.g., ActivityNet, Charades-STA) are restricted to common human activities and lack coverage of rare, abstract, or domain-specific concepts.
• Performance Gap: Models exhibit a significant performance drop when handling "rare" concepts compared to "common" ones.
• Annotation Quality: Manual annotation is expensive and unscalable, while automated pipelines (e.g., using ASR) often fail to align queries precisely with visual content.
• Reasoning Limitations: Current Multimodal Large Language Models (MLLMs) possess strong video understanding capabilities (e.g., matching, classification) but weak direct grounding abilities. Furthermore, existing Chain-of-Thought (CoT) approaches often lack explicit self-correction mechanisms, merely describing content without refining predictions.

2. Methodology

The authors propose a two-pronged solution: a new large-scale dataset (OmniVTG) and a novel Self-Correction Chain-of-Thought (CoT) training paradigm.

A. OmniVTG Dataset Construction

OmniVTG is a large-scale dataset (2,124 hours, 359k queries) constructed via a Semantic Coverage Iterative Expansion pipeline:

1. Target Word Identification: The authors analyze the vocabulary of existing VTG datasets to identify underrepresented words (gaps) using a BERT tokenizer and spell-checking.
2. Targeted Video Collection: Instead of searching for raw words (which yields irrelevant results), an LLM (Gemini-2.5-Pro) translates target concepts into specific, event-centric search keywords (e.g., translating "meticulous" to "watchmaker meticulous assembly"). These keywords are used to scrape videos from platforms like YouTube and Bilibili.
3. Caption-Centric Automated Annotation: Leveraging the observation that MLLMs are better at dense captioning with timestamps than direct grounding, the system prompts MLLMs to generate dense, timestamped captions that explicitly include the target rare words.
4. Quality Control: A subset of annotations is manually verified to create a high-quality test set.

B. Self-Correction CoT Training Paradigm

The authors propose a three-stage training pipeline to bridge the gap between video understanding and grounding:

1. Supervised Fine-Tuning (SFT): The model is trained on four tasks to build foundational skills:
   • Temporal Grounding: Predicting timestamps.
   • Event Captioning: Describing events given timestamps.
   • Query-Clip Matching: Verifying if a segment matches a query.
   • Event Status Classification: Determining if an event is "Not Started," "In Progress," or "Ended" at a specific time.
2. Self-Correction CoT Finetuning: This stage explicitly teaches the "predict-then-correct" reasoning path. The model is trained on data where it first predicts a coarse segment (Event A) and then uses its understanding capabilities to "zoom in" and refine the prediction to the correct segment (Event B). This forces the model to use its strong understanding skills to verify and correct its own localization errors.
3. Reinforcement Learning (RL): Using Group Relative Policy Optimization (GRPO), the model is further optimized on challenging examples to reinforce the time-aware self-correction reasoning structure.

3. Key Contributions

1. OmniVTG Dataset: A massive, semantic-rich dataset covering a vast vocabulary of rare and common concepts, constructed via an iterative expansion strategy that significantly outperforms existing datasets in vocabulary diversity and scale.
2. Self-Correction CoT Paradigm: A novel training framework that leverages the model's superior video understanding capabilities to iteratively refine temporal grounding predictions, effectively closing the performance gap between rare and common concepts.
3. State-of-the-Art Performance: The approach achieves SOTA results on both the new OmniVTG benchmark and four existing public VTG benchmarks (ActivityNet, Charades-STA, QVHighlights, TVGBench) in zero-shot settings.

4. Experimental Results

• Open-World Performance: On the OmniVTG test set, the proposed method achieves 63.3% R1@0.5, significantly outperforming strong baselines like Time-R1 (33.9%) and UniTime (37.1%).
• Rare Concept Robustness: The performance gap between "Rare" and "Not Rare" concepts is drastically reduced. While Time-R1's performance drops from 33.9% (All) to 29.4% (Rare), OmniVTG maintains a stable 62.4% on Rare concepts.
• Zero-Shot Generalization: The model achieves SOTA zero-shot performance on four public benchmarks without fine-tuning on them, demonstrating superior generalization compared to models trained on closed-domain data.
• Ablation Studies:
  • Removing the CoT stage leads to a significant performance drop, proving the necessity of the self-correction mechanism.
  • Content-aware reflection (using semantically similar video segments for correction) outperforms rule-based reflection.
  • The three-stage pipeline (SFT + CoT + RL) is complementary, with each stage contributing to the final performance.

5. Significance

This work addresses a critical bottleneck in video understanding: the inability of current models to generalize to open-world scenarios with diverse and rare concepts. By shifting the focus from direct grounding to understanding-driven self-correction, the authors demonstrate that MLLMs can be effectively trained to refine their own predictions. The release of OmniVTG provides a necessary benchmark for future research, moving the field beyond limited, closed-domain datasets toward truly open-world video understanding. The proposed training paradigm offers a scalable blueprint for enhancing reasoning capabilities in multimodal models.