Video Understanding: From Geometry and Semantics to Unified Models

Based on the provided paper, here is a detailed technical summary of the survey "Video Understanding: From Geometry and Semantics to Unified Models."

1. Problem Statement

Video understanding is a foundational challenge in computer vision that goes beyond static image analysis. Unlike images, videos inherently involve temporal continuity, viewpoint changes, motion-induced occlusions, and long-range dependencies. The core problem is enabling models to not only perceive visual content but also reason about the dynamic evolution of the world.

The field has historically evolved along two somewhat isolated tracks:

1. Low-level Geometry: Recovering physical structure (depth, camera pose, optical flow).
2. High-level Semantics: Recognizing objects, actions, and events.

However, real-world applications increasingly demand systems that can simultaneously handle geometric consistency and semantic reasoning. The paper addresses the need for a structured overview that bridges these domains and highlights the emerging paradigm of Unified Video Understanding Models, which integrate geometry, semantics, and generation into a single framework.

2. Methodology and Survey Structure

The paper does not propose a single new algorithm but provides a comprehensive taxonomy and literature review organized into three complementary perspectives:

A. Low-level Video Geometry Understanding

This section focuses on recovering physically grounded representations directly from RGB videos.

• Video Depth Estimation: Moves from inference-time alignment (optimizing per-video) to feed-forward prediction (amortized temporal reasoning) and diffusion-based approaches (leveraging generative priors for temporal coherence).
• Camera Pose Estimation: Contrasts correspondence-and-solver pipelines (high precision, geometric guarantees) with pose regression (end-to-end, efficient, suitable for real-time). It highlights the shift from Absolute Pose Regression (APR) to Relative Pose Regression (RPR) for better generalization.
• Optical Flow & Point Tracking: Discusses the evolution from short-range optical flow to Tracking Any Point (TAP) formulations. Key advancements include global reasoning, memory mechanisms for long-horizon tracking, and the use of foundation models (e.g., DINOv2) for robust correspondence.
• Joint Feed-Forward Geometry Models: A critical trend where single networks (e.g., DUSt3R, VGGT) jointly predict multiple primitives (depth, pose, point maps) in a single forward pass, enabling fast, consistent 3D reconstruction without heavy test-time optimization.

B. High-level Semantic Understanding

This section covers tasks requiring spatiotemporal reasoning and cross-modal grounding.

• Video Segmentation: Categorizes methods into class-aware (fixed labels), open-vocabulary (leveraging Vision-Language Models for unseen classes), and class-agnostic (prompt-based, e.g., SAM2).
• Video Object Tracking: Traces the shift from pairwise appearance matching (Siamese networks) to sequence-level modeling (Transformers) and multimodal tracking (integrating depth, thermal, or text to handle occlusion and appearance changes). It emphasizes the need for robustness to missing modalities.
• Video Temporal Grounding (VTG): Reviews the transition from supervised proposal-based methods to MLLM-based approaches. Modern methods utilize pre-trained Large Multimodal Models (LMMs) for zero-shot grounding or employ reinforcement learning and chain-of-thought reasoning to align natural language queries with specific video segments.

C. Unified Video Understanding Models

The final section explores models that integrate geometry, semantics, and generation.

• Video Question Answering (VideoQA): Discusses the shift from static pattern recognition to cognitive video understanding. New benchmarks (e.g., EgoSchema, CinePile) test long-form narrative comprehension and "Spatial Supersensing" (predictive world modeling). Architectures are evolving toward hybrid SSM-Transformer designs to handle long contexts efficiently.
• Unified Understanding & Generation: Analyzes three paradigms for unifying perception and synthesis:
  1. Assembled Systems: LLMs orchestrating external expert models (modular but not end-to-end optimized).
  2. Autoregressive (AR) Unified Models: Mapping text, images, and video to a single token stream (e.g., Video-LaVIT, Emu3).
  3. Hybrid Unified Models: Combining AR decoding with diffusion/flow matching objectives to achieve high-fidelity generation while maintaining instruction following (e.g., Show-o2, TUNA).

3. Key Contributions

• Structured Taxonomy: The paper organizes the vast literature of video understanding into a coherent map connecting low-level geometry, high-level semantics, and unified modeling.
• Identification of the "Unified" Paradigm: It argues that the future of video foundation models lies in systems that simultaneously reason about physical structure (geometry) and semantic meaning, rather than treating them as separate tasks.
• Comprehensive Benchmarking: The survey includes detailed comparisons of state-of-the-art methods across multiple datasets (KITTI, 7 Scenes, DAVIS, etc.) and highlights the performance trade-offs between different architectural choices (e.g., feed-forward vs. diffusion, AR vs. Hybrid).
• Future Directions: It identifies critical open challenges, including the need for active predictive world models, memory-efficient architectures for long-horizon videos, and uncertainty-aware planning for embodied agents.

4. Results and Findings

• Geometry: Joint feed-forward models (like DUSt3R and VGGT) have achieved state-of-the-art results in 3D reconstruction and depth estimation by learning shared representations across multiple geometric tasks, often outperforming task-specific pipelines in generalization.
• Semantics: The integration of Vision-Language Models (VLMs) has revolutionized open-vocabulary segmentation and temporal grounding, enabling zero-shot capabilities that were previously impossible with supervised models.
• Unification: While unified models show promise, they face significant challenges in balancing temporal fidelity with instruction following. Pure AR models struggle with long-video coherence, while hybrid models (AR + Diffusion) offer better synthesis quality but introduce training complexity.
• Robustness: Multimodal tracking and grounding models demonstrate that incorporating auxiliary sensors (depth, events) or text significantly improves robustness against occlusion and appearance changes compared to RGB-only systems.

5. Significance

This survey is significant because it:

1. Bridges the Gap: It connects traditionally separate subfields (geometry vs. semantics) to show how they are converging in modern foundation models.
2. Guides Future Research: By highlighting the limitations of current approaches (e.g., "bag-of-features" in encoders, hallucination in generative models), it sets a clear agenda for developing World Models that can actively predict and reason about the future.
3. Practical Relevance: It underscores the necessity of unified systems for real-world applications like robotics, autonomous driving, and interactive media, where systems must understand what is happening, where it is happening in 3D space, and how to interact with or generate the scene.
4. Foundational Shift: It marks a transition from isolated task performance to holistic, reasoning-capable video agents capable of continuous interaction with dynamic environments.