One Trajectory, One Token: Grounded Video Tokenization via Panoptic Sub-object Trajectory

Technical Summary: One Trajectory, One Token

Problem Statement
Current video tokenization methods, which rely on splitting videos into fixed space-time patches (e.g., ViT3D), face significant scalability challenges. As video duration increases, the number of tokens grows linearly, leading to memory bottlenecks and computational inefficiencies. Furthermore, this patch-based paradigm ignores scene complexity; a static scene and a dynamic action sequence of the same duration generate an identical number of tokens. Existing token reduction strategies, such as patch dropping or merging, often degrade model performance or fail to maintain temporal coherence when camera motion is present. Additionally, these methods struggle to integrate image data seamlessly due to architectural dependencies on 3D convolutions or space-time patch mappings.

Methodology: Grounded Video Tokenization
The authors propose TrajViT, a video encoder based on a new paradigm called grounded video tokenization. Instead of fixed patches, TrajViT organizes tokens around panoptic sub-object trajectories. This approach aligns with perceptual principles (Spelke's core cognitive principles and the Gestalt Principle of common fate), treating object parts as coherent entities that persist over time.

The system consists of two primary components:

1. Panoptic Sub-object Trajectory Extraction Pipeline:

   • Key Frame Detection: The video is segmented into clips by detecting key frames where new objects are likely to appear. This is achieved by comparing color histograms and luminance intensity between consecutive frames using a parallelizable algorithm.
   • Segmentation and Tracking: Within each clip, the pipeline uses DirectSAM (a lightweight segmenter) to segment all objects in key frames. It then employs SAM2 (a tracker) to propagate these masks across subsequent frames within the clip.
   • Merging: To handle objects spanning multiple clips, the system performs an additional tracking step between the final frame of one clip and the first frame of the next. Segments with high Intersection over Union (IoU) are merged to maintain object identity across the entire video.
   • Output: This process generates a set of trajectories, where each trajectory represents a distinct object or sub-object over time.

2. Trajectory Encoder:

   • Appearance Encoding: For each trajectory, the model extracts hierarchical visual features from a lightweight convolutional network (ResNet-18). It applies mask pooling to aggregate features within the segmentation mask for each frame.
   • Temporal Position Encoding: Bounding box coordinates for the trajectory in each frame are mapped to high-dimensional vectors using sinusoidal embeddings.
   • Aggregation: Both the appearance and temporal position sequences (which vary in length) are processed by a Perceiver Resampler. This module uses a single learnable query to attend to the sequence of frame-level features, compressing them into a single fixed-size embedding.
   • Final Token: The trajectory token is formed by summing the appearance and temporal position embeddings. These tokens serve as the input to a standard transformer encoder.

Key Contributions

• New Tokenization Paradigm: The paper introduces "grounded video tokenization," where token count correlates with the number of distinct object trajectories rather than video duration or frame count. This inherently reflects scene complexity.
• TrajViT Architecture: A novel encoder that extracts panoptic sub-object trajectories and converts them into semantically meaningful, fixed-size tokens without modifying the underlying transformer architecture.
• Seamless Image-Video Integration: Unlike space-time patch methods that require separate pretraining pipelines for images and videos, TrajViT treats image segments as trajectories of length one, allowing for joint training on image and video data in a single stage.
• Efficiency: The method significantly reduces token counts (e.g., 10x reduction on the Panda dataset) while maintaining temporal coherence and robustness to lighting changes, occlusions, and camera motion.

Experimental Results
TrajViT was evaluated against standard ViT3D and state-of-the-art token merging baselines (e.g., ToMe, AutoMerge, RLT) across multiple benchmarks:

• Video-Text Retrieval: On datasets like ActivityNet, VATEX, MSR-VTT, and Charades, TrajViT outperformed ViT3D by a large margin (e.g., +6% top-5 recall on average) while using 10x fewer tokens.
• Action Classification: TrajViT achieved superior performance on Kinetics-400 and UFC-101. On Something-Something V2 (SSV2), it outperformed token merging methods but slightly trailed ViT3D, which the authors attribute to the model over-compressing complex motions in simple egocentric scenes.
• Fine-Grained Tasks: TrajViT showed significant gains in spatial-temporal action detection (AVAv2) and temporal clip retrieval, attributed to the consistency of object trajectories over time.
• VideoLLM Performance: When integrated as the video encoder for VideoLLMs (using Llama-3), TrajViT-LLM surpassed ViT3D-LLM by an average of 5.24% accuracy across six VideoQA benchmarks. It achieved this with 4x faster training time and 18x fewer inference FLOPs.
• Scaling: TrajViT demonstrated superior scaling behavior with increased input frames (up to long videos) and larger pretraining datasets (2M to 8M videos). It also benefited more from the addition of image data (DataComp-50M) compared to ViT3D.

Significance and Claims
The authors claim that TrajViT is the first efficient encoder to consistently outperform ViT3D across diverse video analysis tasks without sacrificing accuracy for efficiency. By grounding tokens in object trajectories, the method addresses the fundamental inefficiencies of patch-based tokenization, offering a robust and scalable solution for long-video understanding. The paper emphasizes that this approach allows models to focus on the semantic complexity of a scene rather than its duration, making it particularly effective for object-centric tasks and long-form video analysis. The authors also note that while the trajectory generation pipeline introduces some overhead, the overall system remains more efficient for videos longer than 64 frames.