Technical Summary: Versatile Video Representation via Feed-Forward 2D Gaussian Splatting Tokenization

Problem Statement

Existing video representation methods, particularly those relying on fixed-grid, patch-wise tokenization (e.g., MAGVIT-v2), suffer from limited versatility in both spatial and temporal domains.

• Spatial Limitations: Uniform allocation of a fixed number of tokens leads to over-encoding in low-information regions (e.g., smooth backgrounds) while potentially under-representing high-detail areas.
• Temporal Limitations: Current approaches process time-steps uniformly without explicitly distinguishing between static content (e.g., backgrounds) and dynamic content (e.g., moving objects). This results in redundant token budgets for unchanging segments, inflating the overall token count and computational cost.
• Generalization Issues: While recent Gaussian Splatting (GS) methods show promise, they often rely on per-video optimization, which is computationally expensive and limits generalizability to unseen videos.

Methodology: Gaussian Video Transformer (GVT)

The authors propose the Gaussian Video Transformer (GVT), a framework that replaces rigid patch-wise tokenization with a feed-forward 2D Gaussian Splatting (2DGS) scheme. The core architecture consists of three main stages:

1. Spatio-Temporal Gaussian Embedding (STGE)

Instead of quantizing rigid latent vectors directly, GVT projects latent video features into a set of 2D Gaussians in a feed-forward manner.

• Input: A video clip is first encoded into a sequence of latent tensors $Z$ using a standard patch-wise feature extractor (causal 3D CNN).
• Mechanism: The STGE module utilizes two key sub-modules:
  • Spatio-Temporal Attention (STA): Aggregates features to capture global correlations.
  • Deformable Spatio-Temporal Fusion (DSTF): Uses learnable queries to project information from latent vectors into specific 2D Gaussian parameters (position $\mu$, covariance $\Sigma$, and feature coefficients $\varsigma$).
• Output: A set of 2D Gaussians $G = \{G_t\}$ for each time step, where each Gaussian explicitly models local spatial structure and varying levels of detail.

2. Gaussian Set Partitioning (GSP)

To address temporal redundancy, GVT introduces a strategy to disentangle static and dynamic content.

• Partitioning: A learnable binary mask is generated to classify Gaussians into two sets:
  • Static Gaussians ($G_{static}$): Represent stationary regions (e.g., backgrounds) shared across all time steps. These are computed once (at $t=1$) and reused for subsequent frames.
  • Dynamic Gaussians ($G_{dynamic}$): Represent temporally varying content (e.g., moving objects) and are updated for each time step.
• Optimization: A constraint loss ($L_{GSP}$) regularizes the proportion of dynamic Gaussians, encouraging the model to minimize redundancy while maintaining a learnable token count.

3. Vector Quantization and Reconstruction

• Quantization: The feature coefficients ($\varsigma$) of the 2D Gaussians are quantized using a Vector Quantization (VQ) strategy with a learnable codebook, converting continuous parameters into discrete tokens.
• Rendering: During reconstruction, the quantized Gaussians are rasterized. Static Gaussians are duplicated across all time steps, while dynamic Gaussians are rendered per frame. The final video is reconstructed by decoding the aggregated radiance weights and coefficients back into pixel space.

Key Contributions

1. STGE (Spatio-Temporal Gaussian Embedding): A novel projection module that maps compact latent video representations to 2D Gaussian Splatting in a feed-forward manner, enabling versatile tokenization without per-video optimization.
2. GSP (Gaussian Set Partitioning): A learnable strategy that explicitly partitions Gaussians into static and dynamic sets. This reduces temporal redundancy by reusing static content, enhancing both temporal versatility and representation compactness.
3. GVT Framework: An end-to-end video tokenization framework integrating STGE and GSP. It is the first work to incorporate Gaussian Splatting into video tokenization research, achieving state-of-the-art (SOTA) performance in reconstruction and compression.

Experimental Results

The authors evaluated GVT on four tasks: video reconstruction, action recognition, compression, and generation, using datasets including UCF101, Kinetics (K600/K400), and DAVIS.

• Video Reconstruction:
  • On UCF101 and Kinetics-600, GVT achieved SOTA performance in terms of reconstructed video FVD (rFVD), outperforming MAGVIT-v2 and other tokenizers (e.g., 12.6 rFVD on UCF101 vs. 16.7 for MAGVIT-v2).
  • On DAVIS-2017-480P, GVT demonstrated superior instance-level fidelity (best LPIPS, third-best SSIM) among Radiance Field-based methods. Crucially, it is the only feed-forward method in this comparison, avoiding the hours-long per-video fitting times required by NeRF or 4DGS-based approaches.
• Video Action Recognition:
  • Using reconstructed videos as input for VideoMAE, GVT achieved higher Top-1 and Top-5 accuracy on UCF101 and K400 compared to MAGVIT-v2, indicating better preservation of semantic information.
• Video Compression:
  • GVT demonstrated superior Rate-Distortion (RD) performance. Compared to the H.266/VVC standard, GVT achieved average bit-rate reductions of 54.8% (based on DISTS) and 2.1% (based on LPIPS).
• Video Generation:
  • In unconditional video generation tasks on UCF101, GVT achieved an FVD of 1240, outperforming the MAGVIT-v2 baseline equipped with the same flow-matching paradigm (FVD 3772) and performing competitively with other methods like MCVD and TGAN-v2.

Significance and Claims

The paper claims that GVT represents a significant shift in video representation by leveraging the explicit and adaptive nature of Gaussian Splatting.

• Versatility: By moving away from fixed grids, GVT adapts token density to information content (spatial) and distinguishes between static and dynamic elements (temporal).
• Efficiency: The feed-forward design eliminates the need for costly per-video optimization, making the approach scalable and generalizable to unseen videos.
• Performance: The method establishes new benchmarks in reconstruction quality and compression efficiency while maintaining competitive performance in downstream tasks like action recognition and generation.

The authors acknowledge limitations, noting that current generation capabilities are restricted to unconditional settings due to the lack of pre-trained multimodal alignment models for 2DGS. They view the current generation results as preliminary evidence of the framework's potential, with future work planned to explore conditional generation (e.g., text-to-video).