Technical Summary: HyperGS: Fast and Generalizable Gaussian Video Representation

Problem Statement

While 3D Gaussian Splatting (3DGS) has proven effective for static scenes and dynamic video, existing video adaptation methods rely on per-video optimization. These approaches fit Gaussian parameters to each video independently through iterative, gradient-based optimization, often requiring tens of thousands of steps and hours of computation per video. This lack of amortization prevents the transfer of learned structure between videos, rendering such methods impractical for real-time processing and limiting their ability to generalize to out-of-distribution (OOD) videos or higher resolutions without re-optimization.

Prior attempts to create feedforward alternatives for video representation have largely focused on Implicit Neural Representations (INRs), where a network predicts the weights of a decoder. However, these methods tie the representation to a specific neural architecture and require an additional nonlinear decoding step.

Methodology: HyperGS

The authors propose HyperGS, a feedforward, optimization-free framework that directly predicts explicit 2D Gaussian primitives from video input in a single forward pass. This approach decouples the representation from any implicit decoder, leveraging the efficient rasterization capabilities of Gaussian Splatting.

Architecture

The model consists of a factorized spatiotemporal Transformer and a query-based Transformer:

1. Video Tokenization: Input videos are projected into patch embeddings using 3D convolutions with learnable spatio-temporal positional embeddings (RoPE).
2. ST-Transformer (Encoder): A Factorized Spatio-Temporal Transformer processes video tokens. It applies spatial self-attention (2D RoPE) and temporal self-attention (1D RoPE) sequentially to contextualize features across space and time. A skip connection preserves original patch embeddings to aid stability.
3. Q-Transformer (Decoder): A set of learnable Gaussian tokens (queries) attends to the contextualized video features. Through self-attention, these tokens aggregate relevant information to regress the final 8-parameter Gaussian representation for each frame.
4. Parameter Prediction: The model outputs interpretable parameters for $N$ Gaussians per frame: mean position ($\mu$), anisotropic scale ($s$), rotation ($r$), and color ($c$). These are constrained to valid ranges (e.g., scales are enforced positive, rotations cover $[0, 2\pi)$).
5. Rendering: The predicted Gaussians are rendered via differentiable 2D Gaussian rasterization to reconstruct the video frames.

Training Stability and Regularization

The authors identify a critical training instability where unconstrained optimization drives Gaussians toward "needle-like" anisotropy (extreme stretching along one axis), causing a sudden collapse in performance mid-training. To address this, they introduce adaptive geometric regularizers:

• Effective Rank Regularization ($\mathcal{L}_{erank}$): Uses the Effective Rank of the scale distribution to penalize degenerate, needle-like shapes. The threshold is dynamically adapted using an Exponential Moving Average (EMA) of the 1st-percentile effective rank across the batch.
• Adaptive Scale Regularization ($\mathcal{L}_{scale}$): Tracks the 99th-percentile of scale magnitudes via EMA to prevent unreasonably large scales, penalizing outliers with a squared excess loss.

Key Contributions

1. First Feedforward Gaussian Predictor: HyperGS is the first method to predict explicit Gaussian primitives for video in a single forward pass without per-video optimization.
2. Performance and Efficiency:
   • Achieves 104–105× faster encoding than per-video Gaussian optimization at matched reconstruction quality.
   • Improves PSNR by +2.9–3.1 dB over prior feedforward video encoders (e.g., FastNeRV) on K400, SSv2, and UCF101, despite using a smaller representation size (64K parameters vs. 85K–139K).
   • Encoding speeds reach ~400–2200 FPS, with decoding at ~3900–4400 FPS.
3. Zero-Shot Generalization: The model generalizes to unseen, high-resolution (720p) videos without fine-tuning. It supports patch-grid encoding to tile high-resolution frames, trading representation size for higher fidelity without re-encoding.
4. Stabilization Mechanism: The introduction of adaptive rank-based regularization successfully prevents the "needle-like" degeneration that plagues feedforward Gaussian training.

Experimental Results

• Comparison with Optimization-Based Methods: On 720p DAVIS and UVG datasets, HyperGS matches the reconstruction quality of 2DGS (optimized per video) but reduces encoding time from hours to milliseconds. A patch-grid variant further surpasses per-video optimization in quality.
• Comparison with Feedforward INRs: On K400, SSv2, and UCF101, HyperGS outperforms state-of-the-art feedforward INR methods (TransINR, GINR, FastNeRV, TeCoNeRV) in both PSNR and SSIM, even when those methods are trained and evaluated at higher resolutions (640×480) while HyperGS is trained only at 256×256.
• Comparison with Traditional Codecs: Compared to H.265, HyperGS offers 1.3–2.7× faster encoding and ~1.3× faster decoding while achieving equal or higher fidelity across matched rate-distortion points.
• Ablation Studies:
  • Gaussian Count ($N$): Increasing $N$ from 250 to 3000 linearly improves quality (+7.31 dB on K400) with minimal impact on training time.
  • Architecture: The query-based self-attention mechanism in the Q-Transformer is essential; replacing it with a simple MLP causes a drastic quality drop (~9 dB).
  • Regularization: Adaptive regularization is critical for training stability, outperforming fixed thresholds which are sensitive to model capacity.

Significance and Claims

The paper claims that HyperGS advances Gaussian Splatting as a practical direction for fast and generalizable video representation. By combining the fast, flexible rendering of Gaussian Splatting with the speed and generalization of feedforward prediction, HyperGS overcomes the fundamental bottleneck of per-video optimization. It demonstrates that explicit geometric primitives can be learned directly from pixels, enabling real-time encoding, zero-shot high-resolution rendering, and superior rate-distortion performance compared to both traditional codecs and prior neural video representations.