GSStream: 3D Gaussian Splatting based Volumetric Scene Streaming System

This paper introduces GSStream, a novel volumetric scene streaming system for 3D Gaussian Splatting that leverages collaborative viewport prediction and deep reinforcement learning-based bitrate adaptation to overcome bandwidth challenges and deliver high-quality, real-time immersive experiences.

The Gist

Imagine you are visiting a massive, hyper-realistic 3D museum inside your virtual reality headset. The exhibits are so detailed you can see the dust motes dancing in the light and the texture of every brick. This is the power of a technology called 3D Gaussian Splatting (3DGS). It creates stunningly realistic worlds.

But there's a catch: these worlds are enormous. Trying to stream a full museum to your headset over the internet is like trying to pour an entire ocean through a garden hose. It's too much data, too fast, and your connection would choke.

Enter GSStream, the new system proposed in this paper. Think of GSStream as a super-smart, high-speed delivery service for these 3D worlds. Here is how it works, broken down into simple concepts:

1. The Problem: The "Ocean in a Garden Hose"

Traditional 3D streaming tries to send the whole scene, or at least huge chunks of it, all the time. But you only ever look at one small corner of the room at a time. Sending the whole museum is a waste of bandwidth.

2. The Solution: The "Smart Pizza Delivery"

GSStream solves this by breaking the 3D world into thousands of small, cubic "tiles" (like slices of a pizza, but in 3D). It doesn't send the whole pizza; it only sends the slices you are looking at.

But here's the tricky part: You move your head faster than the internet can react. If you turn your head, the system needs to have the new slice ready before you even get there, or the image will lag and look blurry.

GSStream uses two "superpowers" to solve this:

Superpower A: The "Mind-Reading" Prediction (Collaborative Viewport Prediction)

Imagine you are at a party. If you want to guess where a specific guest will walk next, you could just look at their past movements. But what if you also knew how everyone else at the party tends to move? Maybe everyone who likes art tends to linger near the paintings, while runners tend to circle the room.

• Old systems only looked at your history. "Oh, you looked left last time, so you'll look left again."
• GSStream (The CVP Module) looks at your history plus the habits of 32 other people who visited the same virtual museum. It learns that "People who look like you usually turn right after looking at the statue."
• The Result: It predicts where you will look next with much higher accuracy, getting the data ready before you even turn your head.

Superpower B: The "Traffic Cop" (Deep Reinforcement Learning)

Now, imagine the internet connection is a busy highway. Sometimes it's a clear road (fast internet), and sometimes it's a traffic jam (slow internet). The system needs to decide: Do I send a high-definition 4K slice of the pizza, or a blurry low-res one?

• Old systems used rigid rules. "If the road is slow, send low-res." This is like a traffic cop who only knows two signals: Stop and Go.
• GSStream (The DBA Module) uses Deep Reinforcement Learning (DRL). Think of this as a super-intelligent traffic cop that learns by trial and error. It watches the traffic (your bandwidth), predicts where you are going (the prediction from Superpower A), and makes a split-second decision: "The road is clearing up, and the user is about to look at the expensive vase. Let's send a high-quality slice of that specific tile right now!"
• It handles the fact that different museums have different numbers of tiles (some are small, some are huge) by treating them like a flexible set of blocks rather than a rigid grid.

3. The Training: The "Practice Run"

To make these systems work, the researchers didn't just guess. They built a virtual playground and invited 32 real people to wear headsets and explore 15 different 3D worlds (from a garden to a train station). They recorded exactly where everyone looked, how they moved, and how long they stayed. This created a massive "behavioral map" that taught the AI how humans actually explore 3D spaces.

The Bottom Line

GSStream is like having a personal concierge for your virtual reality world.

1. It breaks the world into manageable pieces.
2. It guesses your next move by learning from your habits and the habits of others.
3. It dynamically adjusts the quality of the pieces it sends based on your internet speed, ensuring you always see the clearest image possible without buffering.

The result? You get a smoother, sharper, and more immersive experience, even on slower internet connections, because the system is sending exactly what you need, exactly when you need it.

Technical Summary

Here is a detailed technical summary of the paper "GSStream: 3D Gaussian Splatting based Volumetric Scene Streaming System".

1. Problem Statement

The emergence of 3D Gaussian Splatting (3DGS) has revolutionized real-time volumetric scene rendering by offering high-fidelity, explicit representations. However, this quality comes at a steep cost: massive data volumes (e.g., hundreds of megabytes for a single indoor scene) that are bandwidth-intensive and difficult to stream under current network constraints.

Existing streaming solutions for volumetric content (like point clouds) face three specific challenges when applied to 3DGS:

1. Lack of Data: There are no comprehensive datasets of user viewport trajectories specifically for 3DGS scenes, hindering the training of accurate prediction models.
2. Uniform Treatment of Users: Current viewport prediction methods often treat all users identically, ignoring individual behavioral inertia and preferences, leading to suboptimal prediction accuracy.
3. Rigid Bitrate Adaptation: Existing Deep Reinforcement Learning (DRL) approaches often assume a fixed number of tiles across scenes. They fail to handle the variable state and action spaces inherent in 3DGS, where the number of tiles and their distribution vary significantly between scenes.

2. Methodology: The GSStream System

The authors propose GSStream, a novel streaming system designed specifically for 3DGS data. The system architecture consists of four main stages:

A. Pre-Processing

• Tiling: The volumetric scene is divided into non-overlapping cubic tiles.
• Multi-Resolution: Each tile is downsampled into $L$ different quality levels (representations) using a voxel grid filter.
• Variable Tile Count: Unlike previous methods, GSStream acknowledges that the number of tiles ($K$) varies per scene.

B. Collaborative Viewport Prediction (CVP) Module

To predict where a user will look next, the system employs a deep learning architecture that combines two types of priors:

1. Historical Priors Extraction (HPE): Uses an iTransformer and MLPs to analyze the specific user's historical viewport sequence (position and orientation) to capture temporal dynamics.
2. Collaborative Priors Extraction (CPE): Uses a Cross-Attention mechanism to learn user-specific embeddings. By analyzing data from multiple users simultaneously, the model captures shared viewing patterns and individual behavioral differences (e.g., some users explore randomly, others stay fixed).

• Output: A predicted future viewport sequence used to guide data selection.

C. DRL-based Bitrate Adaptation (DBA) Module

This module decides which tile representations to transmit based on network bandwidth and predicted viewports.

• MDP Formulation: The problem is modeled as a Markov Decision Process (MDP).
  • State ($s_t$): Current bandwidth, predicted viewport sequence, and tile features (position, importance score, transmission status).
  • Action ($a_t$): Preference scores for transmitting specific tiles at specific quality levels.
  • Reward ($r_t$): A weighted sum of progressive streaming rewards (prioritizing tiles in the Field of View), importance scores (based on Gaussian scaling factors), and a delay penalty.
• Handling Variability (Key Innovation): To address the variable number of tiles (unordered sets with varying dimensions), the authors utilize Set Abstraction (SA) and Feature Propagation (FP) modules (adapted from PointNet++). These modules allow the DRL agent to process unordered sets of tiles regardless of the scene's complexity.
• Algorithm: A Deep Deterministic Policy Gradient (DDPG) algorithm with an Actor-Critic framework is used to learn the optimal bitrate allocation policy.

D. Streaming Pipeline

The server transmits selected tile representations based on the DBA's decision. The client reconstructs the 3DGS scene for display on Head-Mounted Devices (HMDs).

3. Key Contributions

1. First 3DGS Streaming System: GSStream is the first system to integrate 3DGS with a collaborative viewport prediction and DRL-based bitrate adaptation framework.
2. New Dataset: The authors constructed the first comprehensive user viewport trajectory dataset for 3DGS, containing data from 32 subjects across 15 diverse indoor and outdoor scenes. This dataset revealed that 3DGS induces more active and varied user behaviors compared to traditional point clouds.
3. Novel Architecture:
   • CVP: A collaborative learning approach that fuses user-specific embeddings with historical sequences to improve prediction accuracy.
   • DBA: A DRL solution using SA/FP modules to handle variable state/action spaces (different tile counts), overcoming a major limitation of existing DRL streaming methods.
4. Performance: The system achieves superior visual quality and network efficiency compared to state-of-the-art (SOTA) baselines.

4. Experimental Results

The system was evaluated against three baselines: ViVo (rule-based), CaV3 (Transformer-based prediction + MAB), and GS3D (distance-based sorting).

• Visual Quality (SSIM): GSStream achieved the highest Structural Similarity Index Measure (SSIM).
  • Outperformed ViVo by 118.9%, CaV3 by 9.4%, and GS3D by 10.9% in average viewport SSIM across tested bandwidths (40, 80, 120 Mbps).
  • It maintained high visual fidelity even during early-stage viewpoint exploration where other methods degraded.
• Bandwidth Efficiency: GSStream demonstrated stable and optimal bandwidth utilization, avoiding the throughput fluctuations seen in ViVo and CaV3. It effectively utilized available bandwidth to deliver progressive quality.
• Ablation Studies: Removing either the collaborative priors or historical priors from the CVP module resulted in higher Mean Absolute Error (MAE) in prediction, confirming the necessity of fusing both signals.

5. Significance

GSStream addresses the critical bottleneck of bandwidth consumption in high-fidelity 3D content delivery. By leveraging the explicit nature of 3DGS and introducing a flexible, learning-based streaming architecture, it enables:

• Immersive Real-Time Experiences: High-quality rendering on HMDs without buffering or quality drops.
• Scalability: The ability to stream complex scenes with variable geometry over fluctuating network conditions.
• Future Foundation: The introduction of the 3DGS viewport dataset and the handling of variable tile sets provide a foundation for future research in dynamic volumetric video streaming and 3DGS compression codecs.

In summary, GSStream represents a significant step forward in making high-fidelity 3D Gaussian Splatting practical for real-world, network-constrained applications like Virtual Reality (VR) and Telepresence.