💻 computer science

GlobalSplat: Efficient Feed-Forward 3D Gaussian Splatting via Global Scene Tokens

GlobalSplat introduces an efficient feed-forward 3D Gaussian Splatting framework that achieves compact, globally consistent reconstructions with a small footprint and fast inference by learning a global latent scene representation to align multi-view inputs before decoding explicit geometry, thereby overcoming the redundancy and fragility of previous local, heuristic-driven strategies.

Original authors: Roni Itkin, Noam Issachar, Yehonatan Keypur, Xingyu Chen, Anpei Chen, Sagie Benaim

Published 2026-04-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Roni Itkin, Noam Issachar, Yehonatan Keypur, Xingyu Chen, Anpei Chen, Sagie Benaim

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build a 3D model of a room using only a handful of photos.

The Old Way (The "Pixel-By-Pixel" Approach):
Think of previous methods like a team of painters who are told to paint a 3D statue based on photos. The old rule was: "For every single pixel you see in the photo, you must paint a tiny 3D dot (a Gaussian) in the air."

If you take 10 photos, they paint 10 sets of dots. If you take 24 photos, they paint 24 sets of dots.

The Problem: This creates a massive mess. Many of those dots are just painting over the same spot in the air. It's like having 24 different people trying to fill the same bucket with water; you end up with a huge, heavy bucket (gigabytes of data) that takes forever to carry (slow to load) and is full of redundant water. As you add more photos, the bucket gets heavier and heavier, eventually spilling over.

The New Way (GlobalSplat):
The authors of this paper, GlobalSplat, decided to change the rules. They realized that instead of painting dots for every single photo, you should first understand the whole room, and then decide where to put the dots.

Here is how they do it, using a simple analogy:

1. The "Global Brain" (Align First)

Imagine you have a team of 24 detectives looking at a crime scene from different angles.

Old Method: Each detective immediately starts drawing their own sketch of the room on a separate piece of paper. You end up with 24 different, slightly conflicting sketches.
GlobalSplat Method: Before anyone draws anything, the detectives meet in a room and discuss. They pool their memories and create one single, perfect mental map of the room. They agree on exactly where the table is, where the chair is, and how big the room is. They create a "Global Token" (a compact summary) that holds the truth of the entire scene, regardless of how many photos they took.

2. The "Smart Painter" (Decode Later)

Once they have that perfect mental map, they send a single painter to the scene.

The painter looks at the mental map and says, "Okay, I only need 16,000 dots to perfectly represent this room."
Because the painter has the "Global Brain" to guide them, they don't waste dots on empty space or paint the same spot 24 times. They place the dots exactly where they are needed.

Why is this a big deal?

1. The "Suitcase" Analogy (Compactness)

Old Methods: If you want to travel to a new city, you pack a suitcase for every single photo you took. If you took 1,000 photos, your suitcase is the size of a house. It's heavy, expensive to ship, and hard to carry.
GlobalSplat: No matter how many photos you took (12, 24, or 100), you only pack one small backpack. The "Global Brain" inside the backpack knows how to reconstruct the whole city.
- Result: Their 3D model is tiny (only 4 Megabytes, like a short MP3 song) compared to the massive files of other methods (hundreds of Megabytes).

2. The "Speed" Analogy

Old Methods: Loading a 3D scene is like trying to unload a truck full of bricks one by one. It takes a long time (hundreds of milliseconds or even seconds).
GlobalSplat: Because the model is so small and organized, it loads almost instantly (under 78 milliseconds). It's like teleporting the room into existence.

3. The "Consistency" Analogy

Old Methods: Sometimes, if you add more photos, the old methods get confused. The dots from Photo A might clash with the dots from Photo B, creating a glitchy, blurry mess.
GlobalSplat: Because they built the "Global Brain" first, the dots are always consistent. Adding more photos just helps the brain understand the room better; it doesn't make the model bigger or messier.

The Secret Sauce: "Coarse-to-Fine" Training

The paper also mentions a clever training trick. Imagine teaching a student to draw a face.

Bad Training: You tell them, "Draw every single eyelash and pore right now!" They get overwhelmed and the drawing looks messy.
GlobalSplat Training: You tell them, "First, draw the outline of the head. Then, add the eyes and nose. Finally, add the details."
This step-by-step approach ensures the model learns the big picture first, so it doesn't get confused by the tiny details until it's ready.

Summary

GlobalSplat is like upgrading from a chaotic, cluttered warehouse to a highly efficient, smart library.

Old Way: "More photos = More storage, more time, more confusion."
GlobalSplat: "More photos = Better understanding, same tiny storage, instant speed."

It allows us to create high-quality 3D worlds from video that are so small and fast they could run on a phone or a web browser instantly, without needing a supercomputer.

1. Problem Statement

The paper addresses the fundamental challenge of spatial allocation of primitives in Feed-Forward 3D Gaussian Splatting (3DGS). While 3DGS offers efficient rendering, current feed-forward methods (which generate 3D scenes from input images in a single pass without per-scene optimization) suffer from significant trade-offs between representation compactness, reconstruction speed, and rendering fidelity.

The Core Issue: Existing feed-forward approaches typically rely on local, view-aligned strategies (e.g., per-pixel or per-voxel predictions). They unproject pixels from input views into 3D space and attempt to align them globally only after generation.
Consequences:
- Redundancy: As the number of input views increases, these methods "bake in" massive redundancy, leading to scene representations that scale linearly or super-linearly with the number of views (often requiring hundreds of thousands to millions of Gaussians).
- Fragility: Global consistency is difficult to maintain, leading to artifacts in regions of dense overlap.
- Inefficiency: Large representation sizes result in high GPU memory usage, slow inference times, and large disk footprints, hindering real-time applications.

2. Methodology: GlobalSplat

The authors propose GlobalSplat, a framework built on the principle of "Align First, Decode Later." Instead of generating view-centric primitives and merging them later, GlobalSplat first fuses multi-view inputs into a compact, globally consistent latent representation, which is then decoded into explicit 3D Gaussians.

Key Architectural Components:

Scene Normalization & Input Preparation:
- Inputs are normalized into a canonical coordinate system using a similarity transform (averaging camera centers and orientations) to provide a strong geometric prior.
- Input features are augmented with Plücker rays and per-view camera codes (encoding absolute camera position and intrinsics) to inject global geometric context into the patch features.
Dual-Branch Iterative Attention Architecture:
- Learnable Latent Tokens: The model uses a fixed set of $M$ learnable latent scene tokens (e.g., 2048 tokens) that are independent of the number of input views. This forces the model to distill global scene information into a constant budget.
- Disentangled Streams: The architecture employs a dual-branch encoder (Geometry and Appearance) to prevent the model from "cheating" by using texture to mask poor structural predictions.
  - Geometry Branch: Processes structural features.
  - Appearance Branch: Processes texture/color features.
- Mechanism: Both branches use cross-attention to attend to multi-view input features and self-attention to refine global context. The streams are fused via a Mixer MLP to update the latent tokens iteratively.
Decoding & Coarse-to-Fine Curriculum:
- Decoding: Specialized decoders transform the globally-aware latent tokens into explicit 3D Gaussian parameters (position, scale, rotation, opacity, and Spherical Harmonics for color).
- Capacity Curriculum: To prevent representation bloat and ensure training stability, the model uses a coarse-to-fine training strategy.
  - Initially, the model predicts a single merged Gaussian per latent token.
  - As training progresses, the capacity is incrementally increased (e.g., merging 16 candidates into 8, then 4, then 2, finally 8 distinct Gaussians per token).
  - This ensures the model first learns global geometry before refining local details.
Training Objectives:
- Rendering Loss: Standard L2 and perceptual losses on rendered target views.
- Self-Supervised Consistency: The input views are split into two subsets. The model must produce consistent geometry and depth maps from these different subsets, enforcing global coherence without ground-truth 3D labels.
- Regularization: Includes frustum constraints (keeping Gaussians within valid camera views) and specific regularizations for opacity, scale, and rotation to prevent degenerate solutions.

3. Key Contributions

Paradigm Shift: Introduces the "Align First, Decode Later" approach, shifting from view-centric primitive formation to scene-centric primitive formation.
Global Scene Tokens: Demonstrates that a fixed set of latent tokens can effectively encode multi-view scenes, eliminating the redundancy inherent in view-aligned pipelines.
Disentangled Architecture: Proposes a dual-branch encoder/decoder that separates geometry and appearance processing, improving structural soundness.
Coarse-to-Fine Curriculum: A novel training strategy that naturally prevents representation bloat by gradually increasing the decoded Gaussian capacity.
Efficiency: Achieves a strict, view-invariant budget of Gaussians (e.g., 16K) regardless of the number of input views.

4. Experimental Results

The model was evaluated on RealEstate10K (indoor/outdoor real estate) and ACID (aerial drone footage) against state-of-the-art baselines (e.g., ZPressor, DepthSplat, C3G, AnySplat).

Compactness:
- GlobalSplat uses a fixed budget of 16K Gaussians (4 MB disk size) for scenes with 12, 24, or 36 input views.
- This is a >99% reduction compared to dense baselines like ZPressor (393K Gaussians) or AnySplat (millions of Gaussians).
Quality:
- On RealEstate10K with 24 views, GlobalSplat achieves 28.53 PSNR, outperforming highly compact methods like C3G (23.80 PSNR) and competing with much heavier models like ZPressor (28.51 PSNR).
- It demonstrates strong zero-shot generalization to the ACID dataset, maintaining high quality without retraining.
Efficiency:
- Inference Time: **<78 ms** per forward pass (vs. >500ms for many baselines).
- Memory: Peak GPU memory usage is 1.79 GB, significantly lower than baselines (e.g., DepthSplat at 29.84 GB, ZPressor at 3.70 GB).
- Scalability: Unlike baselines where performance degrades or memory explodes as input views increase, GlobalSplat's performance remains stable and efficient.

5. Significance

GlobalSplat establishes a new practical operating point for feed-forward 3D reconstruction. By decoupling the representation size from the number of input views, it solves the scalability bottleneck of current 3DGS methods.

Real-Time Viability: The ultra-low memory footprint and fast inference make it suitable for real-time applications on edge devices or in VR/AR scenarios.
Generalization: The global latent approach proves robust across diverse datasets (indoor vs. aerial), suggesting a more fundamental understanding of 3D scene structure than view-aligned methods.
Future Direction: The paper highlights that while current limitations exist for extreme sparse views or dynamic scenes, the "global token" paradigm offers a promising foundation for efficient 4D reconstruction and adaptive scene representation.

In summary, GlobalSplat proves that global alignment before decoding is superior to local alignment after decoding for feed-forward 3D Gaussian Splatting, delivering high-fidelity, compact, and fast 3D scene reconstruction.