TokenSplat: Token-aligned 3D Gaussian Splatting for Feed-forward Pose-free Reconstruction

TokenSplat is a feed-forward framework that achieves joint 3D Gaussian reconstruction and camera pose estimation from unposed multi-view images by introducing a token-aligned prediction module and an asymmetric dual-flow decoder to enable robust, iterative-free 3D scene modeling.

The Gist

Imagine you are trying to build a perfect 3D model of a room, but you only have a handful of photos taken from different angles. The tricky part? You don't know exactly where the camera was standing for each photo, and the photos might be a bit blurry or overlapping in confusing ways.

This is the problem TokenSplat solves. It's a new AI tool that can look at a pile of unorganized photos and instantly build a high-quality 3D model of the scene and figure out exactly where the camera was for every single shot.

Here is how it works, using some everyday analogies:

1. The Old Way: The "Pixel-by-Pixel" Construction Crew

Previous methods tried to build the 3D world by looking at every single pixel in the photos. Imagine a construction crew where every bricklayer is assigned to one specific pixel on a photo.

• The Problem: If you have 10 photos of the same wall, you now have 10 different bricklayers trying to build the same spot. They argue, overlap, and end up creating a messy, blurry wall. It's like having too many people trying to paint the same spot on a canvas; it just gets muddy.
• The Pose Issue: These crews also needed a GPS tracker (camera pose) to know where to stand. If the GPS was wrong, the whole building would be crooked.

2. The TokenSplat Solution: The "Smart Team Leaders"

TokenSplat changes the game by stopping the pixel-by-pixel approach. Instead, it uses "Tokens."

Think of a Token not as a single pixel, but as a Team Leader responsible for a whole neighborhood of pixels.

• The Magic of Alignment: When TokenSplat looks at 10 photos, it doesn't ask, "What is this one red pixel?" Instead, it asks, "Where is the concept of the red chair in all these photos?"
• It groups these "Team Leaders" together. If a Team Leader in Photo A sees a chair, and a Team Leader in Photo B sees the same chair, they shake hands and merge their information. They agree on where the chair is, smoothing out the noise. This is the Token-aligned Gaussian Prediction. It's like having a single, super-smart architect who looks at all the blueprints at once to decide where the furniture goes, rather than 100 confused painters.

3. The "One-Way Street" Decoder (ADF-Decoder)

One of the hardest parts of this job is separating two things: What the room looks like (the scene) vs. Where the camera was (the pose).

• The Problem: In older AI models, these two things got mixed up. The AI would get confused, thinking a shadow was a wall, or that the camera moved when it didn't. It was like a driver trying to navigate while reading the map and driving the car at the same time, resulting in a crash.
• The TokenSplat Fix: They built a special communication system called the Asymmetric Dual-Flow Decoder.
  • Imagine a One-Way Street.
  • The Camera Token (the driver) looks at the Image Tokens (the scenery) to figure out, "Okay, I am standing here."
  • The Image Tokens (the scenery) listen to the Camera Token, but only receive a very simple, stable signal: "I am here." They don't get confused by the driver's panic or complex thoughts.
  • This keeps the "where am I?" logic separate from the "what does it look like?" logic, ensuring the 3D model stays straight and the camera positions are accurate.

4. Why It's a Big Deal

• No GPS Needed: You don't need to know the camera's location beforehand. TokenSplat figures it out on the fly.
• Handles Crowds: If you throw 28 photos at it (instead of just 3), it doesn't get messy. Because it merges "Team Leaders" (tokens) instead of individual pixels, adding more photos actually makes the model sharper and more complete, rather than cluttered.
• Instant Results: It doesn't need to spend hours tweaking the model (iterative refinement). It takes a look, does the math in one go, and spits out a perfect 3D world.

The Bottom Line

TokenSplat is like upgrading from a chaotic construction site with 100 confused workers to a highly organized team of smart architects. They communicate efficiently, ignore the noise, and build a crystal-clear 3D world from a messy pile of photos, even if they don't know exactly where the photos were taken.

Technical Summary

1. Problem Statement

The paper addresses two critical limitations in current 3D Gaussian Splatting (3DGS) reconstruction pipelines:

1. Dependency on Accurate Camera Poses: Most existing feed-forward 3DGS methods require precise camera poses (often obtained via Structure-from-Motion, which is computationally expensive and error-prone). Without accurate poses, reconstruction quality degrades significantly.
2. Representation Entanglement and Redundancy:
   • Entanglement: Existing pose-free methods often encode scene semantics and viewpoint cues within the same feature embeddings. This makes it difficult to disentangle camera parameters from scene content, leading to error propagation where pose estimation mistakes corrupt the 3D geometry.
   • Redundancy: Most methods use pixel-aligned Gaussian prediction heads. As the number of input views increases, this pixel-centric approach generates redundant, overlapping Gaussians, causing geometric blurring, color inconsistency, and fragmented reconstructions.

2. Methodology: TokenSplat

TokenSplat is a feed-forward framework that jointly reconstructs 3D scenes and estimates camera poses from unposed multi-view images without iterative refinement. Its architecture consists of three core components:

A. Token-aligned Gaussian Prediction Module

Instead of predicting Gaussians at the pixel level, TokenSplat operates in the feature space using image tokens.

• Mechanism: It aligns semantically corresponding information across different views directly within the token feature space.
• Process:
  1. Coarse Positioning & Confidence: The module predicts coarse positions and fusion confidence scores for each token.
  2. Token Fusion: Tokens are grouped based on spatial proximity and fused using softmax-normalized confidence scores. This aggregates multi-scale contextual features.
  3. One-to-Many Mapping: A Gaussian Prediction Head maps each fused token to multiple Gaussians. This decouples Gaussian density from pixel resolution, allowing for denser, more expressive splats while eliminating the redundancy caused by overlapping pixel-level predictions.

B. Asymmetric Dual-Flow Decoder (ADF-Decoder)

To solve the problem of feature entanglement between camera poses and scene geometry, the authors introduce a specialized decoder.

• Learnable Camera Tokens: Separate learnable tokens are initialized to represent camera poses for each view.
• Directionally Constrained Communication: Unlike symmetric attention mechanisms that mix features bidirectionally, the ADF-Decoder enforces a specific flow:
  • Image Tokens: Primarily aggregate scene context from other views to refine the 3D structure.
  • Camera Tokens: Extract geometric cues from image tokens to predict poses. Crucially, they only propagate stabilized, low-frequency pose alignment signals back to the image tokens.
• Result: This design prevents viewpoint-specific cues from contaminating scene semantics, ensuring clean factorization between pose and geometry.

C. Architecture Overview

1. ViT Encoder: A shared Vision Transformer encoder processes input images into tokens.
2. Canonical Scene Decoder: Processes the reference view to establish a canonical scene representation using cross-attention.
3. ADF-Decoder: Refines image and camera tokens for the remaining views.
4. Parallel Heads:
   • Camera Pose Head: Regresses extrinsic parameters (rotation/translation) from camera tokens.
   • Gaussian Prediction Head: Generates dense 3D Gaussians from fused tokens.

3. Key Contributions

1. TokenSplat Framework: A novel feed-forward, pose-free reconstruction system that jointly estimates camera poses and 3D Gaussian scenes, exhibiting strong generalization across diverse scenes.
2. Token-aligned Gaussian Prediction: A module that enables long-range multi-view feature aggregation. By fusing features at the token level rather than the pixel level, it significantly reduces redundancy and fragmentation, producing structurally coherent 3D scenes.
3. Asymmetric Dual-Flow Decoder (ADF-Decoder): A mechanism that disentangles pose reasoning from scene encoding. By enforcing directional information flow, it achieves more accurate pose estimation and higher-fidelity reconstruction without iterative refinement loops.

4. Experimental Results

The method was evaluated on ScanNet and RealEstate10K (RE10K) datasets, comparing against state-of-the-art (SOTA) pose-required (e.g., MVSplat, FreeSplat) and pose-free (e.g., NoPoSplat, VicaSplat, AnySplat) methods.

• Novel View Synthesis (NVS):
  • TokenSplat consistently outperforms all pose-free baselines.
  • In dense-view scenarios (e.g., 8 views on RE10K), it even surpasses pose-dependent methods like FreeSplat, achieving 0.95 dB higher PSNR.
  • It demonstrates superior scalability: When tested on 28 views (trained on 10), TokenSplat maintains stable quality, whereas other methods degrade due to accumulated redundancy.
• Pose Estimation:
  • TokenSplat significantly reduces pose errors (ATE, RPE-t, RPE-r) compared to other pose-free methods.
  • On RE10K (8 views), it reduced Relative Rotation Error (RPE-r) by 0.335 compared to VicaSplat.
• Cross-Dataset Generalization:
  • Trained solely on RE10K, TokenSplat achieves SOTA zero-shot performance on ScanNet, outperforming FreeSplat by up to 0.92 dB in PSNR and reducing pose errors significantly.
• Efficiency:
  • The token fusion approach stabilizes the number of Gaussians as the number of input views increases, unlike pixel-aligned methods where Gaussian count grows linearly with views. This leads to more stable inference times.

5. Significance

TokenSplat represents a significant advancement in 3D reconstruction by removing the dependency on pre-computed camera poses, which is a major bottleneck for real-world applications (e.g., mobile AR/VR).

• Robustness: It solves the "representation entanglement" problem, allowing the model to learn geometry and pose simultaneously without one corrupting the other.
• Scalability: By moving from pixel-aligned to token-aligned prediction, it effectively handles dense multi-view inputs without the geometric blurring and redundancy that plague existing methods.
• Practicality: The ability to perform high-fidelity reconstruction and accurate pose estimation in a single feed-forward pass makes it highly suitable for dynamic, real-time applications where iterative optimization is infeasible.