SceneTok: A Compressed, Diffusable Token Space for 3D Scenes

SceneTok introduces a novel tokenizer that compresses 3D scene view sets into a small, permutation-invariant set of unstructured tokens, enabling state-of-the-art reconstruction, flexible novel view synthesis, and efficient scene generation with significantly higher compression than existing methods.

The Gist

Imagine you want to send a friend a complete, 3D tour of your living room.

The Old Way (The Problem):
Currently, if you want to share a 3D scene, you usually have to send a massive, heavy file. It's like trying to mail a full-scale, brick-by-brick replica of your house. It takes forever to send, takes up a huge amount of space, and if your friend wants to walk around the house from a new angle they didn't see in the photos, the model often breaks or looks blurry.

Other methods try to send a "blueprint" (a grid of data), but blueprints for 3D spaces are still huge and hard to edit.

The New Way (SceneTok):
The researchers behind SceneTok came up with a clever trick. Instead of sending the bricks or the blueprint, they send a tiny, magical bag of "conceptual marbles."

Here is how it works, broken down into simple steps:

1. The Compression (The "Magic Bag")

Imagine you have a 360-degree video of your room. SceneTok looks at this video and says, "I don't need to send every single pixel. I just need to send the essence of the room."

It squashes all that visual information down into a tiny, unordered list of tokens (think of them as abstract marbles or Lego bricks).

• The Magic: These marbles don't have a fixed position. They aren't "the marble for the left wall" or "the marble for the ceiling." They are just a bag of information that collectively describes the room.
• The Size: This bag is incredibly small. The paper says it's 1,000 to 10,000 times smaller than other methods. It's like compressing a whole movie into a single text message.

2. The Decoder (The "Imagination Engine")

Now, your friend receives this tiny bag of marbles. They can't just "look" at the marbles to see the room. They need a special tool to turn those marbles back into a picture.

This tool is called a Generative Decoder.

• How it works: Your friend tells the tool, "Show me the room from the kitchen window." The tool looks at the bag of marbles and uses its "imagination" (powered by AI) to fill in the gaps.
• Handling Uncertainty: If the bag of marbles doesn't have enough info to show exactly what's behind a closed door, the tool doesn't freeze. It says, "I'm not 100% sure, so I'll generate a few plausible options." It handles the "unknown" gracefully, creating a smooth image even when data is missing.

3. The Generation (The "Dream Machine")

The coolest part is what happens before you even have a video of the room.

Because the "bag of marbles" is so small and simple, you can use a standard AI (like the ones that generate images from text) to create a new bag of marbles from scratch.

• The Analogy: Imagine you tell an AI, "Create a cozy living room with a fireplace." The AI doesn't try to build a 3D house brick by brick. Instead, it quickly invents a new, tiny bag of marbles that represents that room.
• Speed: This happens in about 5 seconds. Once the bag of marbles is made, the "Imagination Engine" can instantly render a 3D tour of that new room from any angle you want.

Why is this a Big Deal?

• Speed: It's incredibly fast. You can generate a 3D scene and walk through it in seconds, whereas previous methods took minutes or hours.
• Flexibility: Because the data isn't tied to a rigid grid (like a spreadsheet), you can generate scenes that are totally new, not just variations of existing ones.
• Efficiency: It separates the "thinking" (creating the scene) from the "drawing" (rendering the view). You can make the "thinking" part smarter without slowing down the "drawing" part.

In Summary:
SceneTok is like a universal translator for 3D worlds. It turns a complex, heavy 3D scene into a tiny, lightweight "language of marbles." This allows us to send, store, and even dream up new 3D worlds instantly, without needing supercomputers or massive hard drives. It turns the heavy lifting of 3D graphics into a quick, efficient conversation between AI models.

Technical Summary

1. Problem Statement

The paper addresses the challenge of representing and generating 3D scenes in the era of large-scale generative models. Existing approaches face significant limitations:

• 3D Data Structures (e.g., NeRFs, 3D Gaussians): These are often high-dimensional and lack scalability. Training foundation models on them is prohibitively expensive due to the cubic scaling of 3D data and the scarcity of large-scale 3D datasets.
• View-Space Generation (e.g., Video Diffusion): Models that generate images directly from input views often entangle the generation process with the rendering process. This leads to inefficiencies, such as redundant computation when revisiting views, and requires massive models to ensure consistency.
• Latent Representations: Previous latent approaches (e.g., LVSM, RayZer) often use high-dimensional token sets (thousands of tokens) that are too large for efficient diffusion-based generation, or they fail to generalize to novel camera trajectories (transferability).

The core goal is to create a representation that is highly compressed, unstructured (permutation-invariant), diffusable (suitable for generative models), and capable of rendering novel views (including those deviating from the input trajectory).

2. Methodology

The authors propose SceneTok, a two-stage framework consisting of a scene autoencoder and a latent generative model.

Stage 1: SceneTok Autoencoder (Compression & Rendering)

This stage encodes a set of context views into a compact, unstructured token set and decodes them into novel views.

• Encoder (Scene Perceiver):
  • Input: A set of $N$ context images ($X_C$) and their camera poses ($P_C$).
  • Feature Extraction: Images are first compressed into latent feature maps using a pre-trained VA-VAE (16x spatial compression).
  • Perceiver Architecture: The encoder uses a "Scene Perceiver" module with two branches:
    1. Multi-view Branch: Processes the latent image patches. Camera poses are converted into ray maps (origins and directions) and used to modulate patch tokens via AdaLN (Adaptive Layer Normalization) before multi-view attention.
    2. Scene Token Branch: Processes a set of learnable scene queries ($Q$) via self-attention and cross-attention with the multi-view branch.
  • Output: A highly compressed set of $K$ continuous, unstructured tokens ($Z$). Crucially, the encoder uses 2D Rotary Positional Encoding (RoPE) instead of 3D RoPE to ensure the tokens are order-invariant (permutation-invariant), allowing rendering from arbitrary camera trajectories later.
• Decoder (Generative Renderer):
  • Architecture: A lightweight Rectified Flow decoder (based on Diffusion Transformers).
  • Process: It takes the scene tokens ($Z$) and a novel camera trajectory ($P_T$) to render target views.
  • Uncertainty Handling: The decoder is generative, meaning it samples from a conditional distribution $p(x|Z)$. It learns to sample from a narrow distribution for well-defined regions and falls back to pure generation for uncertain regions (e.g., occluded areas or high-frequency details lost during compression).
  • Output: Rendered novel views using a VideoDCAE decoder (from Open-Sora 2.0) for pixel-space reconstruction.

Stage 2: Latent Scene Generation (SceneGen)

This stage performs conditional generation directly in the compressed token space.

• Model: A Diffusion Transformer trained on the token set $Z$ produced by Stage 1.
• Conditioning: Generation is conditioned on a single (or few) input image(s) and a set of anchor poses ($A$) that define the spatial extent and layout of the scene.
• Efficiency: By decoupling generation (creating tokens) from rendering (decoding tokens to pixels), the system can scale the generative model without impacting rendering speed.

3. Key Contributions

1. Novel Token Paradigm: Introduction of the first 3D scene tokenizer that compresses scene information into a small set of unstructured, permutation-invariant tokens. This disentangles the representation from the spatial grid.
2. Decoupled Rendering and Generation: A two-stage approach where a lightweight generative decoder handles rendering, while a separate diffusion model handles scene generation in latent space. This allows for independent scaling and optimization.
3. Extreme Compression: The token representation is 1–3 orders of magnitude smaller than existing explicit (3D Gaussians) or latent (LVSM) representations while maintaining state-of-the-art quality.
4. Transferability: The method supports true Novel View Synthesis (NVS), meaning it can render scenes from camera trajectories that deviate significantly from the input context, unlike view-interpolation methods.
5. Efficiency:
   • Rendering: 32 novel views rendered in ~1 second on an RTX 4090.
   • Generation: Full scene generation (tokens + rendering) takes ~5 seconds on a consumer GPU, significantly faster than prior paradigms.

4. Experimental Results

The method was evaluated on RealEstate10K, DL3DV, and ACID (zero-shot).

• Reconstruction Quality (NVS):
  • SceneTok achieves State-of-the-Art (SOTA) performance in PSNR, LPIPS, SSIM, rFVD, and rFID compared to baselines like MVSplat, DepthSplat, LVSM, and RayZer.
  • It significantly outperforms LVSM and RayZer in transferability (rendering novel trajectories), achieving much higher True-Pose-Similarity (TPS) scores.
• Compression Efficiency:
  • SceneTok uses only ~32K–65K floats for representation, compared to millions for explicit methods (e.g., MVSplat uses ~46M floats) and ~1.5M for LVSM.
• Generation Performance:
  • SceneGen performs single-view generation with quality comparable to large-scale models like SEVA and DFoT but is orders of magnitude faster (5s vs. 146s–1620s).
  • It runs efficiently on consumer hardware (RTX 4090), whereas other methods often require high-end clusters or encounter Out-of-Memory (OOM) errors.
• Uncertainty Analysis:
  • Experiments show a direct correlation between token information (via masking) and output variance. The generative decoder successfully handles uncertainty by sampling plausible details for unseen regions.

5. Significance

SceneTok represents a paradigm shift in 3D scene representation and generation:

• Scalability: By compressing 3D scenes into a small, unstructured token set, it makes training large-scale generative models on 3D data feasible, overcoming the "cubic cost" barrier of 3D structures.
• Efficiency: The decoupling of generation and rendering allows for rapid iteration and real-time applications, making high-quality 3D world generation accessible on consumer hardware.
• Generalization: The order-invariant nature of the tokens enables robust generalization to novel camera trajectories, a critical requirement for open-world understanding and reasoning tasks.

In summary, SceneTok provides a highly efficient, scalable, and high-quality bridge between 3D scene understanding and generative AI, enabling the creation of 3D worlds in seconds rather than hours.