🤖 machine learning

LoST: Level of Semantics Tokenization for 3D Shapes

The paper proposes LoST (Level-of-Semantics Tokenization), a novel method that orders 3D shape tokens by semantic salience and utilizes a Relational Inter-Distance Alignment (RIDA) loss to achieve state-of-the-art reconstruction and efficient autoregressive generation with significantly fewer tokens than prior geometric hierarchy-based approaches.

Original authors: Niladri Shekhar Dutt, Zifan Shi, Paul Guerrero, Chun-Hao Paul Huang, Duygu Ceylan, Niloy J. Mitra, Xuelin Chen

Published 2026-03-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Niladri Shekhar Dutt, Zifan Shi, Paul Guerrero, Chun-Hao Paul Huang, Duygu Ceylan, Niloy J. Mitra, Xuelin Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to describe a complex 3D object, like a dragon, to a friend over the phone.

The Old Way (Geometric Level-of-Detail):
Current methods are like describing the dragon by starting with a single dot, then adding a few more dots to make a rough cloud, then adding more dots to make a vague blob, and finally, after thousands of dots, you have a dragon.

The Problem: If you stop after the first few dots, your friend just sees a cloud. If you stop halfway, they see a weird blob. It's inefficient because you have to send so many dots just to get the basic idea of "dragon" across.

The New Way (LoST - Level of Semantics Tokenization):
This paper introduces a new way to describe 3D shapes called LoST. Instead of building the shape from the ground up with dots, LoST organizes the description by importance (semantics).

Think of it like describing a movie instead of a pile of bricks:

The First Token (The Plot): You say, "It's a dragon." Instantly, your friend's brain conjures a complete, recognizable dragon. They know the general shape, the wings, the tail.
The Next Few Tokens (The Genre): You add, "It's a fire-breathing dragon." Now the image gets clearer.
The Later Tokens (The Details): Finally, you add, "It has a scar on its left wing and is made of emerald scales."

With LoST, even if you only send the first few "words" (tokens), the listener gets a plausible, complete picture. You don't need to send thousands of words to get the main idea; you just need the right ones in the right order.

The Secret Sauce: How did they teach the computer to do this?

The computer needs to learn that "Dragon" is more important than "Scale texture." To do this, the researchers invented a clever training trick called RIDA.

Imagine you have two teachers:

Teacher A (The 3D Artist): Knows how to build 3D models but doesn't really understand what a "dragon" feels like conceptually.
Teacher B (The Art Critic - DINO): A famous AI that looks at 2D pictures and instantly understands concepts like "majestic," "scary," or "dragon-like," even if it can't build 3D models.

The RIDA Trick:
The researchers didn't just ask Teacher A to copy Teacher B's words (which is hard because they speak different languages: 3D vs. 2D). Instead, they asked Teacher A to learn the relationships between things.

Teacher B says: "A dragon and a lizard are similar. A dragon and a toaster are very different."
Teacher A (The 3D model) learns: "Okay, I need to arrange my 3D shapes so that dragons and lizards are 'close' to each other in my brain, and toasters are 'far' away."

By learning these relationships rather than just copying details, the 3D model learns to organize its "words" (tokens) by meaning. It learns to put the "Dragon" concept at the very beginning of the list.

Why is this a big deal?

Speed & Efficiency: Because the most important information comes first, you can stop the transmission early and still get a good result. The paper shows LoST can generate high-quality 3D shapes using 0.1% to 10% of the data (tokens) that previous methods needed. It's like sending a text message instead of a 500-page novel to get the same point across.
Better Quality: Because the computer focuses on the "big picture" first, the shapes it generates look more natural and recognizable right from the start, rather than looking like abstract blobs.
Smart Search: Since the computer understands the meaning of the shape (not just its geometry), you can search for a "scary dragon" and it will find it, even if the 3D model looks slightly different geometrically.

In a Nutshell

Previous 3D AI was like building a house by laying one brick at a time, hoping you eventually get a house. LoST is like handing the builder a blueprint that starts with "This is a house," then "It has a roof," then "It has red bricks." You get a recognizable house almost immediately, and you only need to send a tiny fraction of the instructions to get a perfect result.

1. Problem Statement

The paper addresses a critical bottleneck in Autoregressive (AR) 3D generation: the lack of an optimal tokenization strategy.

Current Limitations: State-of-the-art methods rely on Geometric Level-of-Detail (LoD) hierarchies (e.g., octrees, progressive meshes). These were originally designed for rendering and compression, not generative modeling.
Specific Issues with LoD:
1. Token Bloat: Coarse scales still require many spatial tokens to sketch a basic scaffold, leading to high perplexity and poor sample efficiency.
2. Unusable Early Decoding: Aggressive geometric simplification results in early prefixes that decode into geometrically and semantically implausible "scaffolds" rather than recognizable shapes. This prevents "any-prefix" generation, a key feature for efficient AR workflows.
The Gap: There is no method that orders 3D tokens by semantic salience, allowing short prefixes to instantly decode into complete, recognizable shapes while subsequent tokens refine details.

2. Methodology: LoST (Level of Semantics Tokenization)

The authors propose LoST, a learned tokenization scheme where tokens are ordered by semantic importance rather than spatial resolution.

A. Architecture Overview

Encoder (ViT-based):
- Input: 3D shapes encoded as Triplanes (from a VAE, specifically Direct3D).
- Mechanism: Uses a Vision Transformer (ViT) with Register Tokens ( $T_R$ ). Unlike standard patch tokens, these learnable register tokens are concatenated with triplane tokens and attend to them (via causal masking) to capture a summarized, hierarchical representation.
- Nested Dropout: During training, only a random prefix of the register tokens is kept, while the rest are masked. This forces the model to front-load coarse, principal semantics into the first few tokens and encode finer details in later tokens.
Decoder (Generative DiT):
- Instead of deterministic reconstruction (which fails with few tokens), the decoder is a Diffusion-Transformer (DiT).
- It generates the full sequence of latent features conditioned on a variable-length prefix of the register tokens. As the prefix length increases, the generation transitions from "plausible synthesis" to "high-fidelity reconstruction."

B. Key Innovation: Relational Inter-Distance Alignment (RIDA)

A major challenge is that 3D shapes lack direct semantic supervision (unlike 2D images which have DINO features). To align the 3D latent space with semantic concepts, the authors introduce RIDA:

Teacher: DINO features extracted from 2D renderings of the 3D shapes.
Student: A semantic extractor (Transformer) that maps 3D triplane latents to a new feature space.
Loss Function: Instead of regressing absolute feature values (which fails due to modality differences), RIDA aligns the relative distances (topology) between samples in the student space to match the teacher space.
- Global Relational Contrast: Ensures semantically similar shapes are close in the student space.
- Inter-Instance Rank Distillation: Transfers pairwise Euclidean distance rankings (inspired by Relational Knowledge Distillation) to preserve continuous relational structures.
- Spatial Structure Distillation: Aligns intra-instance token affinities to preserve part-level relationships.
Usage: The pre-trained semantic extractor acts as a perceptual loss to guide the diffusion decoder, ensuring that even with few tokens, the output retains principal semantics.

C. AR Generation (LoST-GPT)

The continuous register tokens are modeled by a GPT-style Transformer.
Unlike discrete tokenizers, LoST-GPT operates in continuous space, using a diffusion head for next-token prediction (following MAR).
It supports any-prefix generation: The model can stop generating early if the shape is simple, or continue for complex details.

3. Key Contributions

LoST Tokenization: A novel method that orders 3D tokens by semantic salience, enabling short prefixes to decode into complete, recognizable shapes.
RIDA Loss: A novel 3D semantic alignment objective that reorganizes the triplane latent space to match the semantic topology of DINO features without requiring computationally expensive rendering loops during training.
Efficient AR Generation: Demonstrates that a simple GPT-style Transformer trained on LoST tokens achieves SOTA performance using 0.1%–10% of the tokens required by prior AR models (e.g., 128 tokens vs. 50,000+).

4. Experimental Results

The paper evaluates LoST against baselines like OctGPT (Octree-based) and VertexRegen (Mesh-based).

Reconstruction Quality:
- LoST significantly outperforms baselines in Chamfer Distance (CD), FID, and DINO similarity.
- Crucially, LoST achieves superior results with far fewer tokens. For example, LoST with 16 tokens often outperforms OctGPT with ~15,000 tokens in semantic metrics.
Autoregressive Generation:
- LoST-GPT achieves the lowest FID (34.25) and highest DINO similarity (0.758) on image-to-3D tasks, surpassing ShapeLLM-Omni and Llama-Mesh.
- Token Efficiency: LoST-GPT uses only 128 tokens for high-quality generation, whereas competitors use thousands to hundreds of thousands.
Semantic Retrieval:
- Using RIDA-aligned features, LoST enables effective 3D shape retrieval based on semantics (e.g., finding a "submarine shaped like a fish" based on semantic similarity rather than just geometric shape), outperforming raw triplane features and direct regression baselines.

5. Significance and Impact

Paradigm Shift: Moves 3D tokenization away from geometric LoD (spatial hierarchy) to Semantic LoD (conceptual hierarchy). This aligns 3D generation with the success of semantic-first tokenization in 2D images (e.g., FlexTok).
Efficiency: Drastically reduces the computational cost of 3D AR generation by minimizing the token sequence length required for high-quality output.
Usability: Enables "any-prefix" workflows where users can generate a rough, recognizable concept instantly and refine it only if necessary, making AR models more practical for interactive applications.
Generalizability: The framework is representation-agnostic and has been shown to work on different latent spaces (e.g., TRELLIS), suggesting broad applicability across 3D generative models.

In summary, LoST solves the inefficiency and semantic incoherence of current 3D tokenization by introducing a semantic hierarchy, supported by a novel relational alignment loss (RIDA), resulting in state-of-the-art, highly efficient autoregressive 3D generation.