💻 computer science

3D Primitives are a Spatial Language for VLMs

This paper introduces 3D geometric primitives expressed in code as a powerful spatial language for Vision-Language Models, demonstrating through the SpatialBabel benchmark, the training-free Code-CoT inference strategy, and the self-supervised S $^{3}$ -FT fine-tuning method that this intermediate representation significantly enhances spatial reasoning and generalizes across diverse VLM architectures without requiring human labels.

Original authors: Junze Liu, Kun Qian, Florian Dubost, Kai Zhong, Arvind Srinivasan, Nan Chen, Anping Wang, Sam Zhang, Alejandro Mottini, Qingjun Cui, Tian Wang

Published 2026-05-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Junze Liu, Kun Qian, Florian Dubost, Kai Zhong, Arvind Srinivasan, Nan Chen, Anping Wang, Sam Zhang, Alejandro Mottini, Qingjun Cui, Tian Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, artistic robot that can look at a picture of a room and describe it to you. But here's the weird thing: if you ask this robot, "How many chairs are in this picture?" it might guess wrong and say "five" when there are only three. It seems confused.

However, if you ask that same robot to write a computer program to build that exact room using simple shapes (like cubes, spheres, and cylinders), it does a fantastic job. It writes code that places exactly three cubes in the right spots. It knows the answer perfectly, but it just can't say it in plain English.

This paper, titled "3D Primitives are a Spatial Language for VLMs," explores this strange contradiction and uses it to make these robots smarter.

Here is the breakdown of their discovery and solution, using simple analogies:

1. The Problem: The "Bilingual" Robot

The researchers found that Vision-Language Models (VLMs) are like people who are fluent in a secret language (code) but struggle with a common language (English) when it comes to space.

The Paradox: The robot can build a perfect 3D model of a scene using code (like LEGO instructions), but if you ask it a simple question about that same scene, it hallucinates (makes things up).
The Test (SpatialBabel): They created a giant test called SpatialBabel. Imagine giving the robot the same picture and asking it to rebuild it using six different "languages" (like Three.js, Unity, or even a special JSON format).
The Shock: The robot's performance varied wildly depending on which language it had to use. In one language, it was a master builder; in another, it was a clumsy amateur. This proved that the robot's "spatial brain" was tangled up with its "coding habits."

2. The First Fix: "Code-CoT" (Thinking in Code First)

Since the robot is better at writing code than answering questions, the researchers tried a new trick called Code-CoT (Code Chain-of-Thought).

The Analogy: Imagine you are trying to solve a tricky math word problem. Instead of trying to guess the answer immediately, you first write out the equation on a piece of paper. Once the equation is written, the answer becomes obvious.
How it works: When asked a question like "Is the red cube to the left of the blue sphere?", the robot is forced to first write the code to build the scene. Once the code is written, the robot "reads" its own code to find the answer.
The Result: This worked like a charm for robots that were already good at coding. It boosted their accuracy significantly because it forced them to use their strong "code brain" to solve the "English question."

3. The Second Fix: "S3-FT" (The Self-Teaching Robot)

What about the robots that aren't good at coding yet? They can't use the "write code first" trick because their code is messy.

The researchers invented a method called S3-FT (Self-Supervised Spatial Fine-Tuning).

The Analogy: Imagine a student who is bad at drawing. Instead of hiring a teacher, the student draws a picture, then immediately looks at their own drawing to see what they got right and wrong, and then tries to draw it again. They are their own teacher.
How it works:
1. The robot looks at a simple picture of geometric shapes (cubes, spheres).
2. It tries to write code to rebuild that picture.
3. The researchers take that code and automatically turn it into a "cheat sheet" (a structured list of facts: "There is a red cube at position X").
4. They feed this cheat sheet back to the robot as a lesson, teaching it to answer questions correctly based on its own code.
The Result: The robot learned to understand space without any human teachers or expensive labels. It took the "spatial knowledge" it had hidden inside its coding ability and transferred it to its general conversation skills.

4. The Big Takeaway

The paper concludes that 3D geometric shapes (primitives) expressed in code act as a "universal translator" for these robots.

As a Diagnostic Tool: If a robot fails to answer a spatial question but succeeds at writing code, we know the problem isn't that it "doesn't see" the objects; it's just that it can't translate its vision into words.
As a Vocabulary: By forcing the robot to think in terms of "cubes, spheres, and cylinders" via code, we give it a structured vocabulary to understand the world.

In short: The researchers found that these AI models know more about space than they let on. By making them "speak" in the language of 3D building blocks (code), they can unlock that hidden knowledge and answer simple questions correctly, all without needing a human to teach them the answers.

Technical Summary: 3D Primitives as a Spatial Language for VLMs

Problem Statement

Vision-Language Models (VLMs) exhibit a striking paradox regarding spatial reasoning. While they often fail at simple natural-language spatial questions (e.g., counting objects or identifying relationships) when presented with an image, the same models can frequently generate executable code (e.g., Three.js, Unity C#) that reconstructs the 3D scene with correct object counts, classes, and approximate positions. This suggests that VLMs possess latent spatial knowledge that is not accessible through direct natural-language querying. The paper posits that this discrepancy represents a "task-routing failure," where the model's ability to externalize spatial understanding via structured code is not effectively transferred to direct question-answering. Furthermore, existing benchmarks often conflate spatial competence with familiarity to a specific output format (natural language or a single code language), failing to isolate genuine spatial reasoning capabilities.

Methodology

The authors propose a framework centered on 3D geometric primitives (cubes, spheres, cylinders, etc.) expressed in executable code as a diagnostic tool and intermediate representation. The methodology consists of three main components:

1. SPATIALBABEL Benchmark

To decouple spatial understanding from format familiarity, the authors introduce SPATIALBABEL, a benchmark evaluating VLMs across two scene domains and six scene-code languages.

Scene Domains:
- Primitive Split: 1,000 procedurally generated Blender scenes containing 3–20 geometric primitives with no major occlusion. This provides exact ground truth with zero label noise.
- Hypersim Split: 405 photo-realistic indoor scenes with complex geometries and substantial occlusion, serving as a harder validation domain.
Scene-Code Languages: The benchmark evaluates reconstruction across six distinct formats: Three.js (JavaScript), Unity C#, Blender Python, Open3D Python, Canonical JSON (a declarative schema introduced in this work), and a Scene Language DSL.
Tasks:
- Reconstruct: Generating executable code to recreate the scene.
- QA: Answering natural-language questions about the scene (localization, relationship, counting, existence, comparison).
Metrics: A composite SpatialBabel-Reconstruct-Score aggregating parse rate, F1 (Hungarian-matched object matching), class accuracy, position fidelity, and scale fidelity.

2. Code-CoT (Code Chain-of-Thought)

A training-free inference strategy designed to close the gap at inference time.

Mechanism: Instead of answering a spatial question directly, the model is first prompted to reconstruct the scene as primitive-based executable code. It then answers the spatial question conditioned on its own structured output.
Rationale: This forces the model to route its reasoning through the "spatial language" of code, where it has demonstrated higher competence, before translating that understanding back to natural language.

3. S3-FT (Self-Supervised Spatial Fine-Tuning)

A training-time intervention to distill spatial knowledge into general visual reasoning, specifically targeting models with weaker coding capabilities.

Pipeline:
1. Pseudo-GT Generation: The base VLM generates Three.js reconstruction code for primitive scenes. A syntactic quality filter ensures the code is valid.
2. Annotation Extraction: The generated code is deterministically parsed into structured spatial annotations (object counts, classes, 3D poses, relations). These are combined with rule-generated natural-language QA pairs and scene descriptions.
3. Fine-Tuning: The same VLM is fine-tuned (using LoRA) on this multi-task mixture.
Key Feature: The process is entirely self-supervised; it requires no human labels and no external "teacher" model, as the supervision signal is derived from the model's own high-fidelity code generation.

Key Results

Benchmark Findings (SPATIALBABEL)

Language-Dependent Performance: VLM performance varies drastically across scene-code languages. For the same model on the same images, object-detection F1 scores can vary by up to 5.7× depending on the target language.
Weak Correlation: There is a weak correlation (Pearson $r \approx 0.12$ ) between a model's reconstruction F1 score and its direct QA accuracy. A model can be highly accurate at generating code but poor at answering direct questions, confirming the task-routing failure.
Code-CoT Efficacy: Code-CoT improves QA scores on primitive scenes by up to +6.4% (e.g., Gemini-2.5-Pro) and lifts real-photo CV-Bench-3D accuracy by +5.0% for models with strong coding capabilities. However, it shows diminishing returns or regression on cluttered, photo-realistic scenes where the primitive vocabulary is insufficient.

S3-FT Performance

In-Distribution Gains: Training on primitive images alone significantly improves performance on the SpatialBabel-Primitive-QA task. For Qwen3-VL-8B, S3-FT yields improvements of +4.6% to +8.6% across inference modes.
Out-of-Distribution Transfer: Crucially, training only on primitive scenes transfers to general visual reasoning benchmarks.
- HallusionBench: +17% improvement.
- CV-Bench-2D: +9.7% improvement.
- CV-Bench-3D: +3.3% improvement.
Generalization: The recipe transfers across different model families (Qwen, InternVL, LLaVA) and scales, though gains are capacity-dependent.

Significance and Claims

The paper establishes that 3D geometric primitives in executable code serve as both a diagnostic tool and a transferable spatial vocabulary for VLMs.

Diagnostic Tool: The SPATIALBABEL benchmark reveals that spatial understanding is often entangled with specific code-generation priors. The large performance gaps across languages indicate that standard benchmarks may be measuring format familiarity rather than true spatial reasoning.
Transferable Vocabulary: The success of Code-CoT and S3-FT suggests that VLMs have implicitly "tokenized" 3D space through geometric primitives during pre-training. By leveraging this structured representation, models can access latent spatial knowledge that is otherwise inaccessible via direct natural-language prompting.
Self-Supervised Distillation: S3-FT demonstrates that high-quality spatial reasoning can be distilled into general VLMs without human annotation or teacher models, provided the base model can generate sufficiently accurate primitive reconstructions. This lowers the barrier for improving spatial capabilities in open-weight models.

The authors conclude that while the approach shows clear limits on highly complex, occluded photo-realistic scenes (where the primitive approximation breaks down), it provides a robust pathway for enhancing spatial reasoning in VLMs through the intermediate representation of executable 3D code.