💻 computer science

SpatialStack: Layered Geometry-Language Fusion for 3D VLM Spatial Reasoning

The paper introduces SpatialStack, a hierarchical fusion framework that progressively aligns multi-level vision, geometry, and language features to overcome the limitations of existing VLMs in 3D spatial reasoning, resulting in the state-of-the-art VLM-SpatialStack model.

Original authors: Jiang Zhang, Shijie Zhou, Bangya Liu, Achuta Kadambi, Zhiwen Fan

Published 2026-03-31

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jiang Zhang, Shijie Zhou, Bangya Liu, Achuta Kadambi, Zhiwen Fan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to navigate a messy living room. You want it to know: "Is the coffee table closer to me than the sofa?" or "How far is the bed from the door?"

Current AI models (the "brains" of these robots) are great at recognizing what things are (e.g., "That's a sofa!"). But they are terrible at understanding where things are in 3D space. They often get lost, bump into walls, or can't tell left from right.

The paper "SpatialStack" introduces a new way to fix this. Here is the simple breakdown:

The Problem: The "One-Size-Fits-All" Mistake

Think of a standard AI model like a student who only reads the summary of a textbook.

The Vision Encoder: This is the student's eyes. It looks at a photo and says, "I see a chair."
The Geometry Encoder: This is a special tool that measures the room. It knows the chair is 2 meters away and 30cm tall.
The Old Way: Previous AI models tried to combine these two by taking the final summary from the Geometry tool and shoving it into the student's brain at the very end.

The Flaw: By the time you get to the "final summary," all the fine details are gone. It's like trying to build a house using only the blueprint's title page. You know that there is a house, but you don't know where the windows go or how thick the walls are. The AI loses the "fine-grained" details needed for precise navigation.

The Solution: The "Layered Sandwich" (SpatialStack)

The authors realized that understanding space happens in layers, just like peeling an onion or building a house.

Shallow Layers: These see the tiny details (edges, corners, exact distances).
Deep Layers: These see the big picture (the whole room layout, the relationship between objects).

SpatialStack stops trying to mix everything at the end. Instead, it creates a layered sandwich:

It takes the tiny details from the Geometry tool and feeds them into the early layers of the AI's brain.
It takes the big picture from the Geometry tool and feeds them into the later layers of the AI's brain.

The Analogy: Imagine a construction crew.

Old Method: The architect draws the whole building, hands the final blueprint to the bricklayer, and says, "Build this." The bricklayer has no idea where the pipes go or how to cut the bricks.
SpatialStack Method: The architect talks to the bricklayer every step of the way.
- "Here is the exact measurement for this brick (Shallow Layer)."
- "Now, here is how this wall fits into the whole house (Deep Layer)."
- "And finally, here is the plan for the whole neighborhood (Context)."

By feeding information at every single step of the thinking process, the AI learns to "see" in 3D, not just 2D.

Why This Matters

This new method, called VLM-SpatialStack, allows AI to:

Navigate: "Walk to the sofa, but stop 2 feet before you hit it."
Measure: "How far is the bed from the window?"
Reason: "If I stand here, is the blackboard to my left or right?"

The Result

The paper tested this on many difficult 3D puzzles. The new "Layered Sandwich" AI beat all the previous models. It didn't just get better at one thing; it got better at everything related to space, from simple distance checks to complex navigation plans.

In short: Instead of giving the AI a single, blurry map at the end of the journey, SpatialStack gives it a high-definition GPS update at every single step of the way. This allows robots and AI assistants to finally understand the physical world the way humans do.

1. Problem Statement

Large Vision-Language Models (VLMs) have made significant strides in semantic understanding but struggle with reliable 3D spatial reasoning, a critical capability for embodied AI and physical agents.

The Core Limitation: Existing VLMs fail to effectively encode 3D geometry and spatial relationships, often confusing "left" vs. "right" or failing to estimate relative distances.
The Bottleneck in Current Methods: Recent attempts to integrate geometry into VLMs (e.g., using encoders like DUST3R or VGGT) typically rely on single-level fusion. They extract only the final-layer features from geometry encoders and fuse them with vision tokens.
The Consequence: This approach discards rich hierarchical geometric signals. Shallow layers contain fine-grained local structures (crucial for depth/edge detection), while deep layers contain global semantics. Fusing only the deep layer creates a bottleneck, limiting the model's ability to perform both low-level perception (e.g., "which object is closer?") and high-level reasoning (e.g., "navigate to the sofa").

2. Methodology: SpatialStack

The authors propose SpatialStack, a general hierarchical fusion framework that aligns vision, geometry, and language representations across the model hierarchy rather than just at the end.

A. Key Architectural Components

Multi-View Geometry Encoder: The system uses VGGT (Visual Geometry Grounded Transformer) as the geometry encoder. Unlike previous methods that only use the output, SpatialStack extracts intermediate hidden states from specific layers (e.g., layers 11, 17, and 23) to capture multi-level geometric abstractions.
Layered Geometry-Language Fusion:
- Instead of fusing geometry features into the vision encoder (the conventional approach), SpatialStack injects them directly into the LLM Decoder.
- Hierarchical Alignment: Features from shallow geometry layers are injected into early LLM decoder layers, while features from deep geometry layers are injected into deeper LLM layers.
- Mechanism: The geometry tokens are processed by layer-specific projectors (mergers) to match the spatial resolution and embedding dimension of the LLM's hidden states. They are then added as residuals to the corresponding LLM decoder layers.
Training Strategy:
- The Vision Encoder and Geometry Encoder (VGGT) are frozen.
- Only the LLM decoder and the geometry token merger modules are trainable.
- Training uses standard next-token prediction (cross-entropy loss) on a diverse dataset of spatial instructions, allowing the model to learn how to align and utilize geometric priors without auxiliary losses.

B. Theoretical Insight

The paper provides a quantitative analysis showing a hierarchical correspondence:

Shallow Geometry Layers: Provide fine-grained local cues, essential for low-level tasks (depth estimation, distance comparison).
Deep Geometry Layers: Encode global structural relationships, essential for high-level tasks (path planning, relational reasoning).
Naive Multi-Layer Fusion: Simply adding all layers to the vision pathway causes feature interference. SpatialStack solves this by progressively aligning specific geometry layers with specific decoder layers, ensuring synergy rather than noise.

3. Key Contributions

Systematic Analysis: The first study to analyze how fusion layers across vision, geometry, and LLM decoders affect spatial reasoning granularity. It reveals that shallow layers aid perception while deep layers aid reasoning.
SpatialStack Framework: A novel, model-agnostic architecture that performs layered geometry-language fusion. It moves beyond late-stage fusion to inject geometric cues throughout the decoding process.
VLM-SpatialStack Implementation: A concrete realization using the Qwen series (Qwen2.5-VL and Qwen3.5) that achieves state-of-the-art (SOTA) performance.
Generalization: The framework demonstrates robust zero-shot generalization across diverse 3D tasks without compromising general multimodal capabilities.

4. Experimental Results

The authors evaluated VLM-SpatialStack on multiple benchmarks, including VSI-Bench, SPAR-Bench, BLINK-Spatial, and CV-Bench.

State-of-the-Art Performance:
- VSI-Bench: SpatialStack-5B (based on Qwen3.5) achieved 67.52% overall accuracy, outperforming all other open-source models (including Spatial-MLLM, VG-LLM, and Cambrian-S) and approaching proprietary models like Gemini-2.5 Pro.
- CV-Bench: Achieved 85.5% overall accuracy, setting a new SOTA for 2D and 3D spatial perception.
- SPAR-Bench: Scored 71.39%, significantly outperforming baselines.
Ablation Studies:
- Injection Depth: Injecting geometry features directly into the LLM decoder outperformed fusing them into the vision encoder.
- Layer Selection: Using a combination of layers (11, 17, 23) with progressive alignment yielded better results than single-layer injection or naive multi-layer fusion.
- Ordering: The progressive hierarchical alignment (Shallow Geo $\to$ Shallow LLM) was proven superior to reverse or random ordering.
General Capabilities: The model maintained strong performance on general multimodal benchmarks (MMBench, Video-MME), indicating no catastrophic forgetting of general language or visual skills.

5. Significance

Paradigm Shift: SpatialStack challenges the prevailing "late fusion" paradigm in multimodal AI. It demonstrates that hierarchical alignment is crucial for tasks requiring both fine-grained detail and global context.
Embodied AI: By enabling reliable 3D spatial reasoning (e.g., navigation, manipulation, distance estimation), this work bridges the gap between perception and action for physical AI systems.
Extensibility: As a model-agnostic framework, SpatialStack can be applied to any base multimodal LLM, offering a scalable path toward "spatially aware" AI agents.
Efficiency: By freezing the heavy geometry and vision encoders and only training the fusion/decoder layers, the approach is computationally efficient while delivering superior results.

In summary, SpatialStack solves the 3D reasoning bottleneck in VLMs by recognizing that geometry is hierarchical. By injecting multi-level geometric features directly into the language backbone in a structured, layer-by-layer manner, it achieves a level of spatial understanding previously unattainable by open-source models.