🤖 machine learning

Zero-Shot 3D Question Answering via Hierarchical View-to-Token Transportation

This paper proposes \texttt{KeyVT}, a hierarchical zero-shot 3D question answering framework that optimizes input context by selecting task-relevant views based on semantic and geometric criteria and reducing redundancy through optimal transport-based token selection, achieving performance comparable to training-based methods without fine-tuning.

Original authors: Dongsheng Wang, Dawei Su, Hui Huang

Published 2026-06-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Dongsheng Wang, Dawei Su, Hui Huang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a giant, 3D room filled with furniture, objects, and details. You want to ask a very smart AI (a "Vision-Language Model") a specific question about this room, like "What color is the phone on the table?"

The problem is that the AI is like a person with a very small backpack. It can only carry a limited amount of "visual luggage" (images or data) at one time. If you try to shove the entire 3D room into that backpack, it won't fit, or the AI will get overwhelmed and confused.

This paper introduces a new method called KeyVT to solve this packing problem. It acts like a super-efficient travel agent that helps you pack the most important items into that small backpack so the AI can answer your question perfectly, without needing to be retrained on new data.

Here is how KeyVT works, broken down into two simple steps:

Step 1: Choosing the Best "Postcards" (Key Views)

Imagine you are standing in that 3D room and you take hundreds of photos from different angles. You can't show the AI all of them.

The Old Way: Most methods just pick photos that look similar to your question (e.g., if you ask about a phone, they pick photos that look like phones) or pick photos at random intervals. This often misses the context, like the table the phone is sitting on, or picks too many photos of the same wall.
The KeyVT Way: KeyVT acts like a smart tour guide. It looks at your question and the geometry of the room (where the camera is standing and which way it's facing).
- It divides the room into "neighborhoods" (sub-scenes) based on how close the photos are to each other in space.
- It then decides: "This neighborhood has the phone and the table, so I'll send 3 photos from there. That other neighborhood is just an empty hallway, so I'll skip it."
- The Result: You get a set of photos that are not only relevant to your question but also show the surrounding environment in a way that makes spatial sense.

Step 2: Picking the Best "Stickers" (Key Tokens)

Even after picking the best photos, each photo is made up of thousands of tiny pixels (called "tokens"). If you send all those pixels, you still run out of backpack space.

The Old Way: Some methods just group similar pixels together (like clustering) or throw away the ones that look boring. Sometimes this accidentally throws away a crucial detail, like the specific color of the phone.
The KeyVT Way: KeyVT uses a mathematical concept called Optimal Transport. Think of this as a "moving company" problem.
- Imagine you have a warehouse full of millions of boxes (all the pixels from your photos).
- You need to move them to a new, smaller warehouse (the AI's limited memory).
- Instead of just grabbing boxes randomly, KeyVT calculates the most efficient way to "transport" the essence of the big warehouse into the small one. It finds the fewest number of "representative boxes" (tokens) that can perfectly cover everything important in the original warehouse.
- The Result: It compresses the data so tightly that you can fit the same amount of information into half the space. It removes the duplicate "noise" (like 50 photos of the same empty wall) while keeping the unique, important details.

Why This Matters

The paper claims that by using this two-step "Travel Agent" approach:

It works without training: You don't need to teach the AI new things. It works with existing, powerful AI models right out of the box.
It's better than the competition: In tests on three different 3D datasets, KeyVT answered questions more accurately than other methods that try to pick key photos or compress data.
It's efficient: It allows the AI to "see" more of the 3D world without needing a bigger computer or more memory.

In a nutshell: KeyVT is a smart filter. It first picks the best angles to look at a 3D scene, and then it picks the best details from those angles, ensuring the AI gets a clear, complete, and non-redundant picture of the world to answer your questions.

Technical Summary: Zero-Shot 3D Question Answering via Hierarchical View-to-Token Transportation

Problem Statement

Recent advancements in Vision-Language Models (VLMs) have demonstrated strong spatial reasoning capabilities in 2D domains. Researchers have attempted to extend these capabilities to 3D scene understanding by sampling multiple 2D views from a 3D point cloud and feeding them into pre-trained 2D VLMs. However, this paradigm faces a critical bottleneck: the input budget constraint.

Current methods typically sample a fixed, small number of views (e.g., 8 or 16) due to token limits. Existing strategies for selecting these views often rely on uniform sampling or simple semantic similarity, which introduces noise or overlooks critical spatial context not explicitly mentioned in the query. Furthermore, even when "key views" are selected, they often contain significant redundancy across regions (overlapping semantic information). Simply feeding these views into VLMs wastes the limited input budget on redundant tokens, preventing the model from accessing sufficient fine-grained 3D details necessary for complex reasoning. While 3D-specific Large Language Models (3D-LLMs) exist, they rely on scarce, high-quality 3D-text paired data, limiting their scalability and zero-shot applicability.

Methodology: KeyVT

The authors propose KeyVT, a hierarchical, tuning-free framework designed to optimize input context collection at two levels: View and Token. The goal is to preserve maximum task-relevant 3D details within a fixed input budget.

1. KeyV: Geometry-Aware Key-View Selection

Unlike previous methods that select views based solely on semantic similarity to the query, KeyV incorporates geometric parameters (camera position and orientation) to ensure spatial consistency.

Spatial Segmentation: The method calculates a combined distance metric between the first view and all subsequent views, accounting for both positional distance and angular orientation differences. This metric is used to segment the 3D scene into spatially consistent sub-scenes.
Adaptive Allocation: Not all sub-scenes are equally relevant. The framework computes a relevance score for each sub-scene by combining the maximum and mean semantic similarity scores between the query and views within that sub-scene (using a pre-trained encoder like BLIP2).
View Selection: Based on these relevance scores and the size of the sub-scenes, the method adaptively allocates a specific number of key views ( $N_l$ ) to each sub-scene. This ensures that more relevant or larger sub-scenes contribute more views, capturing coherent spatial contexts rather than isolated, potentially noisy views.

2. KeyT: OT-Guided Key-Token Selection

After selecting the key views, the framework addresses the redundancy among the visual tokens (patches) within these views.

Optimal Transport (OT) Formulation: The method treats the set of all tokens from the selected views as a redundant distribution ( $P$ ) and aims to find a compact set of "virtual tokens" ( $Q$ ) that summarizes the information. This is framed as an Optimal Transport problem, minimizing the cost of transporting mass from $P$ to $Q$ .
Virtual Token Learning: The virtual tokens are optimized via gradient descent (using the Sinkhorn algorithm for efficiency) to minimize the OT distance (Sinkhorn distance) between the original view tokens and the virtual tokens. This allows the model to discover diverse and representative tokens that cover the feature space of the original views.
Token Grounding: Since VLMs cannot directly process abstract virtual tokens, the framework "grounds" each virtual token by selecting the most relevant real patch tokens from the original views. This selection is based on the learned transport plan (the probability matrix from OT), which indicates which original tokens best represent each virtual token. The final input context is formed by concatenating these selected key tokens.

Key Contributions

Hierarchical Framework (KeyVT): A novel two-stage approach that first selects spatially consistent key views and then compresses them into key tokens, enabling 2D VLMs to access rich 3D details without fine-tuning.
Geometry-Aware Selection: The introduction of camera parameters into the view selection process to model spatial relationships, moving beyond simple semantic retrieval to capture coherent sub-scenes.
OT-Based Token Compression: The application of Optimal Transport theory to identify representative tokens across views, effectively reducing redundancy while preserving diverse visual evidence under a strict token budget.
Zero-Shot Performance: Demonstration that this tuning-free approach achieves performance comparable to, and in some cases exceeding, training-based 3D-LLMs and other fine-tuned methods.

Experimental Results

The authors evaluated KeyVT on three benchmarks: ScanQA, SQA3D, and VSI-Bench, using various 2D VLM backbones (e.g., LLaVA-OneVision, Qwen2.5-VL, LLaVA-Video).

Benchmark Performance: KeyVT consistently outperformed existing tuning-free baselines (such as AKS, DivPrune, and FLoC) across all metrics (CIDEr, BLEU, EM, etc.). For instance, on ScanQA with LLaVA-Video-7B, KeyVT achieved a CIDEr score of 100.7, surpassing the base model and other compression methods.
Comparison with Training-Based Methods: The framework achieved performance comparable to or better than training-based approaches like cdViews and Video-3D LLM, despite requiring no task-specific training data or parameter updates.
Ablation Studies:
- Removing the geometry-aware design or sub-scene partitioning led to significant performance drops, validating the importance of spatial consistency.
- The OT-based token compression (KeyT) proved superior to other compression strategies (DivPrune, FLoC), particularly in preserving diverse and representative tokens.
- The method demonstrated robustness to noise in camera parameters (up to 5% noise) and performed well even when camera parameters were estimated by external tools (FastVGGT).
Efficiency: Inference time analysis showed that KeyV has a cost comparable to AKS, while KeyT is more efficient than FLoC, offering a favorable trade-off between computational cost and performance.

Significance and Claims

The paper claims that KeyVT significantly lowers the barrier for deploying 3D-aware AI systems. By leveraging powerful pre-trained 2D VLMs without the need for expensive 3D-text alignment training or large-scale labeled 3D datasets, the method enables effective zero-shot 3D scene understanding.

The authors emphasize that the quality of input context is paramount. By hierarchically selecting views that are both semantically relevant and spatially coherent, and then compressing tokens to eliminate redundancy, KeyVT allows VLMs to reason about 3D environments more effectively. This approach promotes resource-efficient 3D perception, making advanced spatial reasoning accessible to researchers and practitioners with limited computational resources, with potential applications in robotics, embodied AI, and indoor navigation. The work is presented as a step toward democratizing 3D understanding capabilities by reducing reliance on specialized 3D training pipelines.