💻 computer science

Multi-View Relational Distillation for Spatial Reasoning with Vision-Language Models

This paper proposes Multi-View Relational Distillation (MVRD), a method that enhances the geometric reasoning capabilities of Vision-Language Models by distilling patch-wise cosine similarities across views from geometry-grounded teachers, thereby improving spatial understanding while preserving pretrained language alignment and avoiding the computational overhead of feature fusion.

Original authors: Kiet T. Nguyen, Hanbo Shim, Jinwoo Kim, Seunghoon Hong

Published 2026-08-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Kiet T. Nguyen, Hanbo Shim, Jinwoo Kim, Seunghoon Hong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to understand the world. You give it a camera and a microphone, and you want it to not just recognize a "cup" or a "chair," but to understand where those things are in 3D space, how far apart they are, and how they relate to each other. This is the world of Vision-Language Models (VLMs). Think of these models as super-smart students who have read every book in the library and seen millions of pictures. They are great at matching words to images—like knowing that the word "dog" goes with a picture of a furry animal. But here's the catch: while they are brilliant at language, they are often terrible at geometry. They might know what a "microwave" looks like, but they struggle to figure out if it's to the left or right of the toaster, or how big it actually is. This is a huge problem for robots, self-driving cars, and AI assistants that need to navigate real physical spaces. Scientists have been trying to fix this "spatial blindness" by teaching these models geometry, but the usual methods have been like trying to fix a leaky roof by replacing the whole house: they either break the model's ability to understand language, or they make the model so huge and slow that it's useless for real-time tasks.

Enter a new idea called Multi-View Relational Distillation (MVRD), proposed by researchers Kiet T. Nguyen and colleagues. Instead of forcing the student model to memorize the exact "shape" of the teacher's knowledge (which breaks its language skills), they decided to teach it the relationships between things. Imagine you are trying to teach someone the layout of a city. The old way was to force them to memorize the exact GPS coordinates of every building. If they got the coordinates slightly wrong, they might end up in the wrong neighborhood, forgetting where the bakery was. The new way, MVRD, is like saying, "Don't worry about the exact coordinates. Just remember that the bakery is next to the park, and the library is across the street from the bakery." By focusing on these relative connections (the "relations") rather than the absolute positions, the model learns to understand space without losing its ability to speak and understand language.

The researchers tested this on a model called LLaVA-Video-7B. They compared their new method against the old "feature distillation" approach (the coordinate memorization method). The results were clear: the old method made the model better at geometry but caused it to fail at simple language tasks, like counting objects or tracking the order in which things appeared. It was like a student who could draw a perfect map but forgot how to read the street signs. In contrast, MVRD improved the model's spatial reasoning significantly—boosting its average accuracy on a spatial reasoning test (VSI-Bench) to 59.9% for the standard version and 60.4% for a special "Dual Pathway" version—while keeping its language skills intact. In fact, this new method came within just 0.5 points of the best existing method that uses a massive, separate geometry engine, but it did so with far fewer extra parameters (only 0.19B additional parameters) and much lower delay (latency).

The paper suggests that by focusing on "cosine similarities" (a mathematical way of measuring how similar the angles between different pieces of information are) across multiple views of a scene, the model can learn the "skeleton" of 3D space without overwriting its "flesh" of language understanding. This approach also proved robust, working well on different types of base models and generalizing to complex 3D tasks like finding specific objects in a room or describing a scene in detail. Essentially, the researchers found a way to give AI a sense of direction and depth without making it forget how to talk, offering a lighter, faster, and smarter path for robots to navigate our physical world.

Technical Summary: Multi-View Relational Distillation for Spatial Reasoning with Vision-Language Models

1. Problem Statement

Vision-Language Models (VLMs) have demonstrated strong capabilities in image and video understanding but suffer from geometrically fragile visual-spatial representations. This fragility leads to failures in spatial reasoning tasks essential for embodied AI, robotics, and autonomous driving, such as inferring relative distances or object localization.

Existing approaches to grounding VLMs in geometry face two primary limitations:

Supervised Fine-Tuning (SFT): Training on spatial question-answering datasets often perpetuates "spurious visual representations" without fundamentally correcting the underlying geometric understanding.
Feature Fusion: Methods that fuse features from large, geometry-grounded vision models (e.g., VGGT, CUT3R) into VLMs improve spatial reasoning but substantially increase model size and inference latency, posing challenges for real-time applications.

A third approach, Knowledge Distillation, attempts to transfer geometric knowledge from a teacher model to a student VLM without increasing inference cost. However, the authors identify a critical failure mode in standard feature distillation: directly matching the student's multi-view visual features to a geometry-grounded teacher's features disrupts the pretrained alignment between visual and textual representations. This trade-off degrades the model's ability to perform linguistic grounding tasks, such as tracking object appearance order or estimating object size based on text queries.

2. Methodology: Multi-View Relational Distillation (MVRD)

To address the trade-off between geometric grounding and vision-language alignment, the authors propose Multi-View Relational Distillation (MVRD).

Core Concept

Instead of distilling the feature vectors themselves (which forces the student features to converge to a specific geometric space), MVRD distills the cosine similarities (relations) between features across multiple views.

Hypothesis: Multi-view relations encode sufficient geometric correspondences to improve spatial understanding. Crucially, because matching pairwise similarities does not uniquely determine the absolute feature vectors, the student model retains the freedom to select feature representations that satisfy the geometric objective while remaining close to its pretrained vision-language space.
Mathematical Formulation:
- Let $v_{1:T}$ be the student's vision tokens and $\bar{g}_{1:T}$ be the teacher's geometry-grounded features.
- A relational representation $R$ is constructed using a similarity function $r$ (specifically cosine similarity): $R(h)[i, j] = r(h[i], h[j])$ .
- The distillation loss $L_{RD}$ minimizes the difference between the relational matrices of the student and the teacher:
  $L_{RD} := \ell_{RD}(R(f(v_{1:T})), R(\text{sg}(\bar{g}_{1:T})))$
- Here, $f$ is a position-wise MLP projection, and $\text{sg}$ denotes the stop-gradient operator. The total training objective combines the standard SFT loss with the relational distillation loss ( $L_{SFT} + \gamma L_{RD}$ ).

Architectural Enhancements

To further minimize drift from the pretrained space, the authors introduce a Dual Pathway architecture:

Geometric Adapter: A Low-Rank Adaptation (LoRA) module supervised by the composite loss ( $L_{SFT} + \gamma L_{RD}$ ) learns multi-view relational signals.
Semantic Adapter: A parallel LoRA module supervised only by $L_{SFT}$ preserves language-aligned behavior.
Integration: The outputs of these adapters are summed after the distillation layer and passed through the remaining VLM blocks. This decoupling ensures that geometric learning does not perturb the language-relevant subspace.

Design Choices

Similarity Function: Cosine similarity is chosen for its scale and orientation invariance, allowing the model to match relative angles without forcing feature magnitudes to align.
Loss Function: Row-wise negative Pearson correlation is used instead of Mean Squared Error (MSE) to improve robustness to scale variations in similarity scores.
Causal Masking: To adhere to VLM causal attention mechanisms, causal masking and diagonal zeroing are applied before computing the loss.

3. Key Contributions

Diagnosis of Feature Distillation Failure: The paper identifies and demonstrates that standard feature distillation degrades vision-language alignment, causing performance drops in linguistic grounding tasks (e.g., object counting, size estimation) despite improving geometric tasks.
Proposal of MVRD: A novel distillation framework that transfers geometric knowledge via multi-view relational similarities rather than raw features, preserving the pretrained vision-language space.
Dual Pathway Architecture: A lightweight parameter separation mechanism that further isolates geometric learning from semantic alignment, balancing performance across task types.
Efficiency: MVRD achieves performance comparable to feature fusion methods with significantly fewer additional parameters (<25% overhead) and lower latency (<16% overhead compared to feature fusion baselines).

4. Experimental Results

The method was evaluated on VSI-Bench (Visual-Spatial Intelligence Benchmark) and various 3D scene understanding datasets.

VSI-Bench Performance:
- MVRD outperformed Supervised Fine-Tuning (SFT) and standard Feature Distillation (3DRS) on 6 of 8 tasks.
- It showed consistent gains in linguistic grounding tasks (e.g., Object Size, Appearance Order), where feature distillation previously failed.
- The MVRD + Dual Pathway variant achieved an average accuracy of 60.4%, narrowing the gap to the state-of-the-art feature fusion method (VLM-3R) to just 0.5 points, while using the same backbone and dataset.
Representation Analysis:
- Qualitative and quantitative analysis confirmed that MVRD maintains high text-vision alignment (measured by text-based object detection precision) while effectively modeling geometric relations.
- Feature distillation was shown to misalign text tokens with their corresponding visual patches, whereas MVRD preserved this correspondence.
Generalization:
- MVRD successfully transferred to 3D scene understanding tasks (ScanRefer, Multi3DRefer, Scan2Cap, ScanQA, SQA3D), outperforming prior 3D scene understanding generalists and the feature-distillation baseline 3DRS.
- The method proved robust across different VLM backbones (LLaVA-Video, InternVL3, Qwen2.5-VL).

5. Significance and Claims

The paper claims that MVRD offers a computationally efficient alternative to feature fusion for grounding VLMs in 3D geometry. By shifting the distillation target from features to relations, the method resolves the previously undiagnosed conflict between geometric grounding and language alignment.

The authors emphasize that MVRD allows VLMs to acquire geometric understanding "without trading off vision-language alignment." The Dual Pathway variant further demonstrates that decoupling geometric and semantic objectives yields a more balanced model that approaches the performance of heavy feature-fusion architectures with a fraction of the inference cost. The work suggests that relational representations are a more adequate and minimal mechanism for transferring geometric knowledge to pretrained VLMs.