Technical Summary: Uncovering and Shaping the Latent Representation of 3D Scene Topology in Vision-Language Models

1. Problem Statement

Decades of cognitive science suggest that humans navigate 3D environments by forming "cognitive maps"—allocentric, topology-preserving representations of spatial relations that are distinct from egocentric visual inputs. While modern Vision-Language Models (VLMs) demonstrate emergent spatial reasoning capabilities from 2D inputs, it remains unclear whether they construct analogous internal 3D representations. Existing diagnostic tools (e.g., attention maps, end-to-end accuracy) fail to determine if VLMs possess a latent 3D allocentric structure or if they rely on superficial 2D correlations and language shortcuts. Furthermore, current approaches to improving VLM spatial reasoning often rely on injecting explicit 3D modalities (e.g., depth maps) or massive data scaling, rather than shaping the model's existing latent geometry.

The core problem addressed is twofold:

1. Discovery: Do current VLMs inherently encode a latent topological map of 3D scenes, and if so, is it accessible?
2. Shaping: Can this latent representation be mathematically identified and regularized to improve spatial reasoning without architectural changes or external 3D data?

2. Methodology

2.1 Uncovering the Latent Spatial Subspace

The authors propose that while VLMs do encode 3D topology, this signal is heavily overshadowed by non-geometric visual semantics (e.g., color, shape). To isolate the spatial component, they introduce a linear feature extraction technique based on cross-scene averaging:

• Data: A synthetic dataset (SynSpat3D) with 1,000 scenes containing objects with randomized positions, colors, and shapes. This ensures that object identity (color/shape) is statistically independent of spatial position.
• Per-Object Aggregation: Visual tokens corresponding to an object are aggregated across temporal frames and spatial patches to form a per-object activation vector $h^{(s,o)}_\ell$ at decoder layer $\ell$.
• Disentanglement via Averaging: Assuming the latent representation is a linear sum of an identity-conditional component ($uid(o)$) and a position-conditional component ($usp(x)$), the authors average activations for the same object identity across many randomized scenes. Since positions are random, the spatial component averages to zero, leaving a consistent estimate of the identity prototype $\bar{h}(o)$.
• Projection: An orthogonal basis $W_\ell$ is constructed from these identity prototypes. The spatial subspace is then extracted by projecting the original activations onto the orthogonal complement of this basis: $\tilde{h}^{(s,o)}_\ell = (I - W_\ell W_\ell^\top) h^{(s,o)}_\ell$.

2.2 Mathematical Characterization

The authors prove that the extracted spatial subspace corresponds to the Laplacian eigenmaps of the scene's 3D graph:

• Theorem 1: Minimizing the Dirichlet energy of the representation subject to variance constraints yields a representation whose principal components align with the eigenvectors of the scene's graph Laplacian.
• Theorem 2: In the continuous limit, these eigenvectors converge to the physical 3D coordinates (Cartesian axes) of the scene.
• Causal Verification: Through activation steering, the authors demonstrate that injecting vectors along the principal axes of this extracted subspace linearly shifts the model's internal 3D readout, confirming the subspace is causally used by the model.

2.3 Latent Regularization via Dirichlet Energy

Motivated by the geometric identification, the authors introduce a Dirichlet-energy regularizer for fine-tuning:

• Loss Function: A single-term regularizer is added to the standard language modeling loss (cross-entropy) during Supervised Fine-Tuning (SFT):
  $$\mathcal{L}(\theta) = \mathcal{L}_{LM}(\theta) + \lambda_{dir} \cdot \mathcal{E}_X(P^\perp H_\ell(\theta))$$
  where $\mathcal{E}_X$ is the Dirichlet energy of the extracted spatial activations $P^\perp H_\ell$ over the scene's 3D Gaussian-kernel graph.
• Theoretical Guarantees:
  • Theorem 3 (Realizability): Shaping the latent space to minimize Dirichlet energy ensures that spatial relational questions become linearly separable, allowing the linear language head to answer them perfectly.
  • Theorem 4 (Sample Complexity): The regularizer collapses task-irrelevant variance, reducing the effective dimension of the learning problem from the high-dimensional residual stream ($d \approx 4000$) to exactly 3 (the spatial dimensions). This theoretically reduces sample complexity by a factor of $d/3$.
  • Theorem 5 (Risk Decomposition): A moderate regularization weight $\lambda$ guarantees improvement in spatial tasks, while excessive $\lambda$ risks distorting the residual stream and harming language capabilities.

3. Key Results

3.1 Diagnostic Findings

• Representational Imbalance: Direct probing reveals that VLMs (Qwen2.5-VL-7B, InternVL3-8B) encode object color and shape with near-perfect accuracy, while 3D position is only partially recoverable.
• Semantic Dominance: Behavioral counterfactuals show that shuffling object colors causes the model to flip spatial predictions in 41% of cases, nearly as often as physically scrambling the 3D layout (48%). This confirms the model relies heavily on semantic cues rather than a robust geometric map.
• Layer Specialization: The extracted spatial subspace is most distinct in the "middle" layers of the decoder (e.g., layers 13–21 in Qwen2.5-VL), suggesting a functional division where early layers bind visual features, middle layers construct 3D topology, and late layers handle linguistic reasoning.

3.2 Fine-Tuning Performance

Applying the Dirichlet-energy regularizer to 500 steps of LoRA fine-tuning on simple synthetic data yields significant zero-shot improvements on real-world benchmarks:

• Benchmarks: VSI-Bench (fine-grained 3D reasoning) and MindCube (spatial mental modeling).
• Performance Gains:
  • Qwen2.5-VL-7B: Overall accuracy improved from 29.3% (Naive LoRA) to 35.6%.
  • InternVL3-8B: Overall accuracy improved from 36.7% to 42.4%.
• Specific Task Improvements: The method showed the most significant gains in tasks requiring scene topology understanding, such as relative direction (up to +12.1% improvement) and absolute distance estimation.
• Baselines: The method outperformed standard Naive LoRA, 2D spatial regularization (which failed to capture true 3D geometry), and text-domain cognitive map prediction (which forced the model to output coordinates as text rather than internalizing them).

4. Significance and Claims

The paper claims to provide a principled methodology for developing VLMs with robust, intrinsic cognitive maps. Its significance lies in:

1. Demonstrating Latent Existence: Proving that standard VLMs already possess a latent 3D topological representation, but it is obscured by dominant visual semantics.
2. Mathematical Shaping: Introducing a mathematically grounded regularization (Dirichlet energy) that aligns the model's internal geometry with physical 3D space, derived from spectral graph theory.
3. Data Efficiency: Showing that extreme data efficiency is achievable by reducing the effective dimensionality of the learning problem, allowing generalization to complex real-world benchmarks from minimal synthetic data (500 steps).
4. Orthogonality: The approach is orthogonal to existing methods that rely on external 3D modalities or massive data scaling, offering a way to enhance spatial reasoning within the standard VLM architecture without architectural modifications.

The authors conclude that by isolating and shaping this latent subspace, it is possible to significantly enhance the spatial reasoning capabilities of VLMs, moving them closer to the topology-preserving cognitive maps observed in biological agents.