From Human Cognition to Neural Activations: Probing the Computational Primitives of Spatial Reasoning in LLMs

1. Problem Statement

While Large Language Models (LLMs) demonstrate increasing proficiency on spatial reasoning benchmarks, it remains unclear whether this performance stems from structured internal spatial representations (analogous to human cognitive maps) or shallow linguistic heuristics (pattern matching based on surface-level text regularities). Current evaluations rely heavily on output accuracy, which fails to distinguish between genuine spatial computation and memorized linguistic correlations. This lack of mechanistic understanding hinders the development of robust, general-purpose spatial reasoning in foundation models.

2. Methodology

The authors propose a mechanistic evaluation framework grounded in computational theories of human spatial cognition. Instead of broad benchmarks, they decompose spatial ability into three irreducible computational primitives and design controlled task families to isolate them.

A. Task Taxonomy

The study introduces three task families, constructed in English, Chinese, and Arabic to disentangle spatial computation from language-specific cues:

1. Relational Spatial Reasoning: Requires constructing a globally consistent spatial structure from multiple pairwise relations (e.g., "A is left of B, B is above C") to infer indirect relations. This tests relational composition.
2. Perspective Transformation (Mental Rotation): Requires applying global geometric transformations (rotations, reflections, viewpoint changes) to a given configuration. This tests representational transformation and equivariance.
3. Spatial Program Execution: Requires maintaining and updating a latent spatial state over a sequence of movement commands (e.g., "move forward 3, turn right"). This tests stateful spatial updating and path integration.

B. Experimental Setup

• Models: Evaluated multilingual LLMs (Qwen2.5-7B and Llama-3-8B, including base and instruction-tuned variants).
• Inference Protocol: Standard single-pass inference without Chain-of-Thought (CoT) prompting to isolate implicit representations encoded in activations rather than generated reasoning traces.
• Mechanistic Analysis Tools:
  • Linear Probing: Trained to decode spatial variables (relative vectors, orientation, coordinates) from hidden states to measure representational strength ($R^2$).
  • Sparse Autoencoders (SAEs): Used to identify interpretable, sparse features corresponding to spatial concepts.
  • Causal Interventions: Employed activation patching (replacing activations with counterfactual states) and feature ablation (removing specific SAE features) to verify if these representations causally influence model behavior.

3. Key Contributions

1. Computational Decomposition: Moves beyond domain-based benchmarks to a taxonomy based on core cognitive primitives (composition, transformation, updating).
2. Mechanistic Evidence: Provides the first causal evidence that LLMs encode spatial information, while simultaneously demonstrating its limitations (transience and fragmentation).
3. Cross-Linguistic Analysis: Reveals mechanistic degeneracy, showing that similar behavioral performance across languages is achieved via distinct internal pathways.
4. Evaluation Framework: Proposes a shift from benchmark-centric accuracy to mechanistic interpretability for assessing spatial intelligence.

4. Key Results

A. Existence and Nature of Representations

• Intermediate Layer Encoding: Spatial information is consistently encoded in intermediate layers of the network. Linear probes show peak $R^2$ scores (up to 0.37–0.40) in mid-layers (e.g., layer 19 for Qwen2.5).
• Transience: Representations decay sharply in final layers responsible for token prediction. This suggests that while spatial structures are constructed, they are not robustly preserved for final decision-making.
• Causal Influence: Activation patching in intermediate layers (layers 14–18) successfully shifts model outputs toward counterfactual trajectories, proving these representations are functionally relevant, not just decodable artifacts.

B. Task-Specific Performance & Failure Modes

• Relational Reasoning: Shows non-monotonic accuracy drops as complexity increases, suggesting a "compositional collapse" where models lose track of multi-constraint structures.
• Orientation/Perspective: Performance is near chance level with near-zero probe $R^2$. Models rely on linguistic priors (e.g., frequent directions) rather than geometric state tracking.
• Program Execution: Exhibits the strongest performance. However, error analysis reveals that many "successes" are driven by arithmetic competence rather than dedicated spatial reasoning.

C. Cross-Linguistic Findings

• Mechanistic Degeneracy: While behavioral performance is comparable across English, Chinese, and Arabic, the internal features driving these behaviors are largely distinct. SAE analysis shows low overlap in causally important features across languages.
• Language Invariance: Coordinate-based tasks (Program Execution) show the smallest cross-linguistic gap, suggesting numerical formats are more language-invariant than relational or orientation concepts.

D. Feature Analysis

• Sparsity: SAEs identified spatially selective features (e.g., cardinal directions, coordinate ranges) that account for only 3–5% of total features.
• Dissociation: The most frequently activated features are not necessarily the most causally important for spatial reasoning.

5. Significance and Implications

• Limited Spatial Cognition: Current LLMs do not possess robust, general-purpose spatial reasoning. Instead, they exhibit conditional, fragmented, and context-dependent spatial encoding. They construct spatial structures when linguistic cues make it accessible but fail to integrate them into a unified, persistent cognitive map.
• Beyond Benchmarks: High benchmark accuracy is insufficient evidence of spatial intelligence. Models may succeed via arithmetic shortcuts or linguistic heuristics while failing at true geometric reasoning.
• Future Directions: To achieve human-like spatial reasoning, future models need architectures that better preserve and integrate spatial representations across processing stages, rather than relying on transient intermediate states. The study advocates for mechanistic evaluation as a necessary complement to behavioral testing.

Conclusion

The paper concludes that while LLMs have acquired the ability to encode spatial information, these representations are fragile, task-specific, and weakly integrated. The gap between the existence of these representations and their effective use in final decisions highlights a fundamental limitation in current foundation models, suggesting that true spatial intelligence requires more than just scaling or instruction tuning.