💬 NLP

Grid Spatial Understanding: A Dataset for Textual Spatial Reasoning over Grids, Embodied Settings, and Coordinate Structures

This paper introduces GSU, a text-only grid dataset for evaluating LLM spatial reasoning across navigation, localization, and composition tasks, revealing that while frontier models perform well and small models can match them via fine-tuning, most struggle with embodied frames of reference and 3D shape identification, with visual modalities failing to provide generalizable 3D spatial understanding.

Original authors: Risham Sidhu, Julia Hockenmaier

Published 2026-03-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Risham Sidhu, Julia Hockenmaier

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to navigate a giant, invisible 3D maze made of colorful blocks. You can't show the robot a picture of the maze; you can only give it a list of coordinates (like "Block A is at 2, 5, 1") and a set of instructions in plain English.

This paper introduces a new test called GSU (Grid Spatial Understanding) to see if Large Language Models (LLMs)—the brains behind AI chatbots—can actually "think" in 3D space using only words, or if they are just guessing based on patterns they've seen before.

Here is a breakdown of the study using simple analogies:

1. The Three Games (The Tasks)

The researchers created three different "games" to test the AI's spatial brain:

The Navigation Game (The Blindfolded Hiker):
- The Setup: Imagine you are blindfolded in a room. Someone tells you, "Walk 3 steps forward, turn right, walk 2 steps." You have to know exactly where you end up.
- The Twist: There are two ways to give directions.
  - Cardinal (The Map): "North, South, East, West." These never change. If you turn around, "North" is still North.
  - Egocentric (The Body): "Forward, Left, Right." These change depending on which way you are facing. If you turn around, "Forward" is now the opposite direction.
- The Result: Most AIs are great at the "Map" version but get completely lost in the "Body" version. They struggle to update their mental map when they turn a corner.
The Object Localization Game (The Describing Friend):
- The Setup: You are standing in a room with a friend. There is a red block somewhere. You have to tell your friend where the red block is relative to you ("It's to my left") or relative to another object ("It's behind the blue box").
- The Twist: Sometimes the "friend" is facing a different way than you.
- The Result: AIs often get confused about who is facing what. They might say a block is "in front" of you when it's actually "behind" you, because they forget to rotate their perspective to match your viewpoint.
The Structure Composition Game (The LEGO Architect):
- The Setup: You are given a long, boring list of coordinates for hundreds of blocks. You have to look at that list and say, "Oh, these blocks form a tall orange tower, and these form a flat blue floor."
- The Result: AIs are okay at spotting colors, but they often fail to recognize the shape. They might see a 3D cube and call it a "flat square," or miss that a group of blocks forms a "staircase" instead of a "wall."

2. The Big Surprises

Surprise #1: Eyes Don't Help (Much)
You might think that if you give an AI a camera (Vision-Language Models), it would get better at spatial tasks.

The Analogy: It's like giving a person a pair of glasses but asking them to solve a math problem. The glasses don't help with the math.
The Finding: The researchers found that AIs with "eyes" didn't perform significantly better than AIs with just "ears" (text-only). Seeing a picture of a grid didn't teach them how to reason about the grid. They still struggled with the logic of turning and moving.

Surprise #2: The "Big Brain" isn't Always the Smartest
The most powerful, expensive AI models (the "Frontier" models like GPT-4o or Gemini) did well, but they still made silly mistakes, especially in the "Body" (Egocentric) navigation game. They would get the first step right, but then "forget" they had turned around and get lost by step three.

Surprise #3: Small, Trained Models Can Beat Giants
This is the most exciting part. The researchers took a small, cheap AI model and "tutored" it specifically on these grid games.

The Analogy: Imagine a small, specialized mechanic who has practiced fixing only car engines for a year. They can often fix a specific engine better than a general genius who knows everything about everything but hasn't practiced that specific engine.
The Finding: This small, specialized AI actually outperformed the massive, general-purpose giants on these specific spatial tasks. It proved that you don't need a super-computer to understand space; you just need the right training.

3. Why Does This Matter?

Right now, we are building robots and virtual assistants that need to move around in the real world.

If a robot thinks "Forward" means "North" (fixed) instead of "Forward" (relative to its body), it will crash into a wall.
If a robot can't tell the difference between a "tower" and a "wall" just by looking at a list of parts, it can't build things.

This paper tells us that while AI is getting smarter, it still lacks a true "mental map" of 3D space. It's good at memorizing facts, but bad at imagining movement. However, the good news is that we don't need to wait for the next super-AI to fix this; we can just train smaller, cheaper models to be experts in spatial reasoning.

In a nutshell: AI is currently like a person who has read a thousand travel guides but has never actually left their house. They know the names of cities, but if you ask them to navigate a room while spinning around, they get dizzy. This study shows us how to teach them to stop spinning and start walking.

1. Problem Statement

Despite the rapid advancement of Large Language Models (LLMs) and Vision-Language Models (VLMs) in multimodal tasks, their ability to perform pure spatial reasoning remains poorly understood. Current evaluations often conflate spatial reasoning with visual perception, making it difficult to determine if models possess robust internal representations of 3D space or if they are merely relying on visual cues.

The authors identify three critical gaps:

Lack of Isolation: Most benchmarks use visual inputs, preventing the isolation of reasoning capabilities from perception.
Reference Frame Complexity: Models struggle with egocentric reasoning (updating spatial frames relative to a moving agent) compared to allocentric or cardinal (fixed grid) reasoning.
3D Structure Composition: Models often fail to translate lists of coordinates into coherent 3D geometric shapes (e.g., cubes, columns, planes).

The paper proposes that text-only grid datasets serve as a controlled proxy to evaluate the "mental models" of space that LLMs possess, independent of visual noise.

2. Methodology: The GSU Dataset

The authors introduce GSU (Grid Spatial Understanding), a synthetic, text-only dataset designed to evaluate spatial reasoning across three core tasks. The dataset is generated programmatically using coordinate lists and natural language instructions.

Core Tasks

Navigation:
- Follower: Given natural language instructions (e.g., "move right 2, forward 1"), predict the final coordinate.
- Instructor: Given a sequence of coordinates, generate the natural language instructions to traverse them.
- Settings:
  - Cardinal: Fixed axes (North/South/East/West).
  - Egocentric: Rotating axes relative to the agent's current heading (Left/Right/Forward/Backward).
  - Card2Ego: Translating cardinal directions to egocentric directions.
Object Localization:
- Determine the spatial relationship between a target and a reference point.
- Egocentric: Target relative to the viewer's current heading.
- Allocentric: Target relative to a separate reference object (independent of the viewer's heading).
Structure Composition:
- Given a list of colored blocks (coordinates), describe the resulting 3D structure.
- Types:
  - Simple: Single color, single shape.
  - Cohesive: Multiple colors forming one complex shape.
  - Composite: Multiple distinct shapes adjacent to each other.

Evaluation Metrics

Navigation: Exact match accuracy for coordinates and instruction chains; Euclidean distance for partial credit.
Object Localization & Structure Composition: Term overlap metrics ( $O_s$ ) for spatial relations, color terms ( $O_c$ ), numerical values ( $O_n$ ), and shape descriptions ( $O_f$ ). The shape metric includes partial credit for semantically similar shapes (e.g., "rectangular prism" vs. "tower").

3. Experimental Setup

The authors evaluated a wide range of models:

Small Text-Only LLMs: Mistral-7b, Llama-8b, Qwen-7b.
Small VLMs (Vision-Language Models): Pixtral-12b, LlamaVL-11b, QwenVL-7b.
Large Frontier Models: GPT-4o, Gemini-3-Pro, Qwen-72b, GPT-OSS-20b.
Fine-Tuned Models: FlanT5-large (<1B params) and Llama-8b with LoRA.

Training Strategy:

Prompting: Used 1-shot prompting with synthetic reasoning traces for Navigation/Object Localization, and 3-shot for Structure Composition.
Fine-Tuning: Fully fine-tuned FlanT5 and applied LoRA to Llama-8b using synthetic reasoning traces to teach the models how to update spatial frames.

4. Key Results

A. Struggles with Egocentric Frames

Cardinal vs. Egocentric: Models perform significantly better on fixed (Cardinal) grids. In Egocentric settings, even frontier models (GPT-4o, Qwen-72b) struggle to maintain consistent heading updates.
The "Flip" Error: In Egocentric navigation, models often calculate the correct new heading but fail to apply it, defaulting to the fixed Cardinal frame. In Object Localization, they frequently flip "front" and "back" when the reference frame changes.
Reasoning Traces: Analysis shows models often generate correct intermediate logic but fail to propagate the updated state to the final step.

B. Visual Modality Does Not Guarantee Spatial Understanding

VLM Performance: Exposure to visual inputs (VLMs) did not consistently improve performance over text-only counterparts of similar size.
Inconsistency: In some tasks, VLMs performed worse than their text-only versions (e.g., Qwen-7b vs. QwenVL-7b). This suggests that visual pre-training does not inherently transfer to robust textual coordinate reasoning.

C. Frontier Capabilities and Limits

Gemini-3-Pro: The only model to achieve near-perfect scores (1.0) on most base tasks, including complex 3D navigation and structure composition.
GPT-4o: Strong on Cardinal tasks but degrades significantly in long Egocentric sequences, often losing track of the current heading.
Small Models: Untrained small models (e.g., Llama-8b) performed poorly (<50% accuracy) on complex tasks.

D. The Power of Fine-Tuning

LoRA & Full Fine-Tuning: A small model (Llama-8b) fine-tuned with synthetic reasoning traces outperformed GPT-4o on Navigation tasks and matched Gemini-3-Pro on Object Localization.
FlanT5: A fully fine-tuned model with <1B parameters outperformed many 30B+ frontier models on Structure Composition and Navigation Instructor tasks.
Implication: Specialized, lightweight models can achieve frontier-level spatial reasoning if trained specifically on the logic of spatial frames, challenging the assumption that massive scale is required for these capabilities.

5. Key Contributions

GSU Dataset: A novel, text-only benchmark isolating spatial reasoning from visual perception, covering navigation, localization, and structure composition in 2D/3D grids.
Diagnostic of Reference Frames: Empirical evidence that LLMs lack robust internal representations of rotating (egocentric) spatial frames, often defaulting to fixed (cardinal) logic even when instructed otherwise.
Visual Modality Myth: Demonstrates that VLMs do not automatically possess superior spatial reasoning capabilities compared to text-only models of similar size, suggesting visual data is not a silver bullet for spatial understanding.
Efficiency of Specialization: Proves that small, fully fine-tuned models can match or exceed massive frontier models on specific spatial tasks, offering a viable path for specialized embodied agents where latency and hardware constraints are critical.

6. Significance

This work fundamentally shifts the perspective on spatial reasoning in AI. It suggests that the bottleneck for embodied agents is not necessarily the size of the model or the presence of vision, but the internal consistency of spatial state tracking.

For Embodied AI: The findings support the development of lightweight, specialized agents that can be fully fine-tuned for spatial tasks, reducing latency and hardware requirements compared to running massive frontier models.
For Model Architecture: It highlights a specific failure mode in current LLMs: the inability to maintain state updates for rotating reference frames, a critical requirement for real-world robotics and navigation.
Future Directions: The paper calls for research into how to better integrate parametric knowledge with visual information and how to train models to handle partial observability and continuous space, moving beyond the discrete grid abstraction.