🤖 machine learning

SpaRRTa: A Synthetic Benchmark for Evaluating Spatial Intelligence in Visual Foundation Models

This paper introduces SpaRRTa, a synthetic benchmark designed to evaluate the spatial reasoning capabilities of Visual Foundation Models by testing their ability to identify relative object positions, revealing significant disparities in current models' spatial awareness and guiding future development.

Original authors: Turhan Can Kargin, Wojciech Jasiński, Adam Pardyl, Bartosz Zieliński, Marcin Przewięźlikowski

Published 2026-08-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Turhan Can Kargin, Wojciech Jasiński, Adam Pardyl, Bartosz Zieliński, Marcin Przewięźlikowski

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to navigate a messy room. You show it a picture of a cat sitting on a chair next to a lamp. A smart robot needs to know two things: what those things are (a cat, a chair, a lamp) and where they are relative to each other (the cat is to the left of the lamp). For a long time, computers have gotten really good at the first part—identifying objects in photos. But the second part, understanding the 3D layout and direction, has been a bit like a blindfolded person trying to guess where a ball is just by looking at a flat drawing.

This is the world of "Visual Foundation Models." Think of these as super-smart AI brains that have read millions of pictures and learned to recognize patterns. They are the engines behind many modern image tools. However, researchers have noticed a glitch: these brains are great at saying "That's a tree!" but often stumble when asked, "Is the tree to the left or right of the car?" This matters because if we want robots to walk through our houses or drive cars, they need to understand space, not just labels. The big question is: Do these AI brains actually "see" the 3D world, or are they just memorizing tricks to guess the right answer for specific math problems?

To solve this mystery, a team of researchers created a new test called SpaRRTa (Spatial Relation Recognition Task). Instead of asking the AI to do complex math like measuring exact distances or drawing 3D maps, SpaRRTa asks a simpler, more fundamental question: "From this specific viewpoint, is Object A to the left, right, front, or back of Object B?"

To make this test fair and super precise, the researchers didn't use real photos from the internet. Instead, they built a virtual world using a high-end game engine (Unreal Engine 5). They created photorealistic scenes—like a forest, a desert, a snowy town, a bridge, and a busy city—and placed objects like cars, trees, and dogs in them. They could move the camera or even pretend the "viewpoint" was a person standing in the scene, asking the AI to imagine the world from that person's eyes, not just the camera's. This is like asking, "If I were standing behind the car, would the tree be on my left or right?"

The researchers then tested a wide variety of these AI brains, from those that learned by just looking at pictures to those that were specifically taught 3D geometry. They used a clever trick called "probing." Imagine the AI's brain is a locked box of information. Instead of retraining the whole box, they just built a small, simple key (a probe) to see if the spatial information was already inside. They tried three different keys: a simple one that looked at the whole picture at once, a smarter one that could pick out important spots, and a super-smart one that could focus on multiple specific things at once.

Here is what they found, and it's a bit surprising. First, the simple key often failed. It turned out that the AI's "global" view of the picture (looking at the whole image as one big blob) actually hides the spatial details. The information about where things are is stored in the tiny, local patches of the image, like individual puzzle pieces. When you squish all those pieces together into one summary, the direction gets lost. However, when they used the smarter, more focused keys, the AI brains performed much better.

The study suggests that while these models do have spatial awareness, it's hidden deep in the details. Models that were specifically trained with 3D data (like knowing exactly how far away things are) did the best job, but even models that only looked at flat 2D pictures showed they could understand direction if you knew how to ask the right question. Interestingly, the AI struggled much more when asked to take the perspective of a person in the scene (the "allocentric" view) compared to just looking from the camera's view. This is like how it's easy for us to say "the tree is on the left of the car" from our photo, but much harder to say "the tree is on the right of the car" if we imagine ourselves standing behind the car.

The researchers also checked if being good at identifying flowers or airplanes meant you were good at understanding space. The answer was no. The AI could be a master at naming things but terrible at knowing where they are. This proves that SpaRRTa is measuring a totally different skill than just object recognition.

In short, the paper suggests that modern AI brains are not "blind" to space, but they are hiding their spatial knowledge in the fine details of the image rather than in their big-picture summary. To build robots that can truly navigate our world, we need to stop just asking them to recognize objects and start teaching them to understand the relationships between them, using tools like SpaRRTa to guide the way.

Technical Summary: SpaRRTa – A Synthetic Benchmark for Evaluating Spatial Intelligence in Visual Foundation Models

Problem Statement

Visual Foundation Models (VFMs) such as DINO and CLIP have demonstrated remarkable proficiency in semantic understanding, driving applications in classification, object recognition, and scene analysis. However, their deployment in embodied systems is hindered by limited spatial reasoning capabilities. While recent efforts have incorporated 3D tasks (e.g., depth estimation, surface normal prediction) into VFM training, performance across diverse spatial tasks remains inconsistent. This raises a fundamental question: do contemporary VFMs possess a general notion of spatial awareness, or do they merely overfit to specific, quantifiable geometric objectives without capturing the abstractions necessary for human-like spatial understanding? Existing benchmarks often conflate spatial reasoning with semantic recognition or rely on precise metric predictions that may not reflect the model's ability to encode relational spatial structures.

Methodology: The SpaRRTa Benchmark

To address these gaps, the authors introduce SpaRRTa (Spatial Relation Recognition Task), a benchmark designed to evaluate the ability of VFMs to identify static directional relations between visible objects from a specified viewpoint.

1. Synthetic Data Generation

SpaRRTa utilizes Unreal Engine 5 to generate a photorealistic, synthetically derived dataset. This approach allows for:

Full Control: Precise manipulation of object placement, scene layout, and viewpoint.
Diversity: Five distinct high-fidelity environments (Forest, Desert, Winter Town, Bridge, City) ranging from organic landscapes to dense urban settings.
Ground Truth: Automatic generation of perfect spatial annotations without the noise inherent in real-world data.
In-Distribution: The use of assets aligned with common ImageNet super-categories ensures the data remains in-distribution for standard VFMs, isolating geometric understanding from semantic recognition failures.

2. Task Definition

The core task is a four-way classification problem: determining the relative direction from a source object to a target object. The benchmark defines two variants based on the reference viewpoint:

Egocentric (SpaRRTa-ego): The camera is the viewpoint. The model predicts the direction relative to the camera's perspective.
Allocentric (SpaRRTa-allo): A third object (e.g., a human figure) serves as the viewpoint. The model must infer the spatial relation from the perspective of this third object, requiring "perspective-taking" and the decoupling of spatial relations from the direct visual input.

To ensure rigor, the pipeline employs procedural rejection sampling to eliminate geometric ambiguity (e.g., scenarios where the target lies on the boundary between "left" and "front"), ensuring all retained samples have indisputable ground-truth labels.

3. Evaluation Protocol

The authors evaluate a diverse suite of VFMs (including Self-Supervised Joint-Embedding Architectures, Masked Image Modeling, Supervised, and Vision-Language models) using frozen backbones. To probe spatial information, three distinct decoding mechanisms are employed:

Linear Probing (GAP): Global Average Pooling of patch tokens.
AbMILP: Attention-Based Multiple Instance Learning Pooling, which learns scalar weights for patches.
Efficient Probing: Utilizes multi-query cross-attention mechanisms to attend to specific spatial entities.

Key Contributions

The Tool: A photorealistic, synthetically generated evaluation environment built on Unreal Engine 5 specifically for probing directional spatial relation recognition.
The Benchmark: Two standardized challenge sets (Egocentric and Allocentric) designed to disentangle spatial logic from semantic recognition.
The Insights: A comprehensive characterization of how different VFM families encode spatial information, revealing that spatial knowledge is often present but structurally obscured in global representations.

Key Results

1. Patch-Level Encoding vs. Global Pooling

The study reveals a consistent performance hierarchy: Linear Probing < AbMILP < Efficient Probing.

Spatial information is primarily encoded in local patch features rather than global representations.
Global Average Pooling (GAP) significantly dilutes spatial signals, particularly in cluttered scenes.
Probing mechanisms that selectively aggregate patch information (AbMILP, Efficient Probing) yield substantial performance gains, indicating that spatial awareness exists in VFMs but is often inaccessible via standard linear probes.

2. Impact of 3D Supervision

VGGT (a model trained with explicit 3D supervision on camera parameters, depth, and tracks) outperforms its parent model, DINO-v2 (+reg), when using spatially selective probes (AbMILP and Efficient Probing).
However, under Global Average Pooling, VGGT does not outperform DINO-v2 and sometimes performs worse.
Conclusion: Explicit 3D supervision enriches patch-level geometric structure but does not necessarily improve global image-level representations for spatial relation recognition.

3. Egocentric vs. Allocentric Difficulty

Allocentric recognition is consistently more challenging than Egocentric recognition across all models and environments.
While top models approach saturation in Egocentric tasks, a significant performance gap persists in Allocentric tasks, highlighting the difficulty of perspective-taking in current VFMs.

4. Environmental Complexity

Models perform better in visually homogeneous environments (Desert, Forest) and degrade in semantically cluttered scenes (City, Winter Town).
This suggests that visual noise interferes with the identification of relevant patches for spatial reasoning, a limitation common across current visual representations.

5. Orthogonality to Semantic Tasks

Correlation analysis shows a positive correlation ( $R^2 \approx 0.40$ ) between SpaRRTa performance and 3D geometric tasks (depth estimation, camera pose).
There is negligible correlation ( $R^2 \approx 0.00$ ) between SpaRRTa and high-level semantic classification tasks (e.g., fine-grained flower or aircraft recognition).
This confirms that SpaRRTa measures a distinct capability (spatial awareness) that does not emerge automatically from strong semantic features.

6. Mechanism of Representation

Attention flow analysis reveals that 3D supervision (in VGGT) reorganizes information flow in deep layers. Unlike DINO-v2, which maintains high attention from the [CLS] token to objects, VGGT shifts attention toward register tokens and distributes spatial details across local patch tokens. This explains why VGGT fails with global [CLS] probing but excels with efficient, patch-aware probing.

Significance and Claims

The authors claim that SpaRRTa serves as a diagnostic tool for studying spatial representations and a guide for developing future vision models with stronger spatial awareness.

Distinct Capability: The paper argues that 3D awareness is a fundamental, independent capability axis that existing semantic benchmarks fail to capture.
Evaluation Necessity: Standard linear evaluation substantially underestimates spatial awareness; structured aggregation is required to uncover spatial knowledge.
Embodied AI: For embodied agents to navigate and interact effectively, models must move beyond passive semantic recognition to support perspective-dependent reasoning and spatial relation decoding.
Future Direction: The authors suggest that SpaRRTa can guide the development of "spatially aware, action-conditioned world models," potentially simplifying training regimes for downstream embodied tasks.

The paper concludes modestly, noting that while SpaRRTa addresses a fundamental building block of spatial reasoning, it does not yet evaluate other aspects such as metric distance estimation, occlusion reasoning, or temporal understanding.