🤖 AI

SeeSE3: Emergence of 3D Space in Vision Features

This paper demonstrates that self-supervised vision foundation models inherently encode 3D Euclidean space structures within their latent features, enabling novel "Latent-Space Navigation" techniques for visual odometry and localization without explicit 3D reconstruction.

Original authors: Caroline Chen, Sayna Ebrahimi, Fedor Kitashov, Ming-Hsuan Yang, Leonidas Guibas, Viorica Pătrăucean, Maks Ovsjanikov

Published 2026-07-17

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Caroline Chen, Sayna Ebrahimi, Fedor Kitashov, Ming-Hsuan Yang, Leonidas Guibas, Viorica Pătrăucean, Maks Ovsjanikov

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery about how we see the world. For a long time, scientists believed that to truly understand the 3D shape of space—how far away things are, how they rotate, and how we move through them—a being needed to be an active explorer. Think of a baby learning to crawl or a robot with wheels: they have to move their body to figure out that the world is a solid, three-dimensional place. This idea, famously proposed by the mathematician Henri Poincaré, suggested that a "motionless being" could never learn what space is because they would have no reason to tell the difference between moving their eyes and the world actually moving.

But today, we have something new: giant computer brains called "vision foundation models." These are AI systems trained on billions of photos and videos. They are like the ultimate "motionless beings." They don't have legs, they don't have wheels, and they don't move. They just sit there and look at pictures. The big question is: If you feed a motionless computer enough pictures, will it accidentally "discover" the rules of 3D space on its own? Or does it just see a flat, messy jumble of colors? This paper dives into that question, asking if these passive observers can secretly build a map of the world inside their digital brains, even without ever taking a step.

The Paper's Big Discovery: The "Unseen Map"

The researchers behind this study, a team from Google DeepMind and other institutions, decided to test this "Poincaré Task." They wanted to see if the internal "thoughts" (or features) of these motionless AI models were secretly organized like a 3D map. Specifically, they were looking for a connection to SE(3), which is just a fancy math way of describing all the possible ways you can move and turn in 3D space (like walking forward, turning left, or tilting your head).

Here is the twist: The AI models they tested were never taught about 3D space. They weren't shown depth charts, they weren't given GPS coordinates, and they weren't told how to measure distance. They were just trained to recognize patterns in images using a method called "self-supervision" (basically, guessing parts of an image to learn from the rest).

The Main Finding:
The paper suggests that, surprisingly, these motionless AIs did build a hidden 3D map inside their brains. Even though the raw data coming out of the AI looked like a tangled, chaotic mess (like a ball of yarn), the researchers found that with a tiny, simple "adapter" (a small add-on tool they called the Poincaré Adapter), they could "unroll" that mess. Once unrolled, the AI's internal features lined up perfectly with the geometry of 3D space.

Think of it like this: Imagine the AI's brain is a crumpled piece of paper with a map drawn on it. To a normal eye, the map looks broken and impossible to read. But the Poincaré Adapter is like a gentle hand that smooths the paper out flat. Suddenly, the crumpled lines reveal a perfect, straight grid where moving "forward" in the AI's mind actually means moving forward in the real world.

What They Ruled Out (And What They Didn't)

The authors were very careful to clarify what this doesn't mean.

It's not a direct 3D camera: They explicitly ruled out the idea that the AI's raw features are already a perfect 3D map. If you just look at the raw numbers coming out of the AI, they are chaotic and curved, not straight and flat. The "map" is hidden, not obvious.
It's not magic: The paper argues against the idea that you need a robot with wheels to understand space. They showed that a "motionless" observer (the AI) can discover these spatial rules just by looking at static pictures, provided you know how to ask the right questions.
It's not perfect everywhere: While the AI could figure out rotation (turning) very well, it struggled a bit more with figuring out exactly how far something was (translation magnitude). The paper suggests this is because distance is harder to guess from a flat picture without extra clues, but the general direction of movement was still very clear.

How They Proved It (The "Probe" Experiments)

To find this hidden map, the researchers used a few clever tricks, which they call "probes":

The Neighborhood Test: They checked if pictures that are close together in the real world (like two photos taken while walking down a hallway) were also close together in the AI's brain. They found that for some models, like DINOv2, the answer was a resounding "yes." The AI naturally grouped similar views together, just like a map does.
The "Straightening" Test: They tried to see if they could predict the next view just by doing simple math (adding or subtracting numbers) on the AI's current view. On raw data, this failed miserably. But once they used their Poincaré Adapter, the math worked beautifully. They could take the AI's "current thought," add a "move forward" vector, and the result was a "thought" that perfectly matched the next frame in the video.
The Navigation Game: As a final test, they tried to use this hidden map to navigate. They gave the AI a starting point and a destination (like "go 2 meters forward and turn 30 degrees"). Using only the AI's internal features and the adapter, they could predict what the view would look like at the destination without ever generating a new image. It was like the AI could "imagine" the new view just by doing vector math in its head.

The "Visual Grid Code"

The most exciting part of the paper is the idea of a "Visual Grid Code." In biology, we know that animals (including humans) have special brain cells called "grid cells" that fire in a hexagonal pattern to help us navigate. This paper suggests that even passive, motionless AI models might be developing something similar on their own. They aren't just memorizing pictures; they are organizing their knowledge into a geometric structure that mirrors the laws of physics.

The authors found that this ability scales up. The more data the AI saw, the better its hidden 3D map became. Even when they tested the AI on completely new rooms it had never seen before, the "adapter" could still help it navigate, suggesting that this is a fundamental, universal property of how these models learn.

Why This Matters

This isn't just about making better robots. It changes how we think about intelligence. If a motionless computer can figure out the rules of 3D space just by looking at pictures, it means that the geometry of our world is so deeply embedded in visual data that it's almost impossible to miss. The "laws of space" aren't something we have to teach a machine; if we give it enough eyes, it will discover them on its own.

The paper concludes that while the AI doesn't "see" 3D space in the way we do (it doesn't have a body to move), it has built a mathematical skeleton of 3D space inside its code. And with a little help from a simple tool like the Poincaré Adapter, we can finally read that map, allowing us to navigate the world using nothing but the AI's own thoughts.

Technical Summary: SeeSE3 – Emergence of 3D Space in Vision Features

Problem Statement
This paper investigates whether vision foundation models, trained without direct 3D supervision or active agency, inherently construct representations that reflect the intrinsic properties of 3D Euclidean space. Specifically, it addresses the "Poincaré Task": Can a motionless observer, given only passive visual input, "discover" the structure of 3D space? The authors hypothesize that while vision features are high-dimensional and tied to visual content, they may contain a latent subspace isomorphic to the Special Euclidean group $SE(3)$, which governs rigid body transformations (position and orientation). Unlike prior works that probe 3D awareness by regressing specific quantities (e.g., depth, normals) or using high-capacity decoders to extract pose, this work examines whether the structure of the feature space itself aligns with Euclidean geometry.

Methodology
The authors propose a protocol to evaluate the alignment between visual feature spaces and the group of Euclidean transformations $SE(3)$ using static scenes from diverse datasets (ScanNet, ARKitScenes, TUM RGB-D, etc.). The evaluation employs four metrics of increasing complexity:

Topological Alignment (M1): Uses a mutual $k$ -nearest neighbor ( $k$ -nn) metric to compare the neighborhood structure of visual features against camera pose trajectories. This tests if local feature neighborhoods mirror local spatial neighborhoods.
Intrinsic Dimensionality (M2): Estimates the local intrinsic dimension (ID) of the feature manifold. A representation capturing the 6 degrees of freedom of $SE(3)$ should exhibit an ID close to 6.
Linear Equivariance (M3): Tests if the feature space is globally Euclidean by training a linear regressor to predict relative pose changes ( $\Delta P$ ) directly from raw feature differences ( $z_{t+s} - z_t$ ).
The Poincaré Adapter (M4): Recognizing that raw features often form a curved manifold where linear arithmetic fails, the authors introduce a lightweight, trainable Siamese Multi-Layer Perceptron (MLP) adapter, $\phi_\theta$ . This adapter "unrolls" the non-linear feature space to construct a homogeneous coordinate system where pose changes become linear: $\Delta P \approx W(\phi_\theta(z_{t+s}) - \phi_\theta(z_t))$ .

Key Contributions

Formulation of the Poincaré Task: The paper defines a framework for inferring 3D structure from visual features by comparing them to $SE(3)$, introducing specific probes to evaluate encoder performance on this task.
Discovery of Latent Geometric Structure: The authors demonstrate that while no features are directly isomorphic to $SE(3)$, a lightweight "Poincaré adapter" can extract a homogeneous coordinate system from purely self-supervised models (e.g., DINOv2, DINOv3). This reveals that these models possess latent subspaces strongly correlated with 3D space despite lacking explicit 3D training.
Theoretical Analysis: The paper provides a theoretical analysis of the conditions for geometric recovery, showing that rotation is generally easier to encode than translation magnitude due to the dependency of translational flow on scene depth ( $1/Z$ ). It also establishes that self-distillation and large data scales are critical for the emergence of this structure.
Latent Space Navigation: Building on the linearized latent space, the authors propose a novel "Latent Space Navigation" technique. This approach performs visual odometry and localization purely in the latent space using vector arithmetic (inverting the Poincaré adapter), bypassing the need for explicit 3D reconstruction or pixel synthesis.

Results

Topological Alignment: Self-supervised models like DINOv2 and DINOv3 exhibit strong topological alignment with camera poses (mutual $k$ -nn scores $\approx 0.38$ ), outperforming raw pixels and many video-based models. This structure emerges without explicit depth or pose supervision.
Linearization: Raw features fail linear equivariance tests (negative $R^2$ scores), confirming the feature manifold is natively curved. However, the Poincaré adapter successfully linearizes this manifold, achieving high $R^2$ scores (e.g., $\approx 0.65$ for DINOv2 on ScanNet) in mapping feature differences to $SE(3)$ displacements.
Scaling and Generalization: Geometric awareness scales monotonically with the number of training rooms. Zero-shot adapters trained on a subset of rooms generalize effectively to unseen environments, with success rates increasing as training data grows.
Navigation Performance: The "Inverse Poincaré" method (training-free, using pseudo-inverse weights) achieves competitive retrieval performance in latent space navigation, significantly outperforming static baselines. While learned attention correctors improve regression metrics ( $R^2$ ), the simple linear arithmetic approach is sufficient for topological retrieval (finding the correct spatial neighborhood).
Component Difficulty: Analysis reveals that rotation is reliably decodable across configurations, whereas translation magnitude is more sensitive to hyperparameters and requires diverse training data to resolve scale ambiguities.

Significance and Claims
The paper claims that vision foundation models, even when trained as passive observers on static images, naturally organize their latent spaces to reflect the geometry of $SE(3)$. This challenges the prevailing view that discovering spatial structure requires active agency or explicit 3D supervision. The work suggests that the "Visual Grid Code" (a representation isomorphic to physical motion generators) emerges from the statistics of visual data alone.

The significance lies in demonstrating that 3D spatial reasoning can be accessed via lightweight probing of existing foundation models, rather than requiring massive, specialized 3D reconstruction architectures. This enables efficient "Latent Space Navigation," where an agent can predict the visual consequences of physical movement through simple vector operations in the latent space. The authors position this as a fundamental step toward understanding the geometric underpinnings of spatial reasoning in vision models, bridging the gap between passive observation and the discovery of physical laws.

Limitations
The study is limited to static scenes where feature changes are solely attributed to camera motion. Consequently, it does not address the distinction between "changes of position" and "changes of state" (e.g., dynamic objects) as originally posited by Poincaré. The authors note that extending this formalism to dynamic scenes remains an open direction for future work.