🧬 biology

A Deep Learning Model of Mental Rotation Informed by Interactive VR Experiments

This paper proposes a three-component deep learning model that integrates equivariant neural encoding, neuro-symbolic object description, and recurrent decision-making to accurately simulate human mental rotation performance and response times, as validated by both existing literature and new interactive VR experiments.

Original authors: Raymond Khazoum, Daniela Fernandes, Aleksandr Krylov, Qin Li, Stephane Deny

Published 2026-05-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Raymond Khazoum, Daniela Fernandes, Aleksandr Krylov, Qin Li, Stephane Deny

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

Imagine you are holding two different 3D puzzles in your hands. One is sitting on a table in front of you, and the other is floating in your mind's eye, but it's turned sideways. Your brain has to figure out: "If I spin the floating one, will it look exactly like the one on the table, or is it a mirror image?"

This mental gymnastics is called Mental Rotation. For decades, scientists have wondered how our brains actually do this. Is it like a slow, continuous video spinning in our heads? Or is it more like taking giant, discrete jumps?

In this paper, a team of researchers built a computer brain (a Deep Learning Model) to solve this puzzle. They didn't just want the computer to get the right answer; they wanted to build a machine that thinks exactly like a human does. To do this, they didn't just look at old data; they put real people in Virtual Reality (VR) headsets to watch how they actually move their hands to solve the problem.

Here is how their "Human-Like Computer Brain" works, broken down into three simple steps:

1. The "3D Scanner" (The Equivariant Encoder)

The Metaphor: Imagine looking at a flat photo of a cube. A normal computer sees a flat square. But your brain instantly knows, "That's a cube, and I can see its top and side."
What the Model Does: The first part of their model is a special camera that looks at a 2D picture and instantly builds a 3D "ghost" of the object in its memory. It's trained so that if you tell the ghost to spin, the ghost spins perfectly in 3D space, just like a real object. This is the Spatial Representation.

2. The "Translator" (The Neuro-Symbolic Encoder)

The Metaphor: Now that the computer has a 3D ghost, it needs to describe it to a friend who can't see the ghost. Instead of saying "It's a cube rotated 45 degrees," it translates the shape into a simple sentence of directions, like a treasure map: "Go Up, Go Right, Go Back, Go Down..."
What the Model Does: This part takes the 3D ghost and turns it into a Symbolic Description. Crucially, the researchers discovered that humans don't need perfect precision. They just need to know which "Quadrant" (or general zone) the object is in.

The "Quadrant Hypothesis": Imagine the world is divided into four big pizza slices. The model doesn't care if the object is at 10 degrees or 20 degrees; it just cares that it's in the "Top-Right Slice." This simplifies the problem massively.

3. The "Decision Agent" (The Neural Agent)

The Metaphor: You have the treasure map (the symbolic description) of Object A and Object B. The agent is the little voice in your head that says, "Okay, Object A is in the Top-Right slice. Object B is in the Bottom-Left. I need to spin Object A two slices over to match."
What the Model Does: This final part compares the two "maps."

If the maps say they are in the same slice, it decides: "Match!"
If they are in different slices, it decides: "Spin 90 degrees clockwise!" or "Spin 180 degrees!"
It then applies that spin to the 3D ghost, gets a new map, and checks again. It keeps doing this until the maps match.

The Secret Sauce: The VR Experiment

To make sure their computer brain was thinking like a human, the researchers put 19 people in a VR room.

The Old Way: People just looked at pictures and pressed a button.
The New Way: People could grab a joystick and physically rotate the object in VR to help them decide.

What they found:
Humans are surprisingly lazy (in a good way!). They don't spin objects slowly and continuously. Instead, they make one or two big, fast "ballistic" jumps (like flicking a switch) to get the object into the right general "slice" or quadrant. Once it's in the right zone, they stop and decide if it's a match.

The researchers found that their computer model, which was built to make these same "big jumps" based on the "Quadrant" idea, behaved almost exactly like the humans in the VR experiment.

Why This Matters (According to the Paper)

The paper claims this is the first time anyone has built a computer model that:

Solves the mental rotation task correctly.
Mimics the specific way humans move (making a few big jumps rather than a slow spin).
Uses a mix of 3D spatial thinking (the ghost) and symbolic logic (the map) to do it.

The authors argue that our brains likely use a hybrid system: we build a 3D picture in our minds, but we make decisions using simple, abstract rules (like "it's in the right quadrant") rather than calculating every single degree of rotation.

In short: They built a robot brain that learns to rotate objects by first seeing them in 3D, turning them into a simple direction map, and then making a few big, smart jumps to solve the puzzle—just like a human does in a VR game.

Technical Summary: A Deep Learning Model of Mental Rotation Informed by Interactive VR Experiments

Problem Statement
Mental rotation—the ability to compare objects seen from different viewpoints—is a fundamental human cognitive process. While the seminal work of Shepard & Metzler (1971) established that human response times are linearly proportional to angular disparity, suggesting an internal "spatial engine," no existing computational model has successfully solved the mental rotation task while simultaneously offering a mechanistic account of how the brain might execute this process. Current deep learning systems often struggle with spatial reasoning and generalization to novel 3D orientations, failing to replicate the robustness of human performance in both planar and depth rotations.

Methodology
The authors propose a three-stage mechanistic model that integrates deep learning, equivariant representations, and neuro-symbolic reasoning. The model's design is guided by existing literature and a novel set of Virtual Reality (VR) experiments where human subjects manipulated objects via a joystick.

VR Experimental Design: The authors conducted experiments with 19 participants in a VR environment using Shepard-Metzler objects. Unlike traditional 2D setups, the "Action" setup allowed subjects to rotate objects using a thumbstick. Key behavioral findings included:
- Subjects performed a remarkably low number of "ballistic" actions (typically one) to align objects into a rough "quadrant" before making a decision.
- Subjects did not seek perfect alignment but rather placed objects within a $\pm 45^\circ$ range relative to each other.
- Response times showed high variability at large angular disparities, challenging the hypothesis of a strictly fixed, continuous rotation speed.
The Computational Model: Based on these findings, the authors formulated the Quadrant-Dependent Symbolic Representation Hypothesis, which posits that humans reduce the continuous rotation space into four discrete quadrants. The model consists of three stacked modules:
- Module I (Equivariant Neural Encoder): An equivariant neural renderer (EqNR) trained on 2D images of 3D objects. It produces a 3D spatial latent representation that is equivariant to $SO(3)$ rotations, allowing the model to manipulate internal analog representations without explicit 3D supervision.
- Module II (Neuro-Symbolic Object Encoder): A Vision Symbolic Model (VSM) comprising a Vision Transformer (ViT) encoder and an autoregressive Transformer decoder. It maps the 3D spatial latent representations into sequential symbolic descriptions (sequences of directional transitions between cubes). Crucially, this module enforces equivariance at the symbolic level, where the description changes predictably when the object crosses a quadrant boundary.
- Module III (Neural Decision Agent): A Multi-Layer Perceptron (MLP) that takes pairs of symbolic descriptions as input. It performs two functions: (a) judging similarity (match/mismatch) if the objects are in the same quadrant, or (b) predicting the necessary rotation action (e.g., rotate 90° clockwise) to align the objects if they are in different quadrants. This agent operates iteratively, applying predicted rotations to the spatial representation from Module I and re-evaluating until a similarity decision is reached.

Key Contributions

First Mechanistic Model: The paper presents the first neural model that not only solves the mental rotation task but also provides a mechanistic explanation of the cognitive process, bridging the gap between spatial manipulation and symbolic decision-making.
Integration of Representations: The work demonstrates the necessity of combining structurally isomorphic spatial representations (for manipulation) with abstract symbolic descriptions (for decision-making and action selection).
Validation via VR: The model is validated against both standard behavioral metrics (accuracy, response time trends) and novel interaction data (number of manual actions) derived from the authors' VR experiments.
Ablation Studies: Systematic ablations confirm that removing any single component (the equivariant encoder, the symbolic encoder, or the action-taking agent) leads to a failure in generalizing to novel objects or replicating human-like action sequences.

Results

Performance: The full model achieves 96.13% accuracy on the mental rotation task, comparable to human performance (91–95% in VR setups).
Behavioral Fidelity: The model replicates the human tendency to use a minimal number of actions (dominated by 0 or 1 action) to solve the task. It successfully captures the linear trend in response times for disparities from $0^\circ$ to $120^\circ$ , though it struggles to explain the specific increase in response time between $120^\circ$ and $180^\circ$ without assuming angle-dependent neural implementation costs.
Necessity of Components:
- Standard Siamese networks (purely feed-forward) fail to generalize to novel 3D rotations, confirming the need for equivariant spatial representations.
- Removing the symbolic encoder prevents the model from solving the task, highlighting the need for abstract descriptions to guide actions.
- Removing the action-taking agent allows the model to solve the task with high accuracy but fails to reproduce the sequential, action-based behavior observed in humans.

Significance and Claims
The authors claim this work offers new insights into the cognitive process of mental rotation, specifically supporting a hybrid view where rotations are applied to spatial representations, but decisions are guided by symbolic, quadrant-based abstractions. The paper argues that the human mind may utilize "discontinuous jumps" in rotation space rather than a strictly continuous analog process.

From a machine learning perspective, the work suggests that explicitly enforcing representational inductive biases (equivariance and neuro-symbolic integration) is more effective for spatial reasoning than relying solely on scale. The authors note that while fundamental questions remain (e.g., the biological plausibility of the rotation mechanism and data efficiency), their modeling effort serves as a proof-of-concept for integrating deep, equivariant, and symbolic representations to model human spatial reasoning. The work does not claim to solve all aspects of human cognition but aims to provide a viable neural implementation that aligns with behavioral data, adhering to the principle that "what I cannot create, I do not understand."

1. The "3D Scanner" (The Equivariant Encoder)

2. The "Translator" (The Neuro-Symbolic Encoder)

3. The "Decision Agent" (The Neural Agent)

The Secret Sauce: The VR Experiment

Why This Matters (According to the Paper)

More like this