The macaque IT cortex but not current artificial vision networks encode object position in perceptually aligned coordinates

By leveraging the motion aftereffect to dissociate perceived position from retinal input, this study demonstrates that the macaque inferior temporal cortex encodes object location in perceptually aligned coordinates—a history-dependent capability absent in current artificial vision networks but replicable through empirically derived transformations.

The Gist

Here is an explanation of the paper using simple language and creative analogies.

The Big Question: Where is the "Where"?

Imagine you are looking at a picture of a red apple sitting on a table. Your brain has two main jobs:

1. The "What" Job: Identifying that it is an apple.
2. The "Where" Job: Knowing exactly where that apple is sitting on the table.

For decades, scientists thought these jobs were done by two completely separate teams in the brain. The "What" team lived in the bottom part of the visual system (the Ventral Stream), and the "Where" team lived in the top part (the Dorsal Stream).

But this new study asks a tricky question: Does the "What" team actually know where things are, or do they just copy the coordinates from the camera lens (the retina)?

To find out, the researchers used a clever trick called the Motion Aftereffect.

---

The Trick: The "Moving Walkway" Illusion

Imagine you are standing on a moving walkway at an airport (like the ones in big terminals). You stand still for 30 seconds while the walkway moves you forward. When you step off, your legs feel like they are still moving forward, even though you are standing on solid ground. If you try to walk straight, you might accidentally drift backward.

This is the Motion Aftereffect (MAE).

In this study, the researchers did this to the brain:

1. The Adaptation: They showed monkeys and humans a screen full of lines moving to the right for 30 seconds.
2. The Test: They then showed a picture of a stationary object (like a bear) in the exact same spot on the screen.
3. The Result: Because the brain was "tired" from seeing rightward motion, it tricked the observers. The stationary bear felt like it had shifted to the left.

Crucially: The bear didn't actually move. The pixels on the screen were identical. Only the perception changed.

---

The Discovery: The Brain's "GPS" Gets Rewired

The researchers recorded the brain activity of monkeys in the IT Cortex (the "What" team). They asked: When the bear looks like it moved left to the monkey, does the IT cortex also think the bear moved left?

The Answer: Yes!

• Before the trick: The IT cortex said, "The bear is at pixel coordinate X."
• After the trick (Rightward motion): The IT cortex said, "The bear is at pixel coordinate X minus a little bit."

The brain's internal map of "where" the object is shifted in the exact same direction that the human felt the object shift. This proves that the "What" area of the brain isn't just a passive camera; it actively constructs a perceptual map that aligns with our experience, not just the raw data from the eyes.

The Villain: Artificial Intelligence (AI)

Here is where it gets interesting for the future of technology. The researchers tested modern Artificial Neural Networks (ANNs)—the brains behind AI image recognition (like the ones in your phone or self-driving cars).

They asked the AI: "We showed you a moving background, then a stationary bear. Where is the bear?"

The AI Answer: "I don't care. The pixels didn't move, so the bear is still at coordinate X."

No matter how they tweaked the AI (making it deeper, adding memory, or making it watch videos), the AI failed to experience the illusion. It remained stubbornly literal. It saw the pixels, but it didn't "feel" the shift in position.

The "Neuralization" Fix: Giving AI a Brain Upgrade

The researchers then tried a cool experiment. They took the mathematical "rules" of how the monkey's brain shifted its map during the illusion and forced the AI to follow those same rules.

The Result: When they "neuralized" the AI (gave it a monkey-brain filter), the AI suddenly started making the same mistakes as the humans and monkeys! It started saying the bear had moved left.

What this means: The AI has the data to know where things are, but it lacks the dynamic machinery to let past experiences (like seeing motion) change its current perception.

---

The Takeaway: Why This Matters

1. The Brain is a Storyteller, not a Camera: The part of your brain that recognizes objects (IT) is also deeply involved in figuring out where they are in a way that matches your experience, not just the raw light hitting your eye.
2. AI is Missing a Key Ingredient: Current AI is great at recognizing things, but it is "blind" to the subtle ways our past experiences warp our current reality. It lacks the "history-dependent" wiring that makes biological vision so flexible.
3. The Future of AI: To build truly human-like vision, we can't just make AI bigger. We need to teach it to let its "memory" of motion reshape how it sees the present, just like our brains do.

In short: The monkey's brain is a flexible, experience-based mapmaker. The AI is a rigid, pixel-counting robot. Until we teach the robot to get "tired" of motion and shift its perspective, it will never truly see the world the way we do.

Technical Summary

Here is a detailed technical summary of the paper "The macaque IT cortex but not current artificial vision networks encode object position in perceptually aligned coordinates."

1. Problem Statement

The study addresses a fundamental gap in understanding how the primate visual system encodes object location. While the "Two-Stream Hypothesis" traditionally assigns spatial processing ("where") to the dorsal stream and object identity ("what") to the ventral stream (specifically the Inferior Temporal or IT cortex), recent evidence suggests IT also encodes position.

However, a critical ambiguity remains: Are IT position signals perceptually meaningful, or are they merely a trivial byproduct of feedforward retinotopic pooling? In previous studies, object position was tightly coupled with pixel-based ground truth. Consequently, decoders could successfully predict position from IT activity, but it was unclear if this reflected a genuine perceptual code or simply inherited retinal coordinates. The authors sought to dissociate perceived position from pixel-based position to test if IT representations align with human perception.

2. Methodology

The authors employed a multi-modal approach combining macaque neurophysiology, human psychophysics, and in-silico modeling using the Motion Aftereffect (MAE) paradigm.

• The Motion Aftereffect (MAE) Paradigm:

  • Principle: Prolonged exposure to motion in one direction (e.g., rightward) causes a stationary object viewed afterward to appear shifted in the opposite direction (leftward), despite the pixel input remaining identical.
  • Hypothesis: If IT position codes are purely feedforward/retinotopic, they should not change after adaptation. If they are perceptually aligned, IT population codes should shift in the direction opposite to the adapter, mirroring human perception.

• Human Psychophysics:

  • Participants: 79 human subjects.
  • Task: Subjects viewed drifting gratings (30s adaptation + 3s top-up) followed by a brief (100ms) stationary object image. They reported the perceived centroid of the object via mouse click.
  • Stimuli: 40 naturalistic images of 8 object categories (bear, face, car, etc.) embedded in textured backgrounds at varying positions.

• Macaque Neurophysiology:

  • Subjects: Two adult male rhesus macaques.
  • Recording: Large-scale intracortical recordings from the IT cortex using Utah arrays (96 channels).
  • Protocol: Monkeys passively viewed the same stimuli as humans (with/without motion adaptation). Neural activity was recorded during a 70–170ms window post-stimulus onset.
  • Analysis: Linear decoders (L2-regularized Ridge regression) were trained on pre-adaptation neural data to predict object position ($x, y$). These decoders were then applied to post-adaptation neural data to see if the predicted position shifted.

• Artificial Neural Network (ANN) Modeling:

  • Models Tested:
    • Static Feedforward: VGG-16, ResNet-18, AlexNet, ViT-L32, SimCLR, Robust ResNet-50.
    • Dynamic/Recurrent: SlowFast (video action recognition), ConvRNN (recurrent object recognition).
  • Experiments:
    1. Baseline Decoding: Can position be decoded from ANN features?
    2. Neuralization: Applying empirically derived linear transformations (from IT adaptation data) to ANN features to see if biases can be induced artificially.
    3. Intrinsic Dynamics: Simulating adaptation within ANNs by adding exponential decay functions to unit activations (mimicking repetition suppression) or using temporal inputs in video models to see if biases emerge intrinsically.

3. Key Results

A. Human Perception

• Motion adaptation induced robust, systematic lateral biases in human position judgments.
• Rightward adaptation caused a leftward shift in perceived position ($\Delta P_x \approx -0.20^\circ$), and leftward adaptation caused a rightward shift ($\Delta P_x \approx 0.13^\circ$).
• Vertical ($y$) position estimates remained unchanged, confirming the effect is specific to the motion axis.
• Reliability of estimates remained high, indicating the shift was a bias, not increased noise.

B. Macaque IT Cortex

• Baseline: IT population activity robustly encoded object position (correlation $r \approx 0.6-0.7$), replicating prior findings.
• Adaptation Effects:
  • Motion adaptation altered IT population geometry (measured by Centered Kernel Alignment, CKA), reducing similarity between pre- and post-adaptation response spaces.
  • Crucial Finding: When decoders trained on pre-adaptation data were applied to post-adaptation IT responses, the predicted object positions shifted in the opposite direction of the adapter motion.
  • The magnitude and direction of these neural shifts qualitatively mirrored human perceptual biases.
  • This confirms IT encodes position in perceptually aligned coordinates, not just retinotopic ones.

C. Artificial Neural Networks (ANNs)

• Baseline: Standard feedforward ANNs (VGG, ResNet) could decode object position with accuracy comparable to or exceeding IT ($r \approx 0.8$).
• Failure to Adapt:
  • When presented with identical pixel inputs after simulated motion adaptation, standard ANNs showed zero shift in predicted position.
  • Intrinsic Dynamics: Adding exponential decay (repetition suppression) to ANN units reduced firing rates but failed to generate direction-opponent position biases.
  • Temporal Models: State-of-the-art video models (SlowFast, ConvRNN) with temporal convolutions and recurrence also failed to reproduce the perceptual shift.
• Neuralization Success: When the authors applied empirically derived linear transformations (mapping baseline IT activity to adapted IT activity) onto ANN feature spaces, the ANNs did exhibit the correct position biases. This proves the feature space can represent the shift, but current architectures lack the mechanism to generate it intrinsically.

4. Key Contributions

1. Perceptual Alignment of IT: The study provides definitive evidence that the ventral stream (IT cortex) encodes object position in coordinates that align with human perception, challenging the strict dichotomy that spatial coding is exclusive to the dorsal stream.
2. Mechanism of Adaptation: It demonstrates that motion adaptation reshapes the representational geometry of the IT population, not just firing rates, to produce perceptual biases.
3. Benchmark for AI: It establishes a new benchmark for artificial vision: current state-of-the-art models (even recurrent and video-based ones) fail to capture the dynamic, history-dependent reweighting of spatial representations observed in biological vision.
4. Neuralization Framework: The authors introduce a "neuralization" technique, showing that biological adaptation rules can be transplanted into ANNs to induce perceptual phenomena, highlighting that the missing ingredient in AI is the mechanism of adaptation, not the representational capacity.

5. Significance and Implications

• Neuroscience: The findings suggest that the "what" and "where" streams are more integrated than previously thought. IT is not just a static object recognizer but a dynamic hub where motion history (likely inherited from MT/V4) actively reshapes spatial representations to maintain perceptual stability or generate illusions.
• Artificial Intelligence: Current ANNs are "static" in their spatial coding; they treat identical pixel inputs as identical regardless of sensory history. To achieve human-like robustness and perceptual alignment, future models must incorporate:
  • Coupled form and motion pathways.
  • History-dependent gain control mechanisms (beyond simple suppression).
  • Training objectives that prioritize perceptual consistency over mere categorical accuracy.
• Theoretical: The study bridges the gap between neural dynamics and behavioral illusions, suggesting that perceptual biases are not "errors" but signatures of a system dynamically reweighting representations to reconcile changing sensory inputs with stable percepts.

In summary, the paper reveals that while biological vision systems dynamically re-map object location based on motion history to align with perception, current artificial vision systems remain rigidly tied to pixel coordinates, exposing a fundamental gap in how biological and artificial intelligence process spatial information.