📄 other

Semantic Visual Representations Emerge from Embodied Self-Supervised Learning

This paper introduces Embodied Self-Supervised Learning (E-SSL), a model that leverages temporal consistency and sensorimotor contingencies from natural human interaction to generate high-level semantic visual representations that outperform existing models, align more closely with pre-verbal children's visual cortex, and offer new insights into the origins of human visual development and stream specialization.

Original authors: Arthur Aubret

Published 2026-07-27

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Arthur Aubret

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Brain's Secret GPS: How Moving Helps Us See

Imagine trying to learn what a "dog" looks like by staring at a single, frozen photograph. You might memorize the colors and the fur texture, but you'd miss the way its ears flop when it runs or how its shape changes as it turns its head. Now, imagine learning by actually walking around the dog, watching it from every angle, and feeling how your own body moves to keep it in view. This is the difference between a static computer program and a human baby. For decades, scientists have been puzzled by a big question: How does the human brain, with almost no teacher and no textbooks, learn to understand the world so quickly? We know that babies don't just sit still; they crawl, reach, and roll, constantly moving their eyes and bodies. This paper explores the idea that this movement isn't just a side effect of being alive—it's the actual key that unlocks our ability to see and understand objects.

The researchers are testing two main ideas that have been floating around in science for a while. The first is Temporal Consistency, which is a fancy way of saying "things that happen close together in time are probably related." If you see a red ball, then a second later you see a red ball in a slightly different spot, your brain assumes it's the same ball. The second idea is Sensorimotor Contingencies, which is the rule that "my actions change what I see." If I turn my head left, the world shifts right. If I pick up a cup, the handle appears. The paper asks: If we teach a computer to learn only by watching videos of people moving and looking around, using just these two simple rules, will it learn to see like a human? And more importantly, will it learn to see like a baby who hasn't even learned to talk yet?

The Robot That Learns by "Walking"

Meet E-SSL (Embodied Self-Supervised Learning). Think of it as a digital robot brain that doesn't have a teacher, a textbook, or even a language. Instead of being fed millions of labeled pictures like "this is a cat" or "this is a car," this robot is given 300 hours of raw, first-person video footage recorded from glasses worn by real humans. These videos show exactly what the humans saw, where their eyes were looking, and how their heads and bodies moved.

The robot's job is to figure out the rules of the world on its own. It uses two tricks. First, it looks at two pictures taken a split second apart. If the image hasn't changed much, it assumes the object is the same (Temporal Consistency). Second, it looks at how the human moved their head or eyes between those two pictures. It tries to learn that "when I move my head this way, the world shifts that way" (Sensorimotor Contingencies). It's like a kid learning that if they spin around, the room spins with them, but if they just blink, the room stays still.

What Did They Find?

The results were surprisingly powerful. Even though the robot never heard a single word of language and never saw a label telling it what an object was, it started to build a mental map of the world that looked a lot like a human's.

1. It Learned to Recognize Objects (Almost as Well as Adults)
When the researchers tested the robot on standard picture quizzes (like identifying objects in the ImageNet-1k dataset), it got about 71% of the answers right. That's a huge jump compared to other "bio-inspired" computer models that try to copy the brain but don't use movement. While it's still about 20% behind a fully grown adult human (who has years of language and experience), it's way ahead of the pack of other computer models. It proved that you don't need a teacher to learn what a "mug" or a "dog" is; you just need to move around and watch how things change.

2. It Thinks Like a Baby
Here is where it gets really cool. The researchers compared the robot's "brain" to the brains of real babies. They found that the robot's way of seeing things matched the brain activity of 9-month-old infants much better than it matched the brains of adults.

The "Shape vs. Texture" Test: Babies are famous for learning to recognize objects by their shape rather than their color or texture. A red ball and a blue ball are both "balls" to a baby. The robot, after learning with movement, started to do the same thing. It learned to focus on the shape of an object, just like a toddler.
The "Handle" Discovery: The robot learned that if it sees a handle, it's probably a mug, even if it hasn't seen the whole mug yet. This is a skill that emerges in human children around 18 months old, and the robot developed it just by watching people move.

3. Movement is the Secret Sauce
The paper ran a fascinating experiment to see why movement matters. They simulated a "puppet" version of the robot. In this version, the robot saw the same videos, but the head movements were scrambled or fake. The robot didn't learn as well.

The "Late Walker" Effect: They also simulated a robot that started learning to move its head only after it had already been training for a while. The result? The robot's vision was worse than the one that started moving immediately. This suggests that the moment a human baby starts crawling or walking (locomotion) is a critical "aha!" moment for their vision. It's not just that they see more; it's that their brain finally gets to connect their own muscle commands with what they see, supercharging their learning.

4. The Brain's Two Highways
Finally, the paper discovered something about how the robot's "brain" organized itself. Human brains have two main visual highways:

The Ventral Stream (the "What" pathway): Helps us recognize what an object is (e.g., "That's a cat").
The Dorsal Stream (the "Where/How" pathway): Helps us understand where an object is and how to interact with it (e.g., "I need to grab that cup").

The researchers found that the robot naturally split its learning into these two types of pathways, just like a human brain does. The part of the robot that learned about "time" (Temporal Consistency) got really good at recognizing objects (Ventral). The part that learned about "movement" (Sensorimotor Contingencies) got really good at understanding spatial changes and actions (Dorsal). This suggests that these two distinct pathways in our brains might not be hard-wired from birth, but might actually emerge naturally from the simple act of moving and watching the world change.

The Bottom Line

This paper suggests that the secret to human vision isn't just having a high-quality camera (our eyes) or a super-computer (our brain). It's the fact that we are embodied. We move, we touch, and we change our perspective. By simply moving through the world and watching how our actions change what we see, our brains naturally learn to recognize shapes, understand objects, and split vision into "what" and "where."

While the robot isn't quite as smart as a 3-year-old toddler (who can generalize shapes even better), it proves that you don't need language or massive datasets to start seeing the world. You just need to get moving. It suggests that the moment a baby starts to crawl, their brain gets a massive upgrade, turning a blur of colors into a world of recognizable, graspable objects.

Technical Summary: Semantic Visual Representations Emerge from Embodied Self-Supervised Learning

Problem Statement

The principles driving the development of human visual perception, particularly the acquisition of semantic object recognition with minimal supervision, remain poorly understood. While high-performing artificial vision systems exist, they rely on training data and mechanisms (e.g., aggressive cropping, masking) that differ drastically from human learning. Conversely, bio-inspired models of human vision often fail to capture rich semantic recognition abilities. A critical gap exists in understanding how humans bridge the divide between human-level performance and human-like learning principles. Specifically, it is unclear whether the statistical structure of concurrent visual and motor signals during embodied interaction is sufficient to drive the emergence of human-like visual representations without explicit semantic labels or language input.

Methodology

The authors introduce Embodied Self-Supervised Learning (E-SSL), a computational framework designed to model human visual learning using two core unsupervised principles: Temporal Consistency (TC) and Sensorimotor Contingencies (SMC).

Data and Input

The framework is trained on Nymeria, a large-scale dataset comprising 300 hours of naturalistic, first-person (egocentric) video recorded with head-mounted cameras (ARIA glasses). The dataset includes:

Visual Input: 224 × 224 crops centered on the participant's gaze location.
Motor Signals: Eye movements (gaze translation) and head movements (rotation and translation derived from IMU and eye-tracking).
Preprocessing: Pairs of temporally adjacent images ( $x_t, x_{t+\delta t}$ ) are sampled. To simulate diverse lighting conditions consistent over time, identical color transformations are applied to both images in a pair.

Learning Objectives

The model learns via two complementary loss functions implemented using standard SSL architectures (MoCoV3, SimCLR, VICReg):

Temporal Consistency (TC): Encourages similar embeddings ( $z_t, z_{t+\delta t}$ ) for temporally adjacent visual inputs, assuming that objects and scenes remain stable over short time intervals.
Sensorimotor Contingencies (SMC): Aligns a parameterized movement vector ( $m'_{t, t+\delta t}$ ) with an embedding of the visual transition ( $m_{t, t+\delta t}$ ) derived from concatenating visual representations before and after the movement. This forces the model to learn the relationship between motor actions and their visual consequences.

Evaluation Protocol

The learned representations are evaluated via linear probing on a diverse set of downstream tasks without additional semantic labels during pre-training:

Object/Scene Recognition: ImageNet-1k, Cifar10, Places365.
Semantic/Instance Segmentation: PascalVOC, DAVIS.
Depth Estimation: NYUv2.
Pose Estimation: MVImgNet.
Developmental Benchmarks: Few-shot recognition of attributes (color, texture, size) and shape caricatures to compare against pre-verbal children.
Neurobiological Alignment: Representational Similarity Analysis (RSA) against fMRI data from the ventral visual cortex of infants (2-month, 9-month) and adults.

Key Contributions and Results

1. Emergence of Semantic Representations

E-SSL, trained solely on human experience without language or explicit supervision, achieves 71.04% Top-5 accuracy on ImageNet-1k. This substantially outperforms established bio-inspired models like Prednet (by up to 58.73%) and other state-of-the-art unsupervised models (LPL, SH, SCFF), despite the latter being evaluated on in-distribution data while E-SSL was pre-trained on out-of-distribution egocentric data. The model narrows the performance gap to human adults by approximately 10–20 percentage points.

2. Alignment with Pre-Verbal Development

The representations learned by E-SSL exhibit properties similar to those of pre-verbal children:

Attribute Prototypes: The model successfully forms reliable prototypes for color, texture, and size.
Shape Bias: While TC alone favors texture encoding, the addition of SMC boosts shape encoding, mirroring the developmental shift in toddlers (around 18 months) who begin to prioritize shape over texture.
Neural Alignment: E-SSL representations show significantly higher correlation with fMRI data from the ventral visual cortex of 9-month-old infants ( $\rho = 0.289$ ) compared to random baselines or Prednet, suggesting it captures the visual geometry of the infant brain.

3. The Role of Self-Induced Locomotion

Experiments simulating different embodied conditions reveal that visuomotor alignment is critical:

Embodied vs. Puppet: Agents with aligned motor and visual signals ("Embodied") consistently outperform those with uncorrelated signals ("Puppet").
Head Movement: Removing head movement signals significantly hinders learning, whereas removing eye movement signals has a lesser impact.
Timing: Early access to aligned visuomotor signals (simulating early locomotion) yields better performance than delayed alignment, predicting that the onset of self-locomotion substantially enhances visual perception.

4. Functional Specialization (Ventral vs. Dorsal Streams)

The study demonstrates that TC and SMC naturally induce a functional organization resembling the human visual cortex's dual-stream architecture:

Ventral Stream (Semantic): Tasks like object and scene recognition perform best with intermediate SMC weighting and intermediate temporal scales ( $\Delta T \approx 2s$ ).
Dorsal Stream (Action/Geometry): Tasks requiring spatial reasoning and pose estimation benefit from stronger SMC weighting and shorter temporal scales ( $\Delta T \approx 0.2s$ ).
Gradient Divergence: Gradient analysis shows that neurons and weights involved in optimizing TC and SMC progressively diverge in deeper network layers, mirroring the anatomical separation of ventral and dorsal pathways.

Significance and Claims

The paper posits that Temporal Consistency and Sensorimotor Contingencies constitute fundamental principles underlying human visual development. The primary significance of this work lies in:

Bridging the Gap: Demonstrating that bio-inspired learning principles, when applied to large-scale human experience, can produce semantic visual representations that rival current bio-inspired models and approach human performance.
Developmental Insights: Providing a computational framework to rigorously test hypotheses about how specific experiences (e.g., the onset of locomotion) shape visual development, predicting that self-induced movement facilitates both spatial and semantic learning.
Neuroanatomical Origins: Suggesting that the functional specialization of the ventral and dorsal streams may emerge from general unsupervised learning dynamics (TC and SMC) rather than being rigidly specified by innate architectural constraints.
Bio-Inspired AI: Offering an alternative to standard machine learning approaches that rely on artificial transformations, proposing instead that the statistical structure of embodied interaction is a viable path to robust visual representation learning.

The authors acknowledge that while E-SSL narrows the gap, a performance difference remains compared to human toddlers (particularly in shape bias and abstract shape recognition), likely due to the absence of language, social interaction, and the use of adult rather than infant data for training.