🧬 biology

Efficient coding along the visual hierarchy

This paper demonstrates that an unsupervised efficient coding model, which compresses visual inputs based on local statistics without labels or backpropagation, successfully builds a human-aligned visual hierarchy from limited data that predicts brain responses and enhances learning when combined with supervised fine-tuning.

Original authors: Ananya Passi, Brian Robinson, Michael Bonner

Published 2026-08-04

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ananya Passi, Brian Robinson, Michael Bonner

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

Imagine your brain is a super-advanced camera that doesn't just take pictures, but learns to understand the world just by looking at it. Unlike the artificial intelligence (AI) we see in movies or video games, which often needs to "study" millions of photos to learn what a cat or a car looks like, your brain can figure things out from just a handful of examples. Scientists have long wondered: how does the brain do this? One leading idea is called "efficient coding." Think of it like a smart compression algorithm. Instead of trying to memorize every single pixel of every image, the brain learns the "rules" of how natural images are built—like how edges, colors, and textures usually fit together. By focusing on the most common patterns and ignoring the random noise, the brain builds a mental library of features that helps it recognize things quickly. The big question is: can this simple, unsupervised "learning by looking" actually build a deep, complex understanding of the world, all the way from simple lines to complex objects, without needing a teacher to tell it what everything is?

In a new study, researchers at Johns Hopkins University decided to test this idea by building a computer model that mimics this biological style of learning. They created a digital brain that learns in layers, much like the visual system in our heads. Instead of being fed labeled pictures (like "this is a dog" or "this is a tree"), the model was simply shown thousands of natural images and told to find the most important patterns within them. It used a mathematical trick called Principal Component Analysis (PCA) to compress the information, keeping only the most dominant variations—like the most frequent colors or shapes—and discarding the rest.

The results were surprisingly powerful. Even without any labels or tasks, this "efficient coding" model started to learn features that looked a lot like what human brains see. The early layers of the model learned to spot simple things like edges and colors. But as the information moved deeper into the network, the layers began to recognize much more complex things, like textures, shapes, and even whole objects. When the researchers tested these features with human volunteers, the people could easily recognize what the computer was "seeing," suggesting the model had learned something very similar to how we perceive the world.

The team also tested a "hybrid" approach, where they let the model learn these patterns first, and then gave it a tiny bit of supervised training (like showing it a few labeled examples) to fine-tune its skills. They found that this hybrid model was incredibly data-efficient. While a standard AI model needed millions of images to learn well, this hybrid model could predict how human brains would react to new images after seeing just 1,000 pictures. In fact, when they gave the hybrid model only one image per category (1,000 images total for 1,000 categories), it still outperformed standard models trained on the same small amount of data.

Perhaps the most exciting finding was that this method made the AI a much faster learner. When the researchers tried to teach the model new categories with very little data, the version that started with efficient coding learned much faster and more accurately than a model that started from scratch. It was as if the efficient coding gave the AI a "head start" by organizing its internal library of features before it even started studying for a specific test. The study suggests that efficient coding isn't just for the early, simple parts of vision; it might be a fundamental principle that shapes our entire visual hierarchy, helping biological brains learn so much from so little. While the researchers note that this doesn't mean supervised learning isn't important, their work suggests that combining the brain's natural ability to find patterns with specific task training could be the key to building smarter, more efficient AI.

Technical Summary: Efficient Coding Along the Visual Hierarchy

Problem Statement

Deep learning models currently achieve state-of-the-art performance in predicting visual cortex representations and visual behaviors. However, they fail to capture a fundamental characteristic of biological vision: the ability to learn useful representations from limited data. While biological systems adapt to the latent statistical structure of sensory stimuli with minimal experience, modern deep learning relies on supervised or self-supervised updates over millions to billions of training samples. This discrepancy suggests that biological vision may be shaped by fundamentally different learning principles. The authors investigate whether efficient coding—the theory that neural representations are shaped to convey maximum information about natural stimuli given biological constraints—can be used to build a deep hierarchy of human-aligned visual features from limited data, without task labels or backpropagation.

Methodology

Model Architecture

The authors developed an 11-layer convolutional neural network based on a scattering network architecture (Guth et al., 2024). The architecture factorizes weights into spatial and channel components:

Fixed Spatial Filtering: Each layer begins with a fixed set of $N$ wavelet filters (specifically, one low-pass $\phi$ filter and eight Morlet band-pass $\psi$ filters at various orientations). These filters detect oriented band-pass features and coarse structures.
Learned Channel Mixing: Following spatial filtering and rectification, a learnable $1\times1$ convolution mixes information across channels.
Subsampling: Every other layer employs strided convolution to reduce spatial dimensions by a factor of two, increasing the effective receptive field in deeper layers.
Normalization: Batch normalization and spatial $L_2$ -normalization are applied to ensure zero-mean, unit-variance outputs and prevent magnitude dominance.

Unsupervised Learning Protocol (Deep Efficient Coding)

The core innovation is a layer-wise, unsupervised learning procedure that requires no global objective, no labels, and no backpropagation:

Local Statistics: For each layer, activations are collected after wavelet filtering, rectification, and global average pooling.
Principal Component Analysis (PCA): The covariance matrix of these spatially pooled activations is computed over a set of natural images (ImageNet). The top $K$ eigenvectors, representing the dominant modes of variance, are extracted.
Weight Optimization: These eigenvectors serve as the weights for the layer's $1\times1$ convolution, effectively compressing the input into a $K$ -dimensional subspace aligned with natural image statistics.
Expansion: In the subsequent layer, the fixed wavelet filters expand this compressed representation by a factor of $N$ .
Rationale: The authors chose PCA over Independent Component Analysis (ICA) or whitening because it directly optimizes for maximally varying features (relevant for sampling-limited regimes in higher-level processing) and provides unique, globally optimal solutions.

Hybrid Learning and Evaluation

The study evaluates three training paradigms:

Unsupervised Efficient Coding: The network is trained solely via the layer-wise PCA procedure.
Conventional Supervised Learning: The network is trained from random initialization using supervised backpropagation on image classification tasks.
Hybrid Learning: The network undergoes the unsupervised efficient coding phase first, followed by supervised fine-tuning on classification tasks.

Evaluation Metrics:

Behavioral Interpretability: Human participants performed a matching task, selecting which cluster of images (high vs. low activation) matched a target cluster, to assess if network channels encode perceptually meaningful features.
Brain Alignment: Encoding models were fitted using regularized linear regression to predict fMRI responses from the Natural Scenes Dataset (NSD) (7T resolution, 8 subjects) based on network layer activations. Performance was measured by the correlation between predicted and actual cortical responses.
Data Efficiency: Classification accuracy on miniImageNet was measured as a function of training samples, including experiments where early layers were frozen to test the transferability of unsupervised features.

Key Results

1. Emergence of Hierarchical Features

The unsupervised deep efficient coding model successfully learned a hierarchy of features without labels.

Low-level layers: Exhibited selectivity for simple features like edges, orientations, and colors.
High-level layers: Exhibited selectivity for complex features, including textures, shapes, and objects.
This progression occurred purely through the hierarchical application of efficient coding principles.

2. Behavioral Relevance

Human Recognition: Participants could reliably identify features represented by specific network channels. The unsupervised model (trained on only 10,000 images) achieved significantly higher accuracy ( $M=0.87$ ) in matching tasks compared to an untrained architecture ( $M=0.68$ ), approaching the performance of a fully supervised model trained on 1 million images ( $M=0.96$ ).
Reaction Time: Reaction times were significantly correlated with channel variance rank ( $\rho = 0.37$ ). Participants identified features from higher-variance channels more rapidly, suggesting that efficient coding organizes representations such that statistically prominent dimensions are perceptually salient.
Necessity of Hierarchy: A control model applying PCA only to the final layer (leaving other layers untrained) performed significantly worse, confirming that the hierarchical structure is critical.

3. Brain Alignment Under Data Limitations

Low-Data Performance: When trained on only 1,000 images (one per ImageNet category), the hybrid learning approach (unsupervised pre-training + supervised fine-tuning) significantly outperformed conventional supervised learning in predicting fMRI responses in the ventral visual stream, particularly in deeper layers (layer 6+).
Data Scaling: As training data increased to 10,000 images, the performance gap between hybrid and conventional learning narrowed, though hybrid learning maintained an advantage in the unsupervised-only baseline.
Spatial Distribution: The performance benefits of hybrid learning were distributed widely across the visual cortex, not confined to specific regions.

4. Data-Efficient Category Learning

Faster Convergence: Hybrid models required fewer supervised training samples to reach the same classification accuracy as conventionally trained models on miniImageNet.
Robustness to Freezing: When early to intermediate layers (up to layer 5) were frozen after unsupervised pre-training, the hybrid model maintained high classification accuracy. In contrast, freezing the same layers in randomly initialized networks caused a dramatic performance drop.
Limited Categories: The advantage of unsupervised pre-training was most pronounced in settings with few categories and few samples per category, facilitating rapid category learning.

Significance and Claims

The authors claim that their findings suggest efficient coding principles may shape representations across the entire visual hierarchy, not just in low-level processing (e.g., V1). The study proposes a new theoretical perspective where biological vision is optimized by a combination of:

Bottom-up efficient coding: Unsupervised adaptation to the statistical structure of natural images, which builds a foundation of reusable, behaviorally relevant features.
Top-down goal-driven learning: Supervised refinement of these representations for specific behavioral tasks.

The paper argues that this hybrid mechanism explains the data efficiency of biological vision, allowing for the rapid learning of useful representations from limited experience. Unlike previous goal-driven models that rely on end-to-end backpropagation, this framework demonstrates that local, unsupervised learning rules can carve out hierarchical representations that are neurally aligned and task-relevant. The authors conclude that efficient coding serves as a powerful initialization for supervised learning, offering a potential solution for computer vision tasks where labeled data is scarce.