Extremely coarse learning objectives induce human-aligned representations in AI vision models

Technical Summary: Extremely Coarse Learning Objectives Induce Human-Aligned Representations in AI Vision Models

Problem Statement
While deep neural networks trained on fine-grained visual tasks (e.g., 1,000-class ImageNet classification) have been shown to develop internal representations resembling the primate ventral visual stream, a critical question remains unexamined: how complex must the optimization objective be to yield brain-like representations? Previous research has progressively embraced finer-grained learning signals, from object classification to contrastive self-supervised objectives that distinguish individual images. However, there is little evidence that the biological visual system relies on comparably fine-grained supervision during development; early categorization often proceeds along coarse distinctions (e.g., animate vs. inanimate) before subordinate-level identification. This study investigates whether training on extremely coarse categorical distinctions is sufficient to induce representations that align with human vision, or if fine-grained supervision is strictly necessary.

Methodology
The authors developed a data-driven, modality-agnostic framework to systematically vary the granularity of the learning signal while controlling for all other training variables.

1. Coarse Label Generation: Instead of using manual annotations or predefined semantic hierarchies (like WordNet), the authors derived coarse category labels directly from the statistical structure of visual inputs using pretrained neural networks. They extracted feature representations of ImageNet training images from source models (e.g., CLIP, AlexNet, ViT, DINOv3) and computed the principal components (PCs) of these representation spaces.

   • Labels were generated by partitioning the image set via median splits along the first $n$ principal components.
   • This creates a nested hierarchy of categories: 2 classes (PC1), 4 classes (PC1+PC2), 8 classes, up to 64 classes.
   • This approach ensures the categories are derived from the dominant axes of variation in natural image statistics without manual intervention.

2. Training Regime: Hundreds of neural networks were trained from scratch on the ImageNet corpus. The study varied only the number of training classes ($K \in \{2, 4, 8, 16, 32, 64\}$) while holding architectures (AlexNet-style CNNs, ResNet-50, ConvNeXt, ViT), hyperparameters, data augmentations, and random seeds constant. A 1,000-class baseline (standard ImageNet) and untrained controls were included for comparison.

3. Evaluation Metrics: Representational alignment was measured using Representational Similarity Analysis (RSA). The authors compared the Representational Dissimilarity Matrices (RDMs) of network layers against three biological/behavioral benchmarks:

   • Human fMRI: Natural Scenes Dataset (NSD), focusing on the ventral visual stream.
   • Macaque Electrophysiology: THINGS Ventral-stream Spiking Dataset (TVSD), covering V1, V4, and Inferotemporal (IT) cortex.
   • Human Behavior: THINGS dataset, utilizing 4.7 million "odd-one-out" similarity judgments from ~12,000 participants to create a 66-dimensional behavioral embedding.
   • Note: Layer selection was performed on an independent partition of stimuli to avoid overfitting, and scores were reported on a held-out reporting partition.

Key Contributions and Results

1. Coarse Supervision Matches or Exceeds Fine-Grained Neural Alignment:

   • Networks trained on as few as eight broad categories developed representations that matched or exceeded the neural alignment of models trained on 1,000 classes.
   • This result held for both human fMRI (ventral stream) and macaque electrophysiology (IT cortex).
   • In early visual cortex (V1 in macaques, early visual stream in humans), alignment was largely independent of training granularity, with untrained networks often performing as well as trained ones, consistent with strong architectural priors in early vision.

2. Superior Behavioral Alignment:

   • Strikingly, coarsely trained networks (specifically those trained on 8 classes derived from CLIP embeddings) achieved higher alignment with human perceptual similarity judgments than all other models evaluated.
   • This included surpassing leading large-scale vision models (CLIP, DINOv3) and networks trained with fine-grained supervision or self-supervision.
   • This advantage was robust across different architectures (CNNs and Transformers) and training seeds.

3. Distinct Representational Geometry:

   • Visualization of the representational space revealed that coarse-trained networks learn a qualitatively different geometry than their source networks. While the source network's coarse classes overlapped in its own representation space, the coarse-trained network learned to pull these classes apart into distinct regions.
   • Dimensionality reduction experiments confirmed that the high alignment of coarse networks is not merely a result of projecting a fine-grained representation into a low-dimensional subspace; the coarse-trained geometry is distinct and cannot be recovered by simply truncating the principal components of a 1,000-class model.

4. Data Efficiency:

   • Coarse supervision proved highly data-efficient. Networks trained on only 1% of the ImageNet dataset (10,000 images) using 8 coarse classes achieved higher behavioral alignment than 1,000-class networks trained on the full dataset (~1 million images).

5. Generalizability:

   • The effect persisted across different source networks used to generate labels (AlexNet, CLIP, ViT, DINOv3) and even when using handcrafted semantic labels based on interpretable dimensions (natural vs. artificial, size, indoor vs. outdoor), though data-driven labels remained slightly more effective.
   • The benefit was observed across modern architectures including ResNet-50, ConvNeXt, and Vision Transformers.

Significance and Claims
The paper claims that human-like visual representations can emerge from surprisingly simple learning objectives, challenging the prevailing assumption that fine-grained instance-level discrimination or complex self-supervised tasks are necessary for brain alignment.

• Reframing Learning Signals: The findings suggest that the optimization objectives required to learn brain-aligned representations may be far more rudimentary than previously thought. Learning to cluster natural images into extremely coarse categories (as few as eight domains) is sufficient to develop internal representations that mirror human vision.
• Advantage of Coarseness: In the context of behavioral alignment, coarse supervision is not just sufficient but advantageous. It captures the high-level structure of human object perception more effectively than fine-grained training, suggesting that the "broad" categorical distinctions found in early development are critical for shaping the representational geometry of the visual system.
• Pathway to Human-Aligned AI: The authors propose that coarsening the supervisory signal offers a scalable path toward building AI systems more aligned with human perception, as it requires no manual annotation and can be applied to any pretrained model or modality.
• Future Directions: The paper suggests future work should explore hierarchical curricula where coarse supervision builds a representational scaffold for subsequent fine-grained learning, and investigate biological mechanisms (e.g., feedback pathways from prefrontal cortex or thalamus) that might deliver such coarse supervisory signals in the developing brain.

The study concludes that the field should look beyond fine-grained optimization to understand the fundamental factors governing alignment between biological and artificial vision, positing that the simplest forms of optimization may suffice to converge on human-like representations.