Meta-learning as a principle for human-like visual representations

Technical Summary: Meta-learning as a Principle for Human-like Visual Representations

Problem Statement
While pretrained neural networks have achieved unprecedented success in modeling human visual representations, a significant discrepancy remains between machine and human cognition. Current models typically optimize a single, fixed objective, whereas human visual representations must support open-ended, adaptive behavior across a vast array of tasks. The authors hypothesize that the flexibility of human representations arises not merely from scale or data diversity, but from the functional demand to learn new tasks from few observations—a process known as meta-learning (learning to learn). The central question is whether the pressure to "learn to learn" shapes visual representations to align with human similarity judgments, rule learning, and neural activity, even without direct supervision from human data.

Methodology
The study employs a meta-learning framework to reorganize visual features from state-of-the-art frozen image encoders (SigLIP, MAE, DINOv3, and CLIP).

1. Task Generation via Sparse Autoencoders (SAEs): To simulate the diversity of human conceptual space, the authors generate thousands of semantically rich, few-shot learning tasks. They utilize Sparse Autoencoders applied to the residual stream of a pretrained CLIP model. SAE latents decompose entangled activations into sparse, interpretable units corresponding to high-level concepts (e.g., "kitchen items," "small animals"). Each latent defines a task where images with non-zero activations are positive examples, with activation magnitude serving as a graded target.
2. Meta-Learning Architecture: A causal Transformer is trained to predict the output of these tasks given a sequence of images. At each trial, the model receives an image representation from a frozen encoder and the outcome of the previous trial. It learns a linear projection of the encoder features to facilitate in-context learning.
3. Extraction of Static Representations: Crucially, for downstream evaluation, the "meta-learned representation" is defined as the static, linear projection of the encoder features applied independently to each image, stripping away sequential context. This isolates the structural changes induced by the learning-to-learn pressure.
4. Ablation Studies: To identify the drivers of alignment, the authors compare the meta-learned model against:
   • Multitask Learning: A non-sequential model trained on the same task distribution but without episodic structure (no learning-to-learn pressure).
   • Entangled Tasks: Training on raw, entangled residual stream activations instead of sparse SAE latents.
   • Mid-Level Tasks: Training on SAE latents from a lower network layer (Layer 6) to test the role of abstraction level.

Key Results
The meta-learned representations were evaluated across five behavioral and neuroimaging datasets, consistently outperforming the base frozen encoders.

• Behavioral Alignment: Meta-learned representations significantly improved predictions of human similarity judgments (THINGS and Levels odd-one-out tasks) and semantic rule learning (category and reward learning tasks). The gains were most pronounced when the task distribution was both disentangled (sparse) and high-level (abstract).
• Neural Alignment: Using the THINGS-fMRI dataset, meta-learned representations showed systematic improvements in predicting voxel-wise activity in high-level visual cortex, particularly in category-selective regions (EBA, FFA, OFA, LOC, PPA). Early visual areas (V1–V4) showed no significant change, consistent with the high-level nature of the reorganization.
• Geometric Reorganization: t-SNE visualizations and Centered Kernel Alignment (CKA) metrics revealed that meta-learning reshapes the representational geometry to better mirror human conceptual structure. Specifically, it exaggerates the separation between organic and inorganic categories, aligning the model's internal structure more closely with human similarity embeddings.
• Drivers of Alignment:
  • Learning-to-Learn Pressure: Meta-learning significantly outperformed multitask learning, indicating that the pressure to adapt to new tasks on the fly is the primary driver of human-like alignment, not mere exposure to diverse tasks.
  • Task Structure: The benefits were dependent on the task distribution being sparse (disentangled) and high-level. Training on entangled or mid-level features yielded inferior alignment.

Significance and Claims
The paper argues that the structural alignment between human and machine visual representations is a consequence of the functional demand to be repurposed for novel requirements, rather than a by-product of scale or standard feature learning.

• Normative Account: The work offers a normative explanation for why human visual representations are organized as they are: they emerge from the pressure to rapidly acquire new semantic relationships from limited data.
• Mechanism vs. Outcome: Unlike previous approaches that fine-tune models on human data to force alignment, this approach demonstrates that human-like geometry can emerge from general functional demands (meta-learning) alone, without direct supervision from human behavioral or neural data.
• Specificity of Pressure: The results suggest that not all forms of meta-learning produce human-like representations; the effect is specific to learning over a vocabulary of disentangled, high-level concepts.
• Limitations: The authors acknowledge that their task generation relies on existing neural networks (CLIP), inheriting their biases, and that their evaluation is limited to object-centric, static images. They posit that whether these principles hold for dynamic or non-visual inputs remains an open question.

In conclusion, the study posits that the flexibility of human visual representations is a predictable outcome of the evolutionary or developmental pressure to remain adaptable across a broad space of semantically rich tasks.