Misalignment Between Backpropagation and the Hierarchy of Brain Responses to Images

Technical Summary: Misalignment Between Backpropagation and the Hierarchy of Brain Responses to Images

Problem Statement
While deep neural networks (DNNs) and the mammalian visual system share striking similarities in their forward representational hierarchies, the biological plausibility of backpropagation—the core learning algorithm of deep learning—remains a central unresolved question in computational neuroscience. Standard backpropagation requires synaptic symmetry (weight transport) and a top-down computation of gradients, mechanisms that are considered biologically implausible. Although previous work has established that forward activations of DNNs align with the cortical hierarchy, it remains unknown whether the top-down hierarchy of backpropagated gradients exhibits a similar correspondence with neural responses. This study addresses whether the brain implements a mechanism functionally equivalent to backpropagation by testing if backpropagated gradients map onto human brain activity and whether their spatial and temporal organization aligns with the algorithmic predictions of backpropagation.

Methodology
The authors employed a multi-modal neuroimaging approach combined with advanced encoding analyses:

• Data Sources: The study utilized the THINGS-MEG dataset (magnetoencephalography, 4 participants, 22,500 images) for high temporal resolution and the Natural Scenes Dataset (7T fMRI, 8 participants, 10,000 images) for high spatial resolution.
• Models: The primary analysis focused on DINOv3 (a state-of-the-art self-supervised Vision Transformer), chosen for its unsupervised learning objective which aligns with the brain's potential reliance on predictive coding. To ensure generalizability, results were reproduced across eight diverse vision models, including supervised CNNs, supervised ViTs, and vision-language models (CLIP, SigLIP2, etc.).
• Encoding Analysis: The authors adapted standard linear encoding analyses to map model features to neural data.
  • Forward Activations: Extracted layer-wise activations.
  • Backpropagated Gradients: Computed gradients using standard backpropagation between teacher and student networks (for DINOv3) or native loss functions for other models.
  • Residual Backpropagation: To isolate gradient-specific signals, the authors computed a "Residual Backpropagation" score by subtracting the encoding score of forward activations from the encoding score of the combined forward and backward signals.
• Dimensionality Reduction: Due to the high dimensionality of gradient vectors (~4 million), signals were projected onto random matrices and subsequently reduced via Principal Component Analysis (PCA) to 300 dimensions prior to encoding.
• Training Dynamics: A subset of experiments involved training DINOv3-S from scratch to observe the evolution of encoding scores across training iterations.

Key Contributions

1. Extension of Encoding Analyses: The study extends standard forward-activation encoding to map backpropagated gradients onto neural data, providing a direct test of learning signal correlates in the brain.
2. Residual Gradient Mapping: By introducing a conservative residual analysis, the authors demonstrate that backpropagated gradients explain neural variance that cannot be accounted for by forward activations alone.
3. Systematic Dissociation: The paper provides empirical evidence that while forward activations align with the brain's bottom-up hierarchy, backpropagated gradients do not follow the expected top-down temporal and spatial organization predicted by biologically plausible backpropagation theories.

Results

• Predictive Power of Gradients: Backpropagated gradients from DINOv3 and other models reliably predict both fMRI and MEG signals, particularly in higher-level visual cortex and at later latencies. This mapping holds across diverse architectures and learning objectives.
• Residual Variance: Residual backpropagation (gradients independent of forward activations) significantly predicts brain activity, specifically encoding representations in the late visual cortex (e.g., ventromedial and lateral-occipitotemporal cortex) and during later time windows (0.5s–1.7s post-stimulus).
• Spatial Misalignment:
  • Forward Pass: Shallow layers predict early visual areas (V1), while deep layers predict associative cortex, reflecting a bottom-up hierarchy.
  • Backward Pass: Contrary to the top-down prediction of backpropagation, deep gradients best align with early sensory areas, and shallow gradients align with associative regions. This indicates a negative spatial correlation, effectively reversing the expected hierarchy.
• Temporal Misalignment:
  • Forward Pass: Encoding peaks sequentially, with shallow layers peaking earlier and deep layers later.
  • Backward Pass: All layers of backpropagated gradients peak simultaneously with the final layer of the forward pass. There is no sequential top-down delay; the gradients do not exhibit the temporal hierarchy expected if error signals were propagated sequentially through cortical layers.
• Training Dynamics: Forward and backpropagation scores increase logarithmically during training, expanding toward higher-level regions and later latencies. In contrast, residual backpropagation follows a non-monotonic trajectory, peaking at 10,000 iterations before declining as the teacher and student networks converge.

Significance and Claims
The authors conclude that while deep networks and the brain may share similar representational content (as evidenced by forward activation alignment), they likely rely on fundamentally different mechanisms to learn these representations. The study claims that the observed spatial and temporal organization of backpropagated gradients in the brain diverges significantly from the patterns expected under a biologically plausible backpropagation mechanism (e.g., sequential top-down propagation of error signals).

The paper posits that these findings provide empirical constraints on theories of biological learning. While the results do not rule out gradient-based learning entirely, they suggest that if such mechanisms exist in the brain, they do not manifest with the specific spatial and temporal signatures of standard backpropagation. The authors suggest that alternative frameworks, such as local Hebbian rules, contrastive learning, or predictive coding architectures that do not require explicit error gradient propagation, may better explain the observed neural dynamics. The work highlights a potential gap between the efficiency of biological learning and current deep learning algorithms, suggesting that understanding the brain's actual learning strategy could inspire more efficient and robust machine learning approaches.