A hierarchy of spatial predictions across human visual cortex during natural vision

By analyzing 7T fMRI data from humans viewing natural images, this study reveals that the visual cortex implements distinct prediction regimes across the visual field, where central vision exhibits a hierarchical sensitivity to predictability from low to high levels, while peripheral vision shows amplified effects where even early areas respond to high-level unpredictability.

The Gist

Imagine your brain is a super-smart detective who never stops guessing what's about to happen next. This paper is about how that detective works when you look at the world around you, like a busy street or a forest.

For a long time, scientists and AI experts have argued about two big questions:

1. Does the brain guess automatically all the time, or only when it's really trying hard?
2. What exactly is the brain guessing about? Is it guessing simple things like "is that a straight line?" or complex things like "is that a dog?"

The authors of this paper decided to solve this mystery by looking at how human brains react to 73,000 natural photos. They used a special type of MRI scanner (7T fMRI) that acts like a high-definition camera for the brain, and they used a powerful AI "painter" to help them figure out what was predictable and what was surprising in those photos.

Here is the story of what they found, broken down into simple parts:

1. The Brain is Always Guessing (The "Surprise" Meter)

First, they wanted to know if the brain actually guesses during normal looking, or if it just waits for things to happen.

They used an AI tool that acts like a predictive painter. Imagine you show the AI a picture of a forest, but you cover up a small patch in the middle. The AI tries to "paint" what it thinks is hidden under that patch based on the trees around it.

• If the AI guesses correctly, the hidden part was predictable.
• If the AI guesses wrong, the hidden part was unpredictable (a surprise).

The Result: When they looked at the brain's reaction, they found that surprise makes the brain work harder. When the hidden part of the image was unpredictable (the AI got it wrong), the brain lit up more. When it was predictable, the brain stayed calm. This proves that the brain is constantly running a "surprise meter" in the background, even when we are just casually looking at pictures.

2. The Central Vision: A Specialized Assembly Line

Next, they looked at what the brain was guessing. They divided the brain's visual areas into a hierarchy, from the very beginning (V1) to the later stages (V4). Think of this like a factory assembly line.

• The Factory Floor (V1): This area handles the basics. The study found that in the very center of your vision (where you look directly at things), this area is mostly guessing about simple, low-level details like edges, lines, and textures. It's like a worker checking if a brick is straight.
• The Managers (V2, V3, V4): As the information moves up the line, the brain starts guessing about complex, high-level things like shapes, objects, and scenes. It's like the manager checking if the whole wall looks like a house.

The Analogy: In your central vision, the brain is like a perfectly organized team where the entry-level workers check the bricks, and the bosses check the building. This matches the "classic" theory of how the brain works.

3. The Peripheral Vision: The "Big Picture" Shortcut

Here is where it gets really interesting. The researchers also looked at what happens in your peripheral vision (the edges of your sight, where things are blurry).

They found that the rules change completely out there.

• Even the "entry-level workers" (V1) in the peripheral vision started guessing about complex, high-level things (like "is that a car?") instead of just simple lines.
• The brain seems to say, "Hey, the edges are blurry and unreliable. I can't guess the tiny details, so I'm just going to guess the big picture instead."

The Analogy: Imagine you are looking at a crowd of people from far away (peripheral vision). You can't see the buttons on their shirts (low-level details), so your brain just guesses, "That's a group of people" (high-level). But if you zoom in with a telescope (central vision), your brain starts guessing, "That's a red button," "That's a blue button."

4. Why Does This Matter?

This study solves a big argument in science.

• Some scientists thought the brain only guesses about big, complex things (like AI models do).
• Others thought the brain guesses about everything, from tiny lines to big objects, in a strict hierarchy.

The Conclusion: The brain is actually both.

• When you look directly at something (high reliability), it uses a hierarchical system: guessing small details first, then big ideas.
• When you look at the blurry edges (low reliability), it switches to a shortcut system: guessing only the big ideas because the details are too fuzzy to trust.

The Takeaway

Your brain is a brilliant, adaptive detective. It doesn't use just one strategy.

• In the center of your gaze: It's a detail-oriented detective, checking every brick and every line, building a prediction from the bottom up.
• In the corners of your gaze: It's a quick-thinking strategist, skipping the tiny details and jumping straight to the "big picture" to save energy and stay safe.

This research helps us understand how human vision works and could help build better AI that sees the world more like we do—knowing when to look at the details and when to just guess the big picture.

Technical Summary

1. Problem Statement

The paper addresses two fundamental unresolved questions in the neuroscience of perception and artificial intelligence (AI):

1. Automaticity: Does prediction occur routinely and automatically during naturalistic perception, or is it a strategic, task-dependent process? Previous studies relied on artificial paradigms (e.g., cueing, oddballs) that do not generalize well to natural vision.
2. Representational Content (Level of Abstraction): At what level of abstraction does the brain predict?
   • Classical Predictive Coding: Suggests a hierarchical gradient where early visual areas (V1) predict low-level features (edges) and higher areas predict high-level features (objects).
   • Reverse Hierarchy/Recent AI Accounts: Suggest that even early areas primarily predict high-level abstractions, or that prediction operates at a single high level of abstraction.

The authors aim to resolve these conflicting accounts by analyzing neural responses to 73,000 natural images without experimentally imposing predictions on the observer.

2. Methodology

The study combines high-resolution neuroimaging with deep generative AI models.

• Dataset: The Natural Scenes Dataset (NSD), containing 7T fMRI recordings from 8 human participants viewing 73,000 natural images.
• Stimulus & ROI Selection:
  • Analysis focused on a central image patch (1° visual angle radius).
  • Voxels were selected based on Population Receptive Field (pRF) mapping to ensure their receptive fields fell entirely within the central patch.
  • Analyses were conducted for both foveal (central) and parafoveal (2° eccentricity) regions to test the effect of sensory reliability.
• Spatial Predictability Modeling:
  • Inpainting Model: A Partial Convolution U-Net (PConvUNet) was used. It was trained to reconstruct (inpaint) a masked central image patch based solely on its surrounding context.
  • Prediction Metric: The "predicted" patch (reconstruction) was compared to the "actual" patch.
  • Abstraction Levels: To measure predictability at different levels of abstraction, the actual and reconstructed patches were passed through VGG-16. The Euclidean distance ($\ell_2$) between their feature representations was calculated at every convolutional and dense layer.
• Neural Analysis:
  • Baseline Model: A Gabor filter pyramid model was used to account for low-level local contrast features (orientation, spatial frequency).
  • Regression: Ridge regression was used to predict voxel-wise BOLD responses.
  • Key Metric: The unique variance explained by spatial unpredictability ($\Delta r$) over and above the baseline model.
  • Tuning Analysis: For each voxel, the authors identified which VGG-16 layer's unpredictability score best explained the neural variance, effectively mapping the "level of abstraction" the voxel is sensitive to.

3. Key Contributions

1. Naturalistic Validation: Demonstrated that spatial prediction is a routine, automatic process during natural scene viewing, not limited to artificial experimental conditions.
2. Resolution of Conflicting Theories: Provided evidence that the visual system implements distinct prediction regimes depending on the location in the visual field (fovea vs. periphery), reconciling classical predictive coding with recent high-level prediction theories.
3. Methodological Innovation: Successfully integrated deep generative models (inpainting) with high-resolution fMRI to quantify "inherent" spatial predictability in natural images without explicit experimental manipulation.

4. Key Results

A. Spatial Predictability Modulates Neural Responses

• Universal Effect: Across all visual areas (V1–V4), neural responses are modulated by spatial predictability.
• Direction: Less predictable inputs evoke stronger responses (positive $\beta$ coefficients), consistent with "expectation suppression" or prediction error signaling.
• Magnitude: Spatial unpredictability explained an additional ~7–9% of neural variance in the fovea over and above the strong baseline model of local contrast features.

B. Hierarchy of Predictions in Central (Foveal) Vision

• Gradient Observed: In the fovea, a clear hierarchy exists that parallels the feature-encoding gradient:
  • V1: Most sensitive to low-level unpredictability (early VGG layers).
  • V2–V4: Progressively sensitive to higher-level unpredictability (deeper VGG layers).
  • Stat: ~23% of V1 voxels were tuned to high-level predictability, compared to ~73% in V4.
• Implication: This supports classical hierarchical predictive coding for central vision, contradicting recent claims that early areas only predict high-level features.

C. Eccentricity Modulates Prediction Regimes

• Amplified Effects: Prediction effects were significantly stronger in the parafovea (peripheral vision) than in the fovea (e.g., V1 foveal $\Delta r$ ~9% vs. parafoveal ~65%).
• Shift in Tuning: In the parafovea, the hierarchical gradient collapsed. Even V1 became predominantly tuned to high-level unpredictability.
  • Stat: In the parafovea, ~53% of V1 voxels were tuned to high-level predictability (vs. 23% in fovea).
• Specificity: This shift was specific to predictability; the tuning for bottom-up visual features remained hierarchical and unchanged across eccentricities.

5. Significance and Discussion

The findings suggest that the brain does not use a single, uniform prediction strategy. Instead, it deploys distinct regimes based on sensory reliability:

1. High Reliability (Fovea): When sensory input is high-resolution and reliable, the system engages in fine-grained, multi-level prediction. Early areas predict low-level details (edges, textures), while higher areas predict objects. This supports detailed inference and recognition.
2. Low Reliability (Periphery): When sensory resolution is low, the system defaults to high-level prediction. Even early visual areas skip low-level details and focus on abstract, high-level structure. This is efficient for rapid scene understanding and gaze guidance.

Theoretical Implications:

• Reconciling AI and Neuroscience: Recent AI models (self-supervised learning) and animal studies (mouse/macaque periphery) often find high-level prediction dominance. This paper explains that these findings likely reflect peripheral processing or low-reliability conditions, whereas central vision follows the classical hierarchical model.
• Functional Asymmetry: The visual field is functionally asymmetric. The fovea is optimized for precise, multi-level inference, while the periphery is optimized for efficient, high-level scene gist extraction.

In conclusion, the paper establishes that spatial prediction is ubiquitous in natural vision but its computational architecture is dynamic, shifting from a strict hierarchy in the center of gaze to a high-level dominant regime in the periphery.