🧬 biology

Toward a mechanistic understanding of inference in visual cortex and diffusion models

This paper presents a minimal, interpretable diffusion model based on sparse coding with pairwise interactions that mimics V1 horizontal connections to achieve high-performance image denoising while providing mechanistic insights into both biological perceptual inference and the internal workings of black-box diffusion architectures.

Original authors: Zeyu Yun, Alexander Belsten, Dasheng Bi, Zahra Kadkhodaie, Yubei Chen, Bruno A. Olshausen

Published 2026-07-20

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Zeyu Yun, Alexander Belsten, Dasheng Bi, Zahra Kadkhodaie, Yubei Chen, Bruno A. Olshausen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

The Brain's Detective Work and the AI's Dream Machine

Imagine you are trying to solve a mystery, but the clues are scattered, blurry, and half of them are missing. This is exactly what happens every time you look at the world. Your eyes don't just take a perfect photograph; they receive a jumbled mess of light and shadows. To make sense of it, your brain has to act like a detective, filling in the gaps and guessing what the full picture looks like. This process is called "perceptual inference." For decades, scientists have wondered how the brain does this so effortlessly. They know that neurons in the visual cortex (the part of the brain that sees) are connected in specific ways, like a web of friends who talk to each other to agree on what they are seeing.

At the same time, a new kind of artificial intelligence called a "diffusion model" has become famous for creating stunning images. These AI systems work by starting with pure static noise—like the snow on an old TV screen—and slowly cleaning it up until a clear picture emerges. They are incredibly good at this, but they are also "black boxes." We know they work, but we don't really know how the millions of tiny digital neurons inside them decide what a dog or a face should look like. The big question is: Are the brain and these AI models using the same secret tricks to solve the puzzle of vision? If we can figure out the rules the AI is following, maybe we can finally understand how our own brains work.

The Paper's Big Idea: A Simple Model with a Big Brain

This paper introduces a new, simplified model that tries to bridge the gap between how our brains see and how AI generates images. The authors, a team of researchers from UC Berkeley and other institutions, built a computer program that mimics the primary visual cortex (V1), the first stop for visual information in the brain. Instead of using a massive, complicated deep learning network that is impossible to read, they created a "minimal" model based on a concept called sparse coding.

Think of sparse coding like a puzzle where you only use a few specific pieces to build a picture. In the brain, this means only a small number of neurons fire at any given time to represent an image. The authors took this idea and added a twist: they let the neurons talk to each other in a specific way. In standard models, neurons are often treated as independent workers. But in this new model, the authors gave the neurons a "social network" (an interaction matrix) that lets them influence one another. This allows the model to learn that if one part of an image has a line going up, the next part is likely to continue that line, even if the image is noisy or broken.

What They Found: The Brain's "Social Network"

When the researchers trained this model on natural images (like photos of landscapes and objects), something fascinating happened. The "social network" the model learned looked exactly like the physical connections found in the real human brain. Specifically, the model developed strong connections between neurons that were tuned to similar angles and were lined up in a row. This is known as collinear facilitation.

In the real world, this is why you can easily see a snake slithering through tall grass even if parts of it are hidden. Your brain connects the dots. The paper shows that this model can do the same thing. When they showed it a picture with a broken, hidden contour (like a line that disappears behind a cloud), the model didn't just guess randomly. It used its learned connections to "fill in" the missing line, creating a smooth, continuous curve. This performance was nearly as good as the massive, complex AI models, but with a huge advantage: because their model is simple, the researchers could actually look inside and see how it worked.

The Secret Sauce: How the AI "Thinks"

One of the most exciting discoveries in the paper is how the model handles the "filling in" process. The authors broke down the math to show that the model doesn't just guess; it spreads activity like a ripple in a pond. When a neuron detects a part of a line, it sends a signal to its neighbors. If those neighbors are also active and aligned, the signal gets stronger, reinforcing the idea that a line exists there. If the neighbors are misaligned or inactive, the signal is cut off.

The paper suggests that this process creates a "hierarchy" even though the model only has one layer. About 30% of the neurons in the model learned to disconnect from the direct visual input entirely. These "detached neurons" act like a second, hidden layer that helps the model understand the big picture. They don't see the pixels; instead, they help enforce global rules, like making sure both eyes on a face are in the right place relative to each other. This explains how the model can generate a whole new face that looks realistic, even if it never saw that exact face before. It's not memorizing; it's understanding the geometry of a face.

Why This Matters: From AI to Biology

The paper suggests that the complex, mysterious behavior of modern AI diffusion models might actually be driven by the same simple, biological principles found in our visual cortex. The "black box" of deep learning might just be a very complicated version of the simple, recurrent circuits the authors built.

By proving that a simple, interpretable model can match the performance of giant AI systems, the authors offer a new way to study the brain. They propose that the visual system uses these specific connections to resolve ambiguity, turning a noisy, confusing world into a clear, coherent picture. This isn't just a theory; the model's predictions about how neurons should react to different types of noise align with recent experiments in real biological brains.

In short, this paper suggests that the secret to both seeing and creating art might be the same: a simple set of rules that allow local clues to connect and form a global story. Whether it's a neuron in your brain or a digital weight in an AI, the goal is to find the pattern in the chaos. And now, thanks to this model, we have a much clearer map of how that pattern emerges.

Technical Summary: Toward a mechanistic understanding of inference in visual cortex and diffusion models

Problem Statement
Current diffusion models achieve state-of-the-art generative performance but operate as "black boxes," making it difficult to understand the specific neural computational mechanisms that allow them to assign high probability to natural images or generalize from finite datasets. Conversely, while neuroscience has identified anatomical structures in the primary visual cortex (V1)—specifically horizontal connections between neurons with similar orientation tuning—that likely underlie perceptual grouping and contour completion, it remains unclear how such connectivity emerges from a probabilistic generative model trained on natural images. The central challenge is to bridge these domains by constructing a model that is both a minimal diffusion model with interpretable parameters and a biologically plausible model of V1 inference, capable of explaining how recurrent dynamics resolve visual ambiguity.

Methodology
The authors propose a non-factorial sparse coding model of V1, which they demonstrate is mathematically equivalent to a minimal diffusion model.

Model Architecture:
- The model extends standard sparse coding by introducing a non-factorial prior over latent variables $z$ in the form of an unconstrained, pairwise interaction matrix $M$ .
- The energy function is defined as $E_{\theta,\sigma}(x, z) = \frac{1}{2\sigma^2} \|x - \Phi z\|_2^2 + \gamma(\sigma)(\|\lambda \circ z\|_1 + \frac{1}{2}z^\top M z)$ , where $\Phi$ is a dictionary of image features, $\lambda$ represents sparsity parameters, and $\gamma(\sigma)$ dynamically balances the data likelihood and the prior based on noise level $\sigma$ .
- Inference is performed via a recurrent dynamical system derived from the Maximum A Posteriori (MAP) estimation of $z$ . The update rule (Equation 5) decomposes into a feedforward drive, a recurrent drive (involving the Gram matrix $\Phi^\top\Phi$ and the interaction matrix $M$ ), and a sparsity bias.
Training Strategy:
- Instead of standard Maximum Likelihood Estimation, which is intractable for high-dimensional integration, the model is trained using Denoising Score Matching (DSM).
- The score function is approximated via the MAP estimate $z^*$ , leading to a weighted reconstruction objective.
- Implicit Differentiation: To efficiently train and analyze the model, the authors employ the Implicit Function Theorem (IFT). This allows for the computation of exact analytical gradients through the convergent fixed-point of the recurrent dynamics without unrolling the entire trajectory (avoiding the memory costs of Backpropagation Through Time).
Mechanistic Analysis:
- The authors perform a Jacobian decomposition of the denoising function. By expanding the implicit gradient, they decompose the pixel-space Jacobian $J(x)$ into a latent-space Jacobian $J_z$ projected through the dictionary $\Phi$ .
- $J_z$ is further expanded into a Neumann series, revealing the mechanism of context-dependent activity spreading: signals propagate laterally through $M$ but are strictly gated by the current active sparse code ( $\Sigma$ ), ensuring that only contextually relevant (e.g., collinear) signals survive and amplify.

Key Results

Emergence of V1-like Connectivity: When trained on natural images, the learned interaction matrix $M$ develops strong collinear facilitation. It creates excitatory connections between spatially offset neurons with co-aligned preferred orientations, mirroring the "association field" and horizontal connections observed in superficial V1 layers.
Superior Denoising Performance: The non-factorial model significantly outperforms standard factorial sparse coding models in denoising tasks. While factorial models hallucinate spurious edges in noisy regions, the non-factorial model leverages $M$ to enforce continuity of true structural edges and suppress uncoordinated noise.
Mechanistic Explanation of Diffusion Priors: The analysis reveals that the dominant eigenvectors of the denoising Jacobian (which define the high-probability directions for the model) correspond to oscillatory bases tied to image contours. The model demonstrates that these eigenvectors arise from the infinite accumulation of recurrent lateral spreading constrained by the active sparse code. This mechanism allows the model to assign high probability to a continuous family of natural structural deformations.
Emergent Hierarchy and "Detached Neurons": On structured datasets (e.g., faces), the model learns that approximately 30% of latent variables develop near-zero norms, effectively disconnecting from direct visual input. These "detached neurons" act as an emergent second layer, mediating long-range semantic dependencies and enforcing global consistency (e.g., synchronously shifting facial features) that a single-layer model could not achieve otherwise.
Generative Parity: Despite its architectural simplicity and lack of deep convolutional layers, the model achieves generative parity with parameter-matched U-Net architectures. It produces visually indistinguishable reverse diffusion samples and comparable denoising PSNRs, confirming that the core structural statistics of complex distributions can be captured by this interpretable, biologically grounded architecture.
Hypotheses for V1 Dynamics: The model predicts that the influence of surround excitation in V1 is state-dependent. Under high visual ambiguity (low contrast/noise), the network upweights the interaction matrix $M$ , switching from likelihood-driven suppression to prior-driven facilitation. This suggests a mechanism where excitatory signals are routed along collinear paths only when contextual evidence supports them, resolving low-level ambiguities.

Significance and Claims
The paper claims to provide a concrete, testable mechanistic link between neuroscience and machine learning:

For Neuroscience: It generates specific hypotheses regarding functional connectivity in recurrent neural circuits, suggesting that horizontal connections in V1 emerge naturally to learn co-occurrence statistics for contour integration and that "detached" neural populations may serve a hierarchical role in enforcing global consistency.
For Machine Learning: It elucidates the internal mechanisms of diffusion models, demonstrating that their ability to generate novel images and generalize arises from recurrent dynamics that perform context-dependent contour completion and structural deformation.
Unification: The work posits that visual cortex, deep neural networks, and this minimal sparse coding model are all capturing the same intrinsic statistics of natural images. It suggests that natural images act as sparse signals on a continuous manifold, and that the "black box" nature of deep learning may be obscuring a universal principle of learned, iterative, recurrent computation.

The authors conclude that this framework offers a new direction for mechanistic interpretability in deep learning, suggesting that complex generative behaviors can be understood through simpler, biologically grounded circuit analyses.