⚡ electrical engineering

Ubiquity of Emergent Hebbian Dynamics in Regularized Learning

This paper demonstrates that L2 weight decay in regularized learning can generically induce emergent Hebbian or anti-Hebbian alignment in synaptic updates near stationarity, creating a signature that mimics genuine Hebbian plasticity and complicating the interpretation of neural learning mechanisms.

Original authors: David Koplow, Tomaso Poggio, Liu Ziyin

Published 2026-06-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: David Koplow, Tomaso Poggio, Liu Ziyin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Question: Is the Brain "Thinking" Like a Computer?

For decades, scientists have debated how the brain learns.

The Old View: The brain learns using a simple, local rule called Hebbian learning. The famous saying is: "Neurons that fire together, wire together." If two brain cells activate at the same time, the connection between them gets stronger. This is seen as a purely biological, local process.
The Computer View: Artificial Intelligence (AI) learns using Gradient Descent (a complex math method that calculates errors from the whole system to fix mistakes). This requires a "global" view of the problem, which many scientists thought the brain couldn't do because it lacks a central "error signal."

Because we see "Hebbian" patterns in the brain, many assumed the brain must be using simple local rules and not complex global optimization like AI.

This paper asks a tricky question: If we see a Hebbian pattern (strengthening connections when things fire together), does that prove the brain is using a simple Hebbian rule? Or could a complex, global learning system (like an AI) accidentally look like it's using Hebbian rules?

The Answer: It could be an accident. The paper shows that complex learning systems often mimic Hebbian behavior without actually using Hebbian rules.

The Main Discovery: The "Tug-of-War" Analogy

The authors propose that what looks like a biological rule is actually the result of a balance between two opposing forces. Imagine a Tug-of-War on a slippery slope.

1. The Expansive Force (The Learning Signal)

In both brains and AI, there is a force trying to make connections stronger or change them based on new information.

In AI, this is the Gradient (the direction to reduce error).
In the brain, this is the Learning Signal (neurons firing).
Analogy: Imagine a hiker trying to walk up a hill. They are pushing forward, trying to expand their path.

2. The Contractive Force (Weight Decay)

In AI, we use a technique called L2 Weight Decay. This is a penalty that constantly tries to shrink the numbers (weights) in the model to keep them small and prevent them from getting too wild.

Analogy: Imagine the hiker is also tied to a heavy anchor that is constantly pulling them backward, trying to shrink their path.

The Result: The "Emergent" Hebbian Pattern

The paper proves mathematically that when these two forces balance out (when the hiker stops moving up or down the hill and just stays put), the forward push (learning) must align with the direction of the connection itself.

The Metaphor: Imagine you are pushing a heavy box (the connection) across a floor. You are also being pulled back by a rope (weight decay). To stay in one spot, your push has to be perfectly aligned with the direction the box is already facing.
The Surprise: This alignment looks exactly like the "Hebbian" rule (strengthening the connection). But it's not because the brain decided to use a Hebbian rule; it's because the math of balancing forces forces the system to look that way.

Key Takeaway: You can see "Hebbian" patterns in a system that is actually doing complex, global error correction. The pattern is an emergent signature, not necessarily the underlying mechanism.

The Twist: Noise Creates the Opposite (Anti-Hebbian)

The paper also looks at Noise (randomness).

In the brain, there is always electrical noise. In AI, there is randomness in how data is processed (like in Stochastic Gradient Descent).
The Analogy: Imagine the hiker is now on a very windy, stormy day. The wind (noise) is blowing them around randomly.
The Result: If the wind is strong enough, it pushes the hiker in the opposite direction of the connection. This creates an Anti-Hebbian pattern (weakening connections when they fire together).

The Trade-off:
The paper finds a simple "tipping point":

Strong Weight Decay + Low Noise = Looks like Hebbian (Strengthening).
Weak Weight Decay + High Noise = Looks like Anti-Hebbian (Weakening).

This explains why we see both types of plasticity in the brain. It might not be two different biological rules; it might just be the same learning system reacting to different levels of noise and stability.

Why This Matters (According to the Paper)

Don't Jump to Conclusions: Just because scientists measure a "Hebbian" pattern in a brain experiment, it doesn't prove the brain is using a simple local rule. It could be a complex, global learning system (like backpropagation) that is just balancing out its forces.
Coexistence: The brain might be using complex global learning and simple local rules at the same time. The "emergent" Hebbian signature just makes it harder to tell them apart.
New Experiments Needed: We need to design experiments that can distinguish between a "real" Hebbian rule and this "fake" (emergent) one caused by balancing forces.

Summary in One Sentence

The paper argues that the "Neurons that fire together, wire together" pattern might not be a fundamental biological law, but rather a mathematical side-effect that happens whenever any learning system (biological or artificial) tries to balance its learning drive against the need to stay stable and not grow too large.

Technical Summary: Ubiquity of Emergent Hebbian Dynamics in Regularized Learning

Problem Statement

A central question in computational neuroscience is whether the observation of Hebbian or anti-Hebbian plasticity in synaptic updates uniquely implies an underlying Hebbian computation. Traditionally, biological learning is viewed as driven by local, correlation-based rules (e.g., Spike-Time-Dependent Plasticity), whereas artificial intelligence relies on global, gradient-based optimization (e.g., SGD). It is often assumed that the presence of Hebbian signatures in the brain serves as evidence against global error-driven optimization, as gradient descent is thought to require non-local error signals.

This paper addresses an identifiability question: Does observing Hebbian/anti-Hebbian structure in synaptic updates necessarily imply a mechanistic Hebbian learning rule? The authors propose that the answer is no. They argue that Hebbian and anti-Hebbian signatures can emerge as byproducts of regularized gradient-based learning near stationarity, without requiring explicit Hebbian mechanisms.

Methodology

The authors employ a combination of theoretical analysis and empirical validation to demonstrate how regularization and noise interact with learning signals to produce Hebbian-like dynamics.

Theoretical Framework

Stationarity and Weight Decay: The authors analyze the behavior of learning rules near a stationary point (convergence). They consider a general weight update rule $\Delta W = g(x, \theta) - \gamma W$ , where $g(x, \theta)$ is an arbitrary learning signal (e.g., a gradient) and $\gamma W$ represents L2 weight decay.
Alignment Derivation: By assuming the system is near stationarity ( $E[\Delta W] \approx 0$ ), they derive that the expected learning signal $E[g(x, \theta)]$ must approximately equal $\gamma W$ . Since the Hebbian update direction is proportional to $W h_a h_a^\top$ (where $h_a$ is the presynaptic activity), and $W$ is positively correlated with this direction under standard assumptions (e.g., equal-norm representations), the learning signal is mathematically forced to align with the Hebbian direction.
Noise and Anti-Hebbian Dynamics: The authors extend the analysis to include stochastic noise. They show that strong stochastic perturbations (noise) in the learning signal can induce an anti-Hebbian alignment. They derive a phase boundary where the balance between noise strength ( $\sigma^2$ ) and weight decay ( $\gamma$ ) determines whether the system exhibits Hebbian or anti-Hebbian signatures. Specifically, when noise dominates, the alignment becomes negative (anti-Hebbian); when weight decay dominates, it becomes positive (Hebbian).

Empirical Validation

The authors validate these theoretical claims across various tasks, architectures, and learning rules:

Tasks: CIFAR-10 classification and synthetic non-linear regression (student-teacher models).
Architectures: Multi-layer Perceptrons (MLPs) with various activation functions (ReLU, Tanh, Sigmoid) and Transformers.
Learning Rules: Standard SGD, Adam, Direct Feedback Alignment (DFA), and a "RandomNN" construct (a non-learning, deterministic random error signal).
Metrics: The primary metric is the cosine similarity (Frobenius inner product) between the learning signal (or full update) and the theoretical Hebbian update direction ( $h_b h_a^\top$ ).

Key Contributions

Emergent Hebbian Mechanism: The paper establishes that L2 weight decay generically drives the learning-signal component of many update rules to align with a Hebbian direction near stationarity. This alignment increases monotonically with the strength of weight decay.
Emergent Anti-Hebbian Mechanism: The authors demonstrate that sufficiently strong stochastic noise can flip this signature, inducing anti-Hebbian alignment. This creates a predictable trade-off and a phase boundary between Hebbian and anti-Hebbian regimes based on the ratio of noise to weight decay.
Universality Across Rules: The phenomenon is shown to be independent of the specific learning algorithm. It appears in SGD, Adam, DFA, and even in "RandomNN" (a model that does not learn in the traditional sense), provided weight decay is present.
Transient Dynamics: The study identifies that Hebbian alignment is not limited to the final stationary state. An "initial alignment bump" occurs early in training (dependent on initialization and learning rate), and individual neurons exhibit oscillations between Hebbian and anti-Hebbian phases during steady-state training.

Results

Weight Decay Correlation: In classification and regression tasks, increasing weight decay consistently leads to higher positive alignment between the learning signal and the Hebbian update. This trend holds across different activation functions and network depths.
Noise Trade-off: In regression tasks, increasing noise strength reduces Hebbian alignment and eventually induces anti-Hebbian alignment. The phase boundary between these regimes follows a quadratic relationship ( $\gamma \propto \sigma^2$ ), matching theoretical predictions.
RandomNN Validation: Even a model using a purely random, non-learning update signal exhibits Hebbian alignment when weight decay is applied, confirming that the effect is a property of the regularization constraint rather than the learning algorithm itself.
Generalization: The paper notes that solutions with the best generalization performance often exhibit low average Hebbian or anti-Hebbian alignment, suggesting that the "optimal" regime lies between these two extremes.

Significance and Claims

The authors position this work as a cautionary note for interpreting neurophysiological data rather than a replacement for classical Hebbian theory.

Reinterpretation of Biological Data: The presence of Hebbian or anti-Hebbian signatures in synaptic measurements does not uniquely prove that the brain uses mechanistic Hebbian rules. These signatures can be emergent properties of a global optimization process (like gradient descent) constrained by homeostatic mechanisms (like weight decay) and noise.
Coexistence: The authors argue that mechanistic Hebbian plasticity and emergent Hebbian dynamics can coexist. The findings complicate the interpretation of synaptic measurements, motivating the design of experiments that can distinguish between genuine Hebbian computation and emergent signatures arising from regularization.
Theoretical Bridge: The work provides a theoretical link between machine learning optimization dynamics and biological plasticity, suggesting that the "balance of expansive and contractive forces" (Hebbian expansion vs. homeostatic contraction) is a universal principle governing stability in both biological and artificial systems.

The paper concludes that while true Hebbian processes likely occur in the brain, the ubiquity of emergent Hebbian dynamics in regularized learning suggests that observing such patterns is insufficient evidence to rule out global error-driven learning mechanisms in biological systems.