🤖 machine learning

Benchmarking local Hebbian learning rules for memory storage and prototype extraction

This paper benchmarks seven Hebbian learning rules in recurrent networks, finding that while the original additive rule has the lowest capacity and covariance learning offers robustness, Bayesian-Hebbian rules demonstrate superior storage capacity and prototype extraction performance across various conditions.

Original authors: Anders Lansner, Andreas Knoblauch, Naresh B Ravichandran, Pawel Herman

Published 2026-08-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Anders Lansner, Andreas Knoblauch, Naresh B Ravichandran, Pawel Herman

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine your brain as a massive, bustling library where books aren't just stored on shelves but are scattered everywhere, waiting to be found by a single clue. If you whisper a few words from a story, the library should instantly shout back the whole tale. This is the magic of associative memory: the ability to retrieve a complete memory from a partial or messy hint. It's the reason you can smell a specific cookie and instantly remember your grandmother's kitchen, or hear a few notes of a song and recall the entire melody. Scientists have long tried to build computer models that mimic this brainy trick, hoping to understand how our minds work and to create smarter machines. The big question they've been wrestling with is: what is the best "rule" for the brain's librarian to follow when deciding which connections to strengthen? Should the librarian just link things that happen together, or is there a more sophisticated way to organize the chaos so the library never gets overwhelmed?

In this paper, a team of researchers decided to put seven different "librarian rules" to the ultimate test. They built digital brains—specifically, networks of simple units that act like neurons—and asked them to memorize patterns, some of which were messy or distorted versions of a "perfect" original idea. Think of it like trying to teach a robot to recognize a cat. You show it a thousand photos of cats, some with sunglasses, some upside down, some blurry. The goal is for the robot to look at a new, weird photo and say, "That's a cat!" without ever having seen that exact photo before. The researchers simulated these digital brains using seven different learning strategies, ranging from the classic, simple "link what you see together" method to more complex, math-heavy strategies based on probability and logic.

The results were a bit of a shock to the scientific community. The most famous rule, the one named after Donald Hebb (often summarized as "neurons that fire together, wire together"), turned out to be the worst performer. It was like a librarian who just pastes every new note to every other note on the wall; eventually, the wall is so covered in paper that you can't find anything. In contrast, the rules that used a bit of probability and logic—specifically the "Bayesian-Hebbian" rules—were the champions. These rules acted like a super-smart librarian who not only links things but also calculates the odds of a connection being useful, filtering out the noise. These top-performing rules could store about three times more memories than the classic Hebb rule and were much better at ignoring the "noise" or distortions in the photos.

The researchers also tested how well these rules could handle messy data where the clues were correlated (like a library where every book about "dogs" also accidentally had a page about "cats" glued to it). Here, one of the other rules, called PRCOV, showed it was a master at ignoring these confusing overlaps, even beating the probability-based rules in some specific tasks. However, for the overall job of storing and recalling memories, the probability-based rules were the clear winners. The study didn't just find a winner; it also showed that the structure of the network matters. A "modular" brain, organized into small, tight-knit groups (like neighborhoods in a city), performed slightly better than a giant, undifferentiated web.

Ultimately, this paper suggests that if we want to build artificial brains that are as good as ours at remembering things, we might need to stop using the old, simple "fire together" rule and start using these more sophisticated, probability-based rules. It's a reminder that the brain's secret sauce isn't just about making connections; it's about knowing which connections to make and how to weigh them. While these findings come from computer simulations and not a direct scan of a human brain, they offer a strong hint that the brain might be using a much more clever, math-heavy strategy than we previously gave it credit for.

Technical Summary: Benchmarking Local Hebbian Learning Rules for Memory Storage and Prototype Extraction

Problem Statement
Associative memory (content-addressable memory) is a fundamental function in both computer science and cognitive brain science, underlying operations such as pattern completion, figure-ground segmentation, and long-term memory. While various neural network architectures and learning rules have been proposed to model these functions, a systematic quantitative comparison of different local Hebbian learning rules within a unified framework has been lacking. Specifically, the paper addresses the need to benchmark these rules not only for standard pattern storage but also for the less investigated capability of prototype extraction—the ability to recall a generating prototype from distorted instances without the prototype itself being explicitly stored. Furthermore, the study investigates how these rules scale with network size and their sensitivity to correlations in training data.

Methodology
The authors conducted extensive computer simulations comparing seven distinct local Hebbian learning rules within single-layer, recurrently connected Neural Associative Memory (NAM) networks. The study utilized two network architectures:

Non-modular networks: A single layer of $N$ units.
Modular networks: Comprising $H$ hypercolumns, each containing $M$ minicolumns (units), following a "small world" scaling scheme where $H = M = \sqrt{N}$ .

The networks operated on moderately sparse binary patterns ( $\{0, 1\}$ ) with winner-take-all (WTA) dynamics (k-winners-take-all for non-modular, local single-winner for modular). Training was performed in a one-shot, sample-by-sample manner.

The Seven Learning Rules Compared:

WILL: Based on Willshaw (1969), using binary weights.
HEBB: The original additive Hebbian rule.
HOPF: Sparse Hopfield rule adapted for binary units.
COV: Covariance learning (Tsodyks & Feigelman, 1988), adding a term to Hopfield to zero out weights between independently active units.
PRCOV: Presynaptic covariance learning (Minai, 1997), designed to improve capacity for correlated patterns.
BOM: Bayesian Optimal Learning (Knoblauch, 2011), derived from probabilistic considerations without noise dependence.
BCP: Bayesian Confidence Propagation (Lansner & Ekeberg, 1989), derived from Bayes' rule assuming independent inputs.

Evaluation Metrics
Performance was quantified across four main benchmarks:

Pattern Storage Capacity ( $P_{90}$ ): The maximum number of patterns stored such that 90% of distorted test patterns are recalled error-free.
Weight Information Capacity ( $C$ ): The total stored Shannon information (bits) divided by the number of trainable weights, providing a size-independent metric.
Prototype Extraction: The ability to recall the generating prototype from new distorted instances, where the prototype itself was never presented to the network.
Correlation Resistance: The decline in storage capacity as the correlation ( $f_p$ ) among pattern components increases.

Key Results

Pattern Storage Capacity: The Bayesian-Hebbian rules (BCP and BOMs) demonstrated the highest storage capacity across almost all conditions, outperforming other rules by a significant margin. The HEBB rule consistently showed the lowest capacity. HOPF, COV, and PRCOV exhibited intermediate performance. The WILL rule performed poorly, particularly in prototype extraction tasks.
Weight Information Capacity: The maximum weight information capacity observed for the rules was approximately 0.64 bits/weight, occurring at a recall fraction of 50%. The PRCOV rule showed lower values due to its asymmetric weight matrix, while the HEBB rule remained the lowest.
Prototype Extraction: This task proved generally harder than individual pattern storage. The BCP and BOMs rules maintained superior performance. Notably, while PRCOV could extract prototypes, it did so more rapidly for a low number of prototypes but plateaued earlier than Bayesian rules when trained with many instances. The WILL rule failed entirely in this task due to its binary weight constraints.
Correlation Sensitivity: The PRCOV rule, specifically designed for correlated data, showed the highest resistance to correlations, particularly in prototype extraction tasks, sometimes outperforming Bayesian rules in absolute capacity under high correlation. The HEBB and WILL rules were the most sensitive to correlations.
Architecture Scaling: Modular networks generally stored more patterns than non-modular ones, but the information per weight ( $C$ ) remained comparable. The $\sqrt{N}$ scaling of hypercolumns and minicolumns was effective for the simulated sizes but noted as potentially limiting for scaling to full cortical sizes.

Significance and Claims
The paper claims that its primary contribution is a rigorous, quantitative benchmarking of Hebbian learning rules under standardized conditions, revealing that the popular HEBB rule is suboptimal for associative memory tasks compared to Bayesian-Hebbian approaches.

Neurobiological Implications: The results suggest that if the brain utilizes associative memory mechanisms similar to those modeled, it likely employs learning rules more complex than simple additive Hebbian plasticity, potentially involving intrinsic plasticity or Bayesian inference mechanisms (as seen in BCP).
Engineering Implications: For artificial neural systems, the study indicates that Bayesian-Hebbian rules offer superior capacity and robustness. Furthermore, the failure of the binary WILL rule in prototype extraction suggests that real-valued weights (representing higher information content, ~5 bits) may be necessary for robust generalization, offering insights for neuromorphic hardware design regarding weight precision.
Prototype Extraction: The study highlights prototype extraction as a critical but under-studied capability of NAMs, demonstrating that different learning rules exhibit distinct generalization behaviors (e.g., speed of prototype formation vs. stability against spurious attractors).

The authors conclude that while Bayesian-Hebbian rules (BCP and BOMs) are generally superior, the choice of the "best" rule depends on specific task demands, such as the need for high correlation resistance (favoring PRCOV) versus maximum storage capacity (favoring BCP/BOMs). The paper refrains from claiming these rules definitively model the human brain but suggests they provide a more plausible and efficient framework for understanding synaptic plasticity and designing brain-like AI.

Technical Summary: Benchmarking Local Hebbian Learning Rules for Memory Storage and Prototype Extraction

More like this