Quantifying information stored in synaptic connections rather than in firing activities of neural networks

1. Problem Statement

While information theory has long been used to quantify information encoded in the firing activities of neurons (both biological and artificial), there is a significant theoretical gap regarding the quantification of information stored directly in the distribution of synaptic connections (weights).

• Current Limitations: Existing approaches either provide loose upper bounds (e.g., via noise injection in machine learning) or rely on inferring synaptic coding indirectly through neural firing patterns (e.g., dividing total storage capacity by synapse count in Hopfield networks).
• The Core Challenge: Unlike neural firing, which has an explicit mapping to data, synaptic weights arise implicitly from the learning process. This makes deriving a closed-form expression for the mutual information between data patterns and weight distributions mathematically opaque.
• Goal: To develop a theoretical framework that analytically quantifies the Shannon mutual information (MI) between data patterns and ensembles of synaptic connections, moving beyond the "firing activity" perspective to a "connectivity" perspective.

2. Methodology

The authors propose a framework based on densely connected Hebbian networks performing autoassociative memory tasks.

• Model Setup:

  • Data: $N$ independent, $d$-dimensional data patterns $\{x_1, ..., x_N\}$ are modeled as multivariate log-normal distributions. This choice reflects the heavy-tailed statistics observed in biological neuronal populations.
  • Network: A fully connected network where synaptic weights $w_{ij}$ are determined by the Hebbian learning rule:
    $$w_{ij} = \sum_{k=1}^{N} x_i^k x_j^k$$
  • Objective: Calculate the Mutual Information $MI(w; x_l)$ between a set of synaptic weights $w$ and a specific data pattern $x_l$.

• Analytical Derivation:

  • Approximation: Since the sum of log-normal variables does not have a closed-form distribution, the authors employ the Fenton-Wilkinson approximation. This method approximates the sum of independent log-normal variables as a single log-normal variable by matching the first two moments (mean and variance).
  • Single Connection: They derive the probability distribution $p(w_{ij})$ and the conditional distribution $p(w_{ij}|x_l)$ (which is a shifted version of the sum excluding pattern $l$).
  • Ensembles: The framework is extended from single connections to pairs and arbitrary $n$-tuples of connections. The joint distribution of the logarithm of the weights is approximated as a multivariate Gaussian.
  • Formulas: The authors derive closed-form analytical expressions for the Mutual Information for:
    1. Single connections ($MI(w_{ij}; x_l)$).
    2. Pairs of connections ($MI((w_{ij}, w_{mn}); x_l)$).
    3. Arbitrary ensembles of size $n$ ($MI(w; x_l)$).
  • Key Insight: The MI depends solely on the statistical parameters (means and covariances) of the data patterns involved in the specific synaptic ensemble, making the result context-invariant regarding the total network size.

3. Key Contributions

1. First Analytical Framework for Synaptic Information: The paper provides the first closed-form analytical approximations for the Shannon mutual information between data and synaptic weight ensembles, bypassing the need to simulate neural dynamics.
2. Validation of Distributed Coding: The results mathematically confirm that information is distributed across the network; as the number of stored patterns increases, the information stored in any single connection decreases.
3. Discovery of Synergy: The framework reveals that synaptic connections exhibit synergistic interactions. The information encoded by an ensemble of synapses is greater than the sum of the information encoded by individual synapses ($E[MI_{ensemble}] > n \times E[MI_{single}]$).
4. New Capacity Scaling Laws: By analyzing the relationship between network size, ensemble size, and stored patterns, the authors derive a new scaling law for Hebbian network capacity that differs from traditional Hopfield network limits.

4. Key Results

The authors validated their theory through three sets of simulations with varying covariance structures (exchangeable, stochastic, and random).

• Effect of Data Pattern Count ($N$):

  • The information encoded by a single connection (or an ensemble) about a specific pattern decreases as the total number of stored patterns ($N$) increases.
  • Scaling: $MI \sim N^{-1.0 \text{ to } -1.2}$. This confirms that new patterns dilute the information capacity of existing connections, supporting the principle of distributed coding.

• Effect of Ensemble Size ($n$):

  • The total information encoded by an ensemble of $n$ connections increases as $n$ increases.
  • Scaling: $MI \sim n^{1.1}$.
  • Synergy: Crucially, the average information per connection ($MI/n$) increases as the ensemble size grows. This indicates that connections work together synergistically; the whole is greater than the sum of its parts. This synergy was observed up to at least the 20th order of interaction.

• Network Capacity Scaling:

  • By relating the ensemble size $n$ to the network dimension $d$ (where $n \approx d^2/2$) and assuming linear information requirements per pattern component, the authors derived the total storage capacity $P$ (number of patterns) relative to network size $d$:
    $$P \sim d^{1.2}$$
  • This suggests a superlinear storage capacity for dense Hebbian networks, which is higher than the linear ($P \sim d$) or sublinear ($P \sim 0.14d$) limits often cited for Hopfield networks based on retrieval dynamics. The authors argue this represents the storage capacity (upper bound), whereas retrieval capacity is lower due to information loss during the readout phase.

5. Significance and Implications

• Theoretical Shift: The study shifts the paradigm from analyzing neural coding via "spikes" to analyzing it via "weights," offering a complementary perspective on how learning and memory are physically instantiated.
• Biological Relevance: The findings support the biological principle that memory is not localized to specific "grandmother cells" or synapses but is distributed and synergistic across neural ensembles.
• Machine Learning Applications:
  • Pruning: It suggests that pruning neural networks based solely on the isolated information content of a single weight is misleading, as the weight's contribution is context-dependent and synergistic.
  • Interpretability: The framework offers a tool to quantify information storage in specific circuit motifs, potentially making "black box" deep learning models more interpretable.
  • Architecture Design: It implies that the structural arrangement of connections (motifs) plays a critical role in information storage efficiency.
• Robustness: The authors demonstrate that their conclusions (distributed coding, synergy, capacity scaling) are robust to the approximation errors introduced by the Fenton-Wilkinson method, as they rely on qualitative trends rather than precise numerical values.

In summary, this paper establishes a rigorous mathematical foundation for understanding information storage in synaptic connectivity, revealing that neural networks store information more efficiently through synergistic, distributed interactions than previously understood through the lens of firing rates alone.