Free energy landscape of Dense Associative Memory

Technical Summary: Free Energy Landscape of Dense Associative Memory

Problem Statement
Dense Associative Memories (DenseAMs) represent a class of energy-based neural architectures that utilize higher-than-quadratic neuron interactions to significantly expand memory storage capacity compared to traditional Hopfield networks. While recent research has focused on the energy dynamics of these systems, a comprehensive theoretical framework describing the full free-energy landscape for general DenseAMs has been lacking. Specifically, there is a need to understand how memory retrieval depends on the initial state in higher-order networks and to derive exact free energy functionals for complex interaction types, such as polynomial and Log-Sum-Exponential (LSE) interactions.

Methodology
The authors develop a general analytical framework using large deviations theory to derive the free energy functional for a broad class of associative memories. The methodology proceeds as follows:

1. Model Definition: The system is modeled with $N$ Ising spins ($\pm 1$) and $P$ stored patterns. The overlap between a configuration and the $\mu$-th pattern is defined as $m_\mu$. The Hamiltonian is defined generally as $H(m) = -Nf(m)$, where $f(m)$ is a smooth function representing the interaction cost.
2. Large Deviations Principle (LDP): The authors first determine the probability distribution $Q(m)$ of the order parameter in the absence of the cost function. Using the integral representation of the delta function and the saddle-point approximation in the limit $N \to \infty$, they derive the rate function $I_0(m)$, which characterizes the entropy of the system.
3. Tilted Measure: To incorporate the cost function $f(m)$ (representing the energy landscape), the authors apply the tilted large deviations principle. This connects the rate function in the presence of the cost function, $I_\beta(m)$, to the zero-cost rate function $I_0(m)$ via the relation $I_\beta(m) = I_0(m) - \beta f(m)$, where $\beta$ is the inverse temperature.
4. Fixed Point Derivation: By minimizing the free energy functional $F(m) = \frac{1}{\beta} \inf_m I_\beta(m)$, the authors derive a set of self-consistency equations for the fixed points $\{m_\mu^*\}$. This approach avoids the hard inversion steps often required in standard methods (like Hubbard-Stratonovich transformations) for higher-order interactions.

Key Contributions and Results

• General Free Energy Functional: The paper provides a general expression for the free energy functional $F(m)$ for any smooth interaction function $f(m)$. The functional is given by:
  $$F(m) = \sum_{\nu} m_\nu^* \frac{\partial f}{\partial m_\nu}\bigg|_{m^*} - \frac{1}{\beta} E_\xi \left[ \log \cosh \left( \sum_{\nu} \beta \frac{\partial f}{\partial m_\nu}\bigg|_{m^*} \xi_\nu \right) \right] - f(m^*)$$
  where $E_\xi$ denotes the average over the pattern distribution.

• Polynomial Interaction Models:

  • The authors apply the framework to DenseAMs with polynomial interactions $H = -N \sum (m_\mu)^k / k!$.
  • They derive exact expressions for the free energy and fixed-point equations for any integer $k$.
  • Phase Transitions: For $k=2$ (Hopfield model), the system exhibits a continuous phase transition. For $k \ge 3$, the system undergoes a first-order phase transition.
  • Initial State Dependence: A critical finding is that for $k > 2$, the $m=0$ state (no retrieval) remains a local minimum even after it ceases to be the global minimum. Consequently, memory retrieval becomes highly dependent on the initial state of the system. If the system starts in the basin of attraction of $m=0$, retrieval fails regardless of the noise level, unless a mechanism (like stochastic noise) allows escape from this basin.

• Log-Sum-Exponential (LSE) Model:

  • The framework is applied to the LSE model, which features interactions defined by a logarithm of a sum of exponentials.
  • The authors derive the disorder-averaged ground-state energy and show that for this specific model, the transition threshold coincides exactly with the full-retrieval threshold.
  • Unlike polynomial models, the LSE model achieves error-free retrieval ($m^*=1$) below a critical capacity threshold $\alpha_c$, with no persistent local minima at $m=0$ hindering retrieval once the transition occurs.

• Ground State Energy: The paper evaluates the disorder-averaged ground-state energy $R(m)$ in the extensive limit ($N, P \to \infty$), providing a landscape that describes the gradient dynamics of the retrieval process.

Significance and Claims
The authors claim that their large deviations approach provides a systematic procedure for analyzing diverse and complex associative memory architectures, overcoming limitations of standard techniques when handling higher-order interactions.

• Reproduction of Known Results: The method successfully reproduces classical results for the Hopfield model ($k=2$), validating the framework.
• New Analytical Insights: The paper provides the first exact free energy functional for polynomial DenseAMs with $k > 2$ and clarifies the nature of the phase transition (first-order) and the resulting initial-state dependence.
• Exact Thresholds: For the LSE model, the method yields the exact full-retrieval threshold, aligning with recent findings derived via the random energy model but offering a distinct derivation based on the free energy landscape.
• Generality: The framework is presented as generalizable to other discrete or continuous variables beyond the binary Ising spins used in the demonstration.

The paper concludes that understanding the full free energy landscape is essential for decoding gradient dynamics in DenseAMs, particularly for identifying regimes where retrieval is possible only with specific initial conditions or external noise mechanisms.