Efficient memory sampling by hippocampal attractor dynamics with intrinsic oscillation

This paper proposes a momentum Hopfield model with intrinsic oscillations to demonstrate how hippocampal attractor dynamics can function as biased Markov-chain Monte Carlo sampling, thereby unifying dynamical and functional theories of replay to enable efficient prioritized memory sampling that accelerates reinforcement learning.

The Gist

The Big Picture: The Brain's "Replay" Button

Imagine your brain is a massive library. Every time you learn something new—like the route to a new coffee shop or the rules of a video game—it writes a new book and puts it on a shelf.

Scientists know that when you are sleeping or just sitting still, your brain doesn't just shut down. It hits the "Replay" button. It runs through those books again, very quickly, to strengthen the memories and help you make better decisions later. This is called hippocampal replay.

However, there are two big questions scientists have been arguing about:

1. The Mechanism (How?): How does the brain physically move from one memory to the next? (Is it like a train stopping at every station, or a bird flying between trees?)
2. The Function (Why?): Does the brain just replay everything randomly, or does it pick the most important memories to help you learn faster?

This paper proposes a new theory that answers both questions at once. The author, Tatsuya Haga, suggests that the brain uses a "momentum" system to replay memories, acting like a skater on a frozen pond rather than a train on a track.

---

The Core Idea: The "Momentum" Skater

1. The Old Way: The Stuck Train (Standard Models)

Imagine a standard memory model as a train trying to reach a station (a memory). The train is heavy and has no engine power once it starts moving. It slowly rolls down a hill toward the station. Once it gets there, it stops completely. To go to the next station, it has to be pushed from the outside.

• Problem: This is too slow. It can't easily jump from one memory to another, and it doesn't explain how the brain moves so fluidly between memories.

2. The New Way: The Momentum Skater (The Paper's Model)

The author proposes a new model called the Momentum Hopfield Model. Imagine a figure skater on a frozen pond with many small hills and valleys (these are the memories).

• The Physics: The skater has momentum. When they slide down into a valley (a memory), they don't stop. Because they are moving fast, they shoot up the other side of the valley and glide over to the next one.
• The Result: Instead of stopping at every memory, the skater bounces from one to another in a continuous, flowing motion. This creates a "replay" sequence that looks like a smooth movie rather than a slideshow.

The "Oscillation" Secret:
The paper suggests that the brain's natural electrical rhythms (like the theta and gamma waves we see in EEG scans) are the "wind" that keeps the skater moving. These rhythms give the memory system the energy to keep bouncing between memories without getting stuck.

---

The Magic Trick: Picking the Best Memories (Prioritized Sampling)

Now, let's talk about what the skater chooses to visit.

If the skater just bounces around randomly, they might visit a boring memory (like "what I had for breakfast") 100 times and a crucial memory (like "how to avoid a car accident") only once. That's inefficient.

The paper shows that this "Momentum Skater" system can be biased.

• The Analogy: Imagine the skater is wearing a backpack. If they carry a heavy backpack (representing a high "value" or "importance" of a memory), they move slower and spend more time in that valley. If the backpack is light, they zip right through.
• Real Life Application: In the brain, chemicals like dopamine act as the backpack. If you just learned something that gave you a big reward (or a big surprise), the brain makes that memory "heavier." The skater spends more time there, replaying it over and over.

This is called Prioritized Experience Replay. It's the brain's way of saying, "Don't waste time on the boring stuff; let's practice the stuff that helps us survive and win!"

---

What the Computer Simulations Showed

The author built a computer program based on this "Momentum Skater" theory and tested it in two ways:

1. The Maze Test (Spatial Navigation):
   The computer had to learn how to navigate a maze.

   • Result: When the computer used the "Momentum Skater" to replay the most important paths (the ones near the goal), it learned the maze much faster than if it just replayed paths randomly.
   • Takeaway: The brain doesn't just memorize; it strategically practices the moves that matter most.

2. The Sleep vs. Awake Test:

   • Awake: When the skater is awake, they move with purpose and speed (like running a specific route). The paper found that the model mimics the "smooth, fast" replay seen in awake animals.
   • Sleep: When the skater is asleep, the author added "noise" (random bumps) to the system. Suddenly, the skater started bouncing around randomly, like a ball in a pinball machine. This mimics the "random, Brownian motion" replay seen in sleeping animals, which helps consolidate general memories.

---

Why This Matters

This paper is a bridge. It connects two worlds that scientists usually keep separate:

1. The Physics of the Brain: How neurons fire, oscillate, and move (the "bottom-up" view).
2. The Logic of Learning: How we learn efficiently and make decisions (the "top-down" view).

The Conclusion:
The brain isn't just a static hard drive storing files. It's a dynamic, bouncing system. By using momentum and oscillations, the hippocampus can fluidly jump between memories. By using biases (like dopamine), it can choose to spend extra time practicing the most important lessons.

In one sentence: The brain is like a skater on a frozen pond of memories; it uses its own momentum to keep moving, and it carries heavy backpacks on the most important memories to make sure it practices them the most.

Technical Summary

1. Problem Statement

The paper addresses a fundamental gap in understanding hippocampal memory replay. Two distinct theoretical frameworks exist but remain disconnected:

• Dynamical (Bottom-up) View: Hippocampal replay is modeled as Hopfield-type attractor dynamics where the network minimizes energy to recall patterns. However, standard attractor models converge to fixed points and struggle to generate the sequential transitions observed in biological replay without artificial instability mechanisms (e.g., spike-frequency adaptation).
• Functional (Top-down) View: Replay serves to accelerate learning by prioritizing the sampling of valuable experiences (Prioritized Experience Replay, PER). Current models often treat this as an external algorithmic process rather than an emergent property of the network's dynamics.

The Core Question: How can a single dynamical model explain both the sequential nature of hippocampal replay (including oscillations) and its ability to perform efficient, biased memory sampling for reinforcement learning?

2. Methodology: The Momentum Hopfield Model

The author proposes the Momentum Hopfield Model, an extension of the modern Hopfield network (which uses self-attention mechanisms) by incorporating Hamiltonian dynamics.

Mathematical Formulation

• State Variables: The model introduces a generalized position vector $\mathbf{x}$ (state) and a generalized momentum vector $\mathbf{r}$.
• Hamiltonian (Total Energy): The system is defined by a total energy $H(\mathbf{x}, \mathbf{r}) = E(\mathbf{x}) + K(\mathbf{r})$, where:
  • $E(\mathbf{x})$ is the potential energy derived from the modern Hopfield energy function.
  • $K(\mathbf{r}) = \frac{1}{2}\|\mathbf{r}\|^2$ is the kinetic energy.
• Dynamics: Instead of gradient descent (which minimizes energy), the system evolves via canonical Hamiltonian equations:
  $$\frac{\partial \mathbf{x}}{\partial t} = \frac{\partial H}{\partial \mathbf{r}} = \mathbf{r}$$
  $$\frac{\partial \mathbf{r}}{\partial t} = -\frac{\partial H}{\partial \mathbf{x}} = -\nabla E(\mathbf{x})$$
  This results in a second-order differential equation: $\frac{\partial^2 \mathbf{x}}{\partial t^2} = -\nabla E(\mathbf{x})$.
• Key Property: The total energy is conserved. As the state falls into a potential well (recalling a memory), kinetic energy increases, propelling the state out of the well and into adjacent wells. This creates intrinsic oscillations and sequential transitions without external noise or adaptation mechanisms.

Biological Mapping (CA3-CA1 Circuit)

The model is mapped to hippocampal anatomy:

• CA3: Modeled as a recurrent network with second-order dynamics (coupled oscillators), generating intrinsic oscillations (potentially corresponding to slow gamma rhythms).
• CA1: Acts as a readout layer, projecting the oscillating CA3 activity to precise memory patterns.
• Readout: A specific readout vector $\mathbf{y}$ is introduced to project the noisy oscillating state $\mathbf{x}(t)$ onto clean memory patterns $\boldsymbol{\mu}_i$, enabling precise recall despite internal oscillations.

3. Key Contributions

1. Unification of Dynamics and Function: The paper bridges the gap between attractor dynamics and efficient sampling by showing that Hamiltonian dynamics naturally implements Hamiltonian Monte Carlo (HMC) sampling.
2. Biologically Plausible Oscillation: It demonstrates that sequential replay and intrinsic oscillations (gamma/ripple coupling) emerge naturally from energy conservation, rather than requiring ad-hoc instability terms.
3. Controllable Sampling Bias: The model proves that the frequency of memory recall can be arbitrarily biased by modulating the norms of memory patterns ($\|\boldsymbol{\mu}_i\|$) or adding bias parameters, effectively implementing Prioritized Experience Replay (PER) as a dynamical property.
4. State-Dependent Replay Properties: The model explains the difference between awake and sleep replay:
   • Awake: Deterministic dynamics (low noise) lead to non-Brownian, directed trajectories.
   • Sleep: Adding noise to the dynamics leads to Brownian-motion-like trajectories, consistent with experimental observations.

4. Results

A. Sequential Replay in Spatial Structures

• 1-D Linear Track: The model successfully reproduced sequential replay of place cells. With many memories, the trajectory appeared continuous; with fewer memories, it exhibited discrete jumps, matching experimental data.
• 2-D Spatial Field: The model generated 2-D replay trajectories.
  • Awake Simulation: Trajectories showed a scaling exponent $\alpha > 0.5$ (super-diffusive), indicating directed movement driven by momentum.
  • Sleep Simulation: Adding noise to the internal activation or memory patterns resulted in $\alpha \approx 0.5$ (Brownian motion), replicating sleep-state replay statistics.

B. Theoretical Link to Markov Chain Monte Carlo (MCMC)

• The author proved that the Momentum Hopfield model performs Hamiltonian Monte Carlo (HMC) sampling from a Gaussian mixture distribution where memory patterns are the means.
• Bias Control: The probability of recalling a specific memory $i$ is proportional to $\exp(\frac{\beta}{2}\|\boldsymbol{\mu}_i\|^2 + b_i)$. By modulating the norm $\|\boldsymbol{\mu}_i\|$ (e.g., amplifying patterns associated with high TD errors), the network can prioritize specific memories.

C. Accelerated Reinforcement Learning

• Task: Spatial navigation in a 20x20 grid world using Q-learning.
• Implementation: Memory patterns representing episodes (state-action-reward) were embedded in the network. The sampling frequency was modulated based on:
  1. TD-bias: Proportional to the absolute Temporal Difference error.
  2. Reward-bias: Proportional to proximity to the reward.
• Outcome:
  • TD-bias significantly accelerated learning in both open arenas and complex multi-room environments. It dynamically shifted focus from reward-proximal areas to the whole space as learning progressed.
  • Reward-bias improved learning only in simple environments; it failed in complex mazes because it over-sampled near the reward and under-sampled distant states necessary for pathfinding.
  • Both biased methods outperformed unbiased attractor dynamics and random sampling.

5. Significance

• Theoretical Synthesis: The paper provides a rigorous theoretical link between the dynamical mechanisms of the hippocampus (oscillations, attractors) and its computational function (efficient sampling for decision making).
• Biological Plausibility: It offers a mechanistic explanation for how the CA3-CA1 circuit could implement prioritized replay without external controllers, suggesting that dopaminergic modulation of synaptic weights (changing pattern norms) could drive the bias.
• AI Applications: The model suggests that second-order dynamics (momentum) in neural networks can be used for efficient sampling in reinforcement learning, potentially improving the training of agents in complex environments.
• Future Directions: The work opens avenues for exploring how specific oscillatory frequencies (theta, gamma, ripple) relate to specific sampling rates and how this framework can be applied to Bayesian inference and hidden Markov models in the brain.

In summary, the Momentum Hopfield Model posits that the hippocampus acts as an energy-conserving, oscillatory sampler that naturally transitions between memories. By tuning the energy landscape (via pattern norms), the brain can dynamically prioritize the replay of critical experiences to optimize learning.