Reward changes the use and structure of predictive spatial maps

Technical Summary: Reward Changes the Use and Structure of Predictive Spatial Maps

Problem and Motivation
Reward-related changes in hippocampal place-cell activity present a fundamental ambiguity: do they reflect a different use of an existing spatial map (altered sampling due to behavioral policy changes) or a reshaping of the map itself (reward-dependent plasticity)? Previous findings show place fields accumulating near rewards or shifting toward goals, but standard behavioral observations conflate these two mechanisms because changes in policy alter the trajectories and sensory histories used to sample the map. The authors argue that distinguishing between "how a map is used" and "how it is rewritten" is essential for understanding hippocampal coding, particularly within predictive frameworks like the successor representation, where policy changes inherently alter state occupancy predictions even without parameter changes.

Methodology
The authors constructed a two-pathway computational model to disentangle these contributions:

1. Predictive Recurrent Network (Phase A): A recurrent neural network (RNN) with 512 hidden units learns to predict the next egocentric sensory observation (field-of-view object vectors and wall distances) from the current observation and action. This network develops a spatial representation (place-cell-like tuning) through sensory prediction alone, without explicit reward or goal signals.
2. Goal-Conditioned Controller (Phase B): An actor-critic reinforcement learning controller is added downstream. It combines the RNN's hidden state with an explicitly supplied allocentric goal-displacement vector to select actions. The controller is trained to reach a rewarded location using Proximal Policy Optimization (PPO).
3. Experimental Controls:
   • Behavioral Contrast: Comparing rate maps from pre-learning exploration (random walk) with post-learning goal-directed behavior. Here, both network parameters and sampled trajectories differ.
   • Fixed-Trajectory Replay: To isolate parameter-dependent changes, the authors replayed the identical stored exploratory observation-action trajectory through the network before and after learning. This holds position, heading, and sensory history constant, isolating changes caused solely by learned parameters.
   • Teaching Signals: The model tested the role of a "reward-proximity teaching signal" (an auxiliary decoder trained to predict a smooth function of distance to the reward) and the role of recurrent plasticity (comparing plastic vs. near-rigid recurrent weights).
   • Comparisons: The model was compared against re-analyzed experimental data from reward-omission and reward-relocation experiments in mice.

Key Results

• Separation of Sampling and Plasticity: Goal-directed behavior produced a strong concentration of place fields near the reward. However, fixed-trajectory replay revealed that the underlying parameter-dependent shift was significantly smaller (retaining only ~16–19% of the behavioral shift) and more variable. This indicates that most of the observed field concentration during behavior arises from the change in how the existing map is sampled (policy change), not from a massive rewiring of the map.
• Role of Teaching Signals: When a reward-proximity teaching signal was added to the learning objective, the parameter-dependent shift under fixed-trajectory replay became larger and more consistently directed toward the reward. This suggests that while plasticity allows the map to change, specific teaching signals are required to direct that change reliably toward the reward.
• Role of Plasticity: Restricting recurrent plasticity (near-rigid weights) reduced the parameter-dependent shift by an order of magnitude, confirming that plasticity is necessary for the map to reorganize, even if the teaching signal directs it.
• Relocation Dynamics: When the reward was relocated, goal-related directional tuning adapted more fully than place-field peak locations. Field peaks retained an influence of the previously learned reward location, whereas the overall mass of the fields re-centered on the new goal. This suggests a dissociation where directional tuning can update rapidly while peak locations retain historical influence.
• Reward Withdrawal: Withdrawing both environmental reward and the proximity teaching signal reversed the learned concentration of fields near the reward, returning them toward baseline. This demonstrates that the reward-related organization is not permanently "frozen" but is actively maintained by these signals.
• Comparison with Data: The model qualitatively reproduced features of hippocampal reward coding, including field accumulation, loss of organization upon reward omission, and the coexistence of place-anchored and reward-relative responses upon relocation. However, the model showed a smaller decline in map reliability upon reward withdrawal compared to experimental data, suggesting the model captures specific mechanisms but may lack others (e.g., latent-state inference).

Significance and Claims
The paper claims to provide a concrete framework for distinguishing between the behavioral expression of a predictive spatial map and its modification by reward-dependent learning.

• Mechanistic Distinction: The results support the view that reward affects predictive spatial maps through separable mechanisms: goal information changes how an existing map is used (sampling), whereas reward-related teaching signals reshape the map itself (plasticity).
• Predictive Coding: The study demonstrates that sensory prediction can establish a spatial representation prior to reward, which can then be recruited for goal-directed navigation. The map retains its overall organization while being shaped by behavioral demands.
• Limitations: The authors explicitly state that the model is a computational abstraction, not a circuit model. It assumes an explicit goal-displacement signal and an externally defined proximity teaching signal, rather than learning these from experience. The model does not reproduce trial-by-trial backward progression of reward activity or the full extent of map destabilization seen in reward-omission experiments.
• Future Directions: The paper suggests that future experiments should focus on disentangling reward information, plasticity, and behavior, potentially by manipulating catecholaminergic inputs (VTA/LC) to test if suppressing reward-related teaching reduces field concentration more than it impairs navigation. It also calls for testing whether the system can discover goals and construct teaching signals from sensory history alone.