Rapid value learning reveals generalized and context-dependent codes in frontal cortex

Technical Summary: Rapid Value Learning Reveals Generalized and Context-Dependent Codes in Frontal Cortex

Problem and Motivation
The field of decision neuroscience faces a methodological disconnect: human neuroimaging studies typically observe value representations in task-naïve participants, whereas non-human primate (NHP) single-neuron recordings rely on extensively trained subjects. A prevalent but untested assumption is that these rapidly emerging value representations in humans are equivalent to the overtrained representations observed in NHPs. However, frontal cortex (FC) responses are known to be modulated by novelty, experience, and task state. It remains unclear whether value-guided computations rely on distinct neural codes for novel versus overtrained stimuli, and how rapidly value associations are acquired in frontal neurons to support flexible choice. Furthermore, while temporal-difference reinforcement learning (TDRL) predicts a backward shift of value signals from reward to predictive cues, it is unknown if this dynamic occurs rapidly enough to support within-session learning in frontal regions like the anterior cingulate cortex (ACC) and orbitofrontal cortex (OFC).

Methodology
The study recorded single-neuron activity from the ACC and OFC in two macaques (Macaca mulatta) performing a task designed to separate the effects of training history from reward value.

• Task Structure: Each session consisted of a Conditioning phase followed by a Choice phase.
  • Conditioning Phase: Subjects learned ten novel cue-value associations via secondary conditioning. Novel cues predicted one of five reward magnitudes or probabilities. Crucially, each novel cue was paired with an overtrained secondary reinforcer (a colored bar) that explicitly indicated the value, allowing for rapid acquisition of both magnitude and probability information.
  • Choice Phase: Subjects made free choices between: (1) two novel cues learned that day (Novel trials), (2) two cues experienced thousands of times previously (Overtrained/OT trials), or (3) one of each (Mixed trials).
• Neural Recording: 156 neurons were recorded from the ACC (area 24c) and 160 from the OFC (areas 11/13).
• Analysis: The authors employed General Linear Models (GLMs) to quantify neuronal selectivity for chosen value, choice novelty, and reward attributes. They utilized cross-correlation of t-statistics to assess the similarity of value codes across conditions (Novel vs. OT, Magnitude vs. Probability) and analyzed temporal dynamics to detect Reward Prediction Error (RPE)-like learning signatures (e.g., the shift of value coding from secondary reinforcer to novel cue).

Key Results

1. Rapid Acquisition: Value signals associated with novel cues emerged rapidly, appearing within 4–7 cue presentations in the ACC and 7–10 presentations in the OFC. This timescale matched behavioral adaptation, where subjects performed above chance within 3–5 presentations.
2. Generalized vs. Context-Dependent Codes:
   • ACC: Neurons utilized a generalized value code. The representation of chosen value was highly correlated between Novel and OT trials and generalized across reward attributes (magnitude and probability). ACC neurons were more likely to encode value in both trial types simultaneously.
   • OFC: Neurons exhibited context-dependent coding. While OFC encoded value in both trial types, the population structure differed; distinct subpopulations were recruited for Novel versus OT trials. OFC value coding was more segregated by task dimension (attribute-dependent), particularly in OT trials.
3. Independent Novelty Coding: Both regions encoded choice novelty (Novel vs. OT) independently of value. However, OFC neurons were significantly more likely to encode novelty than ACC neurons. Crucially, novelty signals preceded chosen value signals by approximately 130 ms, suggesting novelty provides a contextual scaffold for subsequent value computation.
4. RPE-Driven Learning Dynamics in ACC:
   • Temporal Shift: In ACC, as value coding emerged at the novel cue (Late trials), it mirrored the pattern previously observed at the secondary reinforcer (Early trials).
   • Decrease at Reinforcer: ACC neurons that acquired value coding at the novel cue showed a significant decrease in value coding at the secondary reinforcer over time, consistent with RPE theory where the reward becomes fully predicted by the cue.
   • Opponent Coding: ACC cue-value learners displayed opponent coding (opposite signs for value at choice vs. outcome) during the Choice phase, a hallmark of RPE coding. OFC neurons did not exhibit this specific dynamic.

Key Contributions

• Bridging Methodologies: The study demonstrates that value representations in frontal cortex can emerge within a few trials (4–7), supporting the validity of comparing single-neuron data from trained NHPs with human neuroimaging data from naïve participants, provided the learning timescale is considered.
• Regional Dissociation: It establishes a functional dissociation between ACC and OFC during rapid learning: ACC maintains a common-currency, generalized value code across experience levels, while OFC relies on experience-dependent subpopulations and strongly encodes choice novelty.
• Mechanism of Acquisition: The paper provides electrophysiological evidence in frontal cortex for RPE-driven learning dynamics (temporal transfer of value signals and reduction of response to predicted rewards) occurring on a timescale of seconds to minutes, not just days.

Significance and Claims
The authors claim these findings establish the necessary (rapid acquisition) and sufficient (generalized format) conditions for comparing value representations across methods, species, and training histories. Specifically, they argue that while the mechanism of rapid learning in ACC aligns with overtrained representations (supporting cross-species parallels), the neural code in OFC is highly sensitive to novelty and training history. This suggests that standard overtraining protocols in NHPs may mask context-dependent coding mechanisms (like those in OFC) that are critical for flexible, novel decision-making in humans. The results highlight that frontal value coding is not monolithic; it involves a generalized, action-oriented code in ACC and a context-sensitive, novelty-modulated code in OFC, both of which are essential for adaptive decision-making.