Explore-exploit instability reveals computational decision-making heterogeneity in early psychosis

This study utilizes computational modeling of decision-making tasks to demonstrate that early psychosis is characterized by distinct neurocomputational subtypes driven by elevated uncertainty sensitivity and decision noise, which cause premature transitions from exploitation to exploration independent of reward learning deficits.

The Gist

The Big Picture: The "Gambler's Dilemma" in Early Psychosis

Imagine you are at a casino with three slot machines. You don't know which one pays out the most, but you can pull the lever and see if you win.

• Exploitation: You find a machine that seems to pay well, so you keep pulling that same lever over and over to maximize your winnings.
• Exploration: You get bored or suspicious that the machine might be "running out of luck," so you switch to a different machine to see if it pays better.

The perfect player knows exactly when to stick with the winner and when to try something new. This balance is called the Explore-Exploit Tradeoff.

This study looked at people in the Early Psychosis (EP) stage (people recently diagnosed with conditions like schizophrenia or bipolar disorder with psychosis) and asked: Do they struggle with this balance?

The Mystery: Why Are They Switching So Much?

The researchers found that people with early psychosis were indeed switching machines way too often. They would find a "winning" machine, pull the lever a few times, and then immediately jump to a different one, even if the first one was still paying out.

The Old Theory: Scientists used to think this happened because these patients couldn't learn which machine was the winner. They thought, "Maybe they just don't understand that Machine A is the best."

The New Discovery: This study proved that theory wrong. The patients could learn which machine was best. Their scores were actually just as good as healthy people's. The problem wasn't that they didn't know the answer; the problem was that they couldn't stick with the answer. They abandoned a good option too soon.

The Two Culprits: "The Jittery Nervous System" and "The Foggy Brain"

To figure out why they were jumping ship, the researchers used advanced computer models (like a digital detective) to look at the tiny mental processes happening inside the brain. They found two main reasons for the instability:

1. The "Over-Sensitive Smoke Detector" (Uncertainty Sensitivity)

Imagine your brain has a smoke detector. For most people, it only goes off when there's real smoke. For these patients, the detector is hypersensitive.

• What happens: Even when the "winning" machine is working fine, their brain screams, "There might be smoke! The environment is changing! We need to check the other machines!"
• The result: They switch machines not because the current one stopped working, but because they are terrified that the rules of the game might have changed. They are constantly looking for a new "safe" option because they feel the world is too unpredictable.

2. The "Static on the Radio" (Decision Noise)

Imagine trying to listen to a radio station, but there is a lot of static (white noise) interfering with the signal.

• What happens: Even if the patient knows Machine A is the best, the "static" in their brain makes it hard to stick to that decision. The signal gets fuzzy, and they randomly decide to try Machine B just because of the noise.
• The result: Their choices become random and inconsistent, not because they are confused, but because the signal-to-noise ratio in their decision-making is low.

The "Three Types" of Gamblers

The most exciting part of the study is that not everyone with early psychosis is the same. The researchers used these computer models to sort the patients into three distinct subtypes, like different character classes in a video game:

1. The "Normative" Player (The Mood Class):

   • How they play: They play the game almost perfectly, just like healthy people. They know when to stick and when to switch.
   • The catch: They are the most likely to have Bipolar Disorder with psychotic features. Their struggle isn't with the game mechanics; it's likely driven by their mood swings, which weren't captured by this specific task.
   • Analogy: They are playing a perfect game of chess, but they are feeling very anxious or depressed about it.

2. The "Hyper-Vigilant" Player (The Uncertainty Class):

   • How they play: They are constantly scanning the horizon. They switch machines because they are convinced the game is rigged or changing.
   • The catch: They have high Uncertainty Sensitivity. They are the ones who have been hospitalized multiple times in their lives.
   • Analogy: They are like a security guard who thinks every shadow is a thief. They are so afraid of missing a change in the rules that they never stay in one spot long enough to win big.

3. The "Noisy" Player (The Cognitive Class):

   • How they play: They struggle to stick to any plan. They switch machines randomly.
   • The catch: They have high Decision Noise and trouble with Negative Symptoms (like lack of motivation or emotional flatness). They also struggle to learn from rewards.
   • Analogy: Their brain is like a radio with bad reception. They know which station is good, but the static keeps making them change the dial randomly.

Why Does This Matter? (The "Precision Medicine" Angle)

For a long time, doctors treated "psychosis" as one big bucket. If you had the diagnosis, you got the same medication.

This study says: Stop treating everyone the same.

• If you are the "Hyper-Vigilant" type, you might need a treatment that helps you feel the world is more stable and predictable (perhaps targeting anxiety or specific brain circuits related to uncertainty).
• If you are the "Noisy" type, you might need a treatment that helps clear the "static" in your brain (perhaps targeting dopamine or norepinephrine systems to improve focus and consistency).

The Takeaway

People with early psychosis aren't necessarily "bad at learning." They are struggling with instability. Some are too scared to stay put (Uncertainty), and some are too distracted to stay on track (Noise).

By using these computer models, doctors can finally see which type of instability a patient has. This is a huge step toward Precision Psychiatry—giving the right treatment to the right person based on how their brain actually works, rather than just based on a label on a chart.

Technical Summary

1. Problem Statement

Psychosis spectrum illnesses (e.g., schizophrenia, bipolar disorder with psychosis) are characterized by significant clinical and neurophysiological heterogeneity, making it difficult to develop individualized treatment strategies. A core deficit in these conditions is impaired goal-directed behavior, specifically an inability to balance exploration (trying new options to gather information) and exploitation (sticking with known rewarding options).

While previous studies have established that individuals with psychosis exhibit excessive choice switching, the underlying computational mechanisms remain unclear. It is debated whether this behavior stems from:

• Reward learning deficits: Failure to learn which options are rewarding.
• Elevated decision noise: Randomness in choice selection.
• Aberrant uncertainty sensitivity: Overestimating environmental volatility or uncertainty.

This study aims to dissect these mechanisms using computational modeling to identify distinct "microcognitive" processes and subtypes within the early psychosis (EP) population.

2. Methodology

Participants:

• Sample: 75 participants with Early Psychosis (EP) and 68 neurotypical controls.
• Design: Longitudinal study with two baseline sessions (~3 weeks apart) to assess test-retest reliability.
• Demographics: Matched for age; groups differed slightly in gender distribution (controlled for in analysis).

Behavioral Task:

• Three-Armed Restless Bandit Task: A dynamic decision-making paradigm where participants choose between three options (Gabor patches).
• Mechanism: Each option has a hidden reward probability that drifts independently and randomly over time (10% chance of changing by ±0.1 per trial).
• Goal: Maximize points by balancing exploration (switching to find better options) and exploitation (sticking with the current best option).

Computational Modeling Approach:

1. Hidden Markov Model (HMM): Used to infer latent states (Explore vs. Exploit) from trial-by-trial choice sequences. This allowed the authors to analyze transition probabilities between states.
2. Model Comparison: The authors compared several learning models to determine the best fit for EP behavior:
   • Reinforcement Learning (RL) Models: Including variants with choice kernels, asymmetric learning, and dual-state learning.
   • Bayesian Learner Model (Kalman Filter): A model that estimates both value and uncertainty, adapting the learning rate based on estimated uncertainty.
   • Key Parameters: The winning model included three free parameters:
     • $\phi$ (Uncertainty Sensitivity): Scales the influence of estimated uncertainty on choice.
     • $\omega$ (Perseveration): Captures value-independent choice stickiness.
     • $\beta$ (Inverse Temperature): Governs decision noise (stochasticity).
3. Clustering: Hierarchical clustering (Ward's linkage) was applied to computational parameters ($\phi$, $\beta$) and cognitive measures (processing speed, verbal memory, executive function) to identify latent subtypes independent of diagnosis.

Statistical Analysis:

• Generalized Linear Mixed Models (GLMMs) for group comparisons.
• Bayesian Information Criterion (BIC) and Akaike Information Criterion (AIC) for model selection.
• Spearman correlations for linking computational parameters to clinical symptoms (PANSS, BPRS) and functioning.

3. Key Results

Behavioral Performance:

• Reward Acquisition: No significant difference in total rewards earned between EP and controls; both performed above chance.
• Choice Switching: EP participants switched choices significantly more often than controls, even after receiving rewards (win-switch) or when selecting non-optimal options.
• Optimal Choices: EP made fewer optimal choices overall.

Computational Mechanisms (HMM & Model Fitting):

• State Transitions: The HMM revealed that EP participants were significantly more likely to transition from Exploit to Explore (leaving a favorable option too soon), particularly after receiving a reward.
• Learning Intact: Transitions from Explore to Exploit were preserved, indicating that the ability to initiate exploitation upon learning a reward is intact. The deficit lies in sustaining exploitation.
• Best-Fitting Model: A Bayesian Learner Model (Kalman filter) outperformed standard RL models.
• Drivers of Over-Exploration:
  1. Elevated Uncertainty Sensitivity ($\phi$): EP participants had significantly higher sensitivity to estimated uncertainty, driving them to switch options prematurely to "reduce" ambiguity.
  2. Increased Decision Noise ($\beta^{-1}$): EP showed a trend toward higher decision noise (lower inverse temperature), leading to less consistent reliance on learned values.
  3. No Reward Learning Deficit: The data did not support the hypothesis that EP participants failed to learn reward contingencies; rather, they abandoned learned rewards due to uncertainty and noise.

Computational Subtypes (Clustering):
Hierarchical clustering identified three distinct subtypes across both EP and control groups, with different distributions:

1. Subtype 1 (Normative): Low uncertainty sensitivity, low noise, strong cognitive performance. Overrepresented in controls and Bipolar Disorder with psychosis.
2. Subtype 2 (Uncertainty-Sensitive): High uncertainty sensitivity, slow processing, poor memory, but intact executive function. Associated with higher rates of lifetime psychiatric hospitalizations.
3. Subtype 3 (High Decision Noise): High decision noise, broad cognitive deficits, and impaired reward learning (difficulty sustaining exploitation). Associated with higher negative symptom severity and Schizophrenia.

Clinical Correlates:

• Functioning: Greater transition to exploration and higher decision noise were associated with poorer real-world functioning.
• Symptoms: Higher decision noise correlated with increased negative symptom severity.
• Subtype Specificity:
  • Subtype 3 (Noise) showed higher disorganized symptoms.
  • Subtype 1 (Normative) showed higher manic symptoms.
  • Subtype 2 (Uncertainty) had the highest hospitalization rates.

4. Key Contributions

1. Mechanistic Refinement: The study moves beyond the binary "reward learning deficit" hypothesis. It demonstrates that excessive switching in early psychosis is driven by instability in the exploitation phase caused by elevated uncertainty sensitivity and decision noise, rather than an inability to learn rewards.
2. Computational Phenotyping: By identifying three distinct computational subtypes (Normative, Uncertainty-Sensitive, High-Noise), the study provides a framework for "precision psychiatry" that cuts across traditional diagnostic boundaries (e.g., Schizophrenia vs. Bipolar).
3. Model Validation: The study validates a Bayesian learner model with uncertainty sensitivity as a superior explanatory framework for restless bandit tasks in psychosis compared to traditional fixed-rate RL models.
4. Clinical Relevance: The findings link specific microcognitive disruptions (e.g., noise vs. uncertainty sensitivity) to distinct clinical profiles (e.g., negative symptoms vs. hospitalization history), suggesting that different patients may require different targeted interventions.

5. Significance and Implications

• Precision Psychiatry: The identification of computational subtypes suggests that treating "psychosis" as a monolithic entity is insufficient. Interventions could be tailored based on whether a patient's primary deficit is noise-driven (requiring stabilization) or uncertainty-driven (requiring cognitive restructuring of volatility beliefs).
• Translational Potential: The restless bandit task is scalable and shares neural circuits (anterior insula, frontoparietal network) across humans, primates, and mice. This allows for direct translation of findings to mechanistic animal studies and pharmacological testing (e.g., targeting dopamine/norepinephrine systems to modulate noise).
• Biomarker Development: The computational parameters (uncertainty sensitivity, decision noise) show moderate-to-high test-retest reliability, offering potential digital biomarkers for tracking disease progression and treatment response, potentially serving as a scalable alternative to complex biomarker panels like B-SNIP.
• Theoretical Advancement: The results support the theory of psychosis as a disorder of aberrant inference, where the brain misinterprets environmental volatility, leading to maladaptive exploration strategies.

In conclusion, this paper demonstrates that the "noise" in psychosis decision-making is not random but structured, arising from specific, measurable computational parameters that define distinct clinical and cognitive subgroups.