Inferring the causes of noise from binary outcomes: A normative theory of learning under uncertainty

This paper presents a normative computational framework that accurately infers environmental volatility and outcome stochasticity from binary feedback using a hidden Markov model with particle filtering, and validates through human experiments that people adaptively adjust their learning rates in accordance with these distinct noise sources.

The Gist

The Big Problem: Is the World Changing, or Are You Just Unlucky?

Imagine you are trying to learn a new game. You make a move, and the result is bad. You have to ask yourself a very important question: "Did the rules of the game just change, or did I just get bad luck?"

• Scenario A (Volatility): The game rules changed. The "good" move is now the "bad" move. You need to learn fast and forget your old habits immediately.
• Scenario B (Stochasticity): The rules stayed the same, but you just got unlucky this one time. You should ignore this single bad result and stick with your old strategy.

The problem is that to your brain, both scenarios look exactly the same: a surprise.

For a long time, scientists had great math to solve this problem, but only for games with smooth, continuous scores (like a thermometer reading). But most real life—and most psychology experiments—uses binary outcomes (Win/Lose, Yes/No, Good/Bad). The old math didn't work well for these "all-or-nothing" results, leading to confusing theories about how we learn.

The New Solution: The "Sea Lion" Detective

The authors of this paper (Fang and Piray) built a brand-new mathematical framework specifically designed for binary outcomes. They call it PF-HMM.

To understand how it works, let's look at their experiment, which they called the "Sea Lion Task."

The Game

Imagine you are on a beach. You have to guess which side of the beach a sea lion is visiting to find treasure.

• The Sea Lion (The Hidden State): The sea lion has a favorite side. Usually, it stays there. But sometimes, it gets bored and switches sides. This switching is Volatility.
• The Waves (The Noise): Even if the sea lion is on the "Right" side, a giant wave might wash the treasure to the "Left" side. This is Stochasticity (randomness).

You can't see the sea lion or the waves. You only see where the treasure appears. Your job is to figure out: Is the sea lion moving, or is the wave just messing with the treasure?

The Old Way vs. The New Way

• The Old Way (The Broken Compass): Previous models tried to force the "smooth" math onto this "binary" game. It was like trying to measure the temperature of a light switch using a thermometer. It worked okay, but it had a glitch: it thought every surprise meant the rules changed, even if it was just bad luck. This made people learn too fast when they shouldn't have.
• The New Way (The Particle Filter Detective): The authors created a new model called PF-HMM.
  • HMM (Hidden Markov Model): This is the detective's notebook. It keeps a running tally of probabilities. "There is a 70% chance the sea lion is on the Right."
  • PF (Particle Filtering): This is the detective's team of 1,000 imaginary detectives (particles). Each detective has a slightly different theory about how often the sea lion switches (volatility) and how often the waves mess things up (stochasticity).

When a new treasure appears, the model checks all 1,000 detectives.

• If the treasure appears where the sea lion usually is, the detectives who thought "waves are crazy" get fired. The detectives who thought "the sea lion is stable" get promoted.
• If the treasure appears in a weird spot, the model asks: "Is this because the sea lion moved (Volatility) or because a wave hit (Stochasticity)?"

The magic of this model is that it can distinguish between the two. It realizes that if the sea lion moves often, you should learn fast. If the waves are crazy, you should learn slow and wait for more data.

What They Found

The researchers tested this with real humans playing the Sea Lion game (and a similar "Turtle" game for losing money).

1. Humans are Smart Detectives: When the game was designed so the sea lion switched sides often (High Volatility), humans learned faster. When the game was designed so the waves were crazy (High Stochasticity), humans learned slower. They intuitively knew the difference between "changing rules" and "bad luck."
2. Thinking Harder Takes Time: The model predicted that when it's hard to tell if the sea lion moved or if it was just a wave, people would take longer to make a choice. The data confirmed this! When the "detectives" in the model were confused (low agreement), the humans took longer to decide.
3. Better than the Old Models: When they compared their new "Sea Lion Detective" model against the old math models, the new one was a clear winner. It explained human behavior much better.

Why Does This Matter? (The Clinical Connection)

This isn't just about sea lions and treasure. This is about how our brains handle uncertainty, which is crucial for mental health.

• Depression and Self-Blame: The paper suggests that people with depression might have a broken "detective." They might mistake Stochasticity (bad luck) for Volatility (a fundamental change).
  • Example: You send a text, and no one replies.
  • Healthy Brain: "Oh, they are probably busy (Stochasticity/Noise). I'll try again later."
  • Depressed Brain (Misattribution): "The rules of my life have changed! I am unlovable (Volatility). I need to change everything about myself immediately."
  • This leads to a cycle of self-blame and maladaptive learning.

The Takeaway

We live in a noisy world. Sometimes the noise is because the world is changing; sometimes it's just random static.

This paper gives us a new, mathematically perfect way to understand how we figure out the difference. It shows that our brains are surprisingly good at separating "changing rules" from "bad luck," but when that system breaks down, it can lead to anxiety and depression. The authors have built a new tool (PF-HMM) that helps us study exactly how that separation happens, opening the door to better treatments for mental health issues.

Technical Summary

1. Problem Statement

Learning in uncertain environments requires distinguishing between two distinct sources of noise:

• Volatility: Changes in the hidden state of the environment (e.g., the rules of the game have changed).
• Stochasticity: Randomness in the observation process given a stable hidden state (e.g., the outcome was noisy despite the rules remaining the same).

The Core Challenge:
While previous normative models (e.g., Kalman filters) successfully handle these distinctions for continuous outcomes, they fail when applied to binary outcomes (e.g., win/loss, yes/no). Existing approaches for binary data typically adapt continuous frameworks using ad hoc approximations (e.g., sigmoid transformations of continuous latent variables). These approximations introduce theoretical inconsistencies:

1. They often lack an explicit stochasticity parameter, conflating it with other variables.
2. They produce spurious learning rate increases following surprising events, even when volatility is zero, because the sigmoid transformation creates a false correlation between surprise and learning rate.
3. They cannot simultaneously and correctly infer both unknown volatility and stochasticity from binary feedback alone.

2. Methodology

A. Theoretical Framework: Hidden Markov Model (HMM)

Instead of approximating continuous models, the authors propose a generative model based on the Hidden Markov Model (HMM), which is naturally suited for binary latent states and binary observations.

• Generative Process:
  • Hidden State ($x_t$): A binary variable (0 or 1) that evolves over time. The probability of switching states is governed by a volatility parameter ($v$).
  • Observation ($o_t$): A binary outcome generated from the hidden state, corrupted by noise governed by a stochasticity parameter ($s$).
• Inference: When $v$ and $s$ are known, the posterior probability of the hidden state can be computed exactly using recursive Bayesian update equations (analogous to the Kalman filter but for discrete states).

B. Computational Model: Particle Filtering HMM (PF-HMM)

In real-world scenarios, $v$ and $s$ are unknown and may fluctuate. To solve this, the authors developed the PF-HMM:

• Mechanism: Combines the exact inference of the HMM with Particle Filtering (PF).
• Rao-Blackwellized Particle Filtering: The model maintains a population of "particles," where each particle represents a hypothesis about the current values of volatility ($v_t$) and stochasticity ($s_t$).
  1. Prediction: Particles drift according to diffusion processes.
  2. Weighting: Particles are weighted based on how well their predicted outcomes match the observed binary feedback.
  3. Resampling: Low-weight particles are discarded; high-weight particles are replicated.
  4. State Inference: For each particle's specific $v$ and $s$ values, the exact HMM equations are used to update the belief about the hidden state ($x_t$).
• Output: The model outputs the expected values of volatility, stochasticity, and the hidden state by averaging across the weighted particle population.

C. Experimental Design

The authors validated the theory using human behavioral data from two experiments:

1. Sea Lion Task (Reward Learning): Participants predicted which side of a beach a sea lion would visit.
2. Turtle Task (Loss Avoidance): A structurally identical task involving loss avoidance.

• Design: A 2×2 factorial design manipulating true volatility (Low/High) and true stochasticity (Low/High) across four blocks.
• Participants: 73 participants (Sea Lion) and 30 participants (Turtle).
• Analysis:
  • Model-Neutral: Fitting a blockwise HMM to extract learning rates without assuming how parameters change.
  • Model-Based: Fitting the PF-HMM to individual data to estimate latent trajectories.
  • Response Time (RT) Analysis: Testing if RTs correlate with "particle likelihood variability" (a measure of inference difficulty).
  • Model Comparison: Bayesian Model Selection (BMS) against Pearce-Hall (PHA) and Hierarchical Gaussian Filter (HGF) models.

3. Key Contributions

1. Normative Framework for Binary Outcomes: The paper establishes the first principled, normative framework for inferring volatility and stochasticity specifically from binary feedback, avoiding the theoretical pitfalls of continuous approximations.
2. PF-HMM Algorithm: Introduction of a computationally tractable algorithm (Rao-Blackwellized Particle Filtering) that allows for the simultaneous, dynamic inference of both noise sources from binary data.
3. Dissociation of Noise Sources: Demonstration that the model can correctly attribute noise to its source (environmental change vs. observation noise), preventing the "explaining away" errors seen in ablated models.
4. Clinical Relevance: The framework provides a computational mechanism to understand psychiatric conditions (e.g., depression, anxiety) where individuals may misattribute stochastic errors to volatility (e.g., interpreting bad luck as a systemic failure), leading to maladaptive learning.

4. Results

Simulation Results

• Optimal Learning Rates: Simulations of the optimal HMM (with known parameters) confirmed that learning rates should increase with volatility and decrease with stochasticity.
• PF-HMM Performance: The PF-HMM successfully recovered the true volatility and stochasticity parameters from synthetic data, showing selective sensitivity (estimating $v$ is unaffected by changes in $s$, and vice versa).
• Ablation Studies: Models with "lesions" (missing one inference module) showed pathological behavior:
  • Stochasticity-ablated: Misattributed all noise to volatility, causing excessive learning rates.
  • Volatility-ablated: Misattributed all noise to stochasticity, causing suppressed learning rates.

Behavioral Results (Human Participants)

• Learning Rate Modulation: Human participants exhibited the predicted normative behavior:
  • High Volatility: Significantly increased learning rates ($p < 0.001$).
  • High Stochasticity: Significantly decreased learning rates ($p < 0.001$).
• Response Times: Participants responded slower when particle likelihood variability was low (i.e., when the model struggled to distinguish between volatility and stochasticity), confirming that the cognitive difficulty of noise attribution is reflected in reaction times.
• Generalization: These effects were replicated in both the reward (Sea Lion) and loss (Turtle) tasks.
• Model Comparison: Bayesian Model Selection strongly favored the PF-HMM over the PHA and HGF models across four datasets (including two external public datasets). The alternative models failed to capture the dissociable effects of volatility and stochasticity.

5. Significance

• Theoretical Advancement: Resolves a long-standing gap in computational learning theory by providing an exact solution for binary outcome learning, replacing heuristic approximations with a rigorous HMM-based approach.
• Psychiatric Insights: Offers a new lens for computational psychiatry. It suggests that disorders like depression may involve a specific failure in the "explaining away" mechanism, where stochastic noise is misinterpreted as environmental volatility, leading to rigid, self-blaming belief updating.
• Methodological Impact: Provides a robust tool (PF-HMM) for analyzing binary decision-making data in neuroscience and psychology, capable of tracking dynamic changes in environmental uncertainty that previous models could not detect.
• Future Directions: The paper highlights the need to extend these inference mechanisms to reinforcement learning and planning (Markov Decision Processes), where the trade-off between exploration and exploitation depends on accurate uncertainty estimation.