Asymmetric Reinforcement Learning Explains Human Choice Patterns in Decision-making Under Risk

This study demonstrates that an asymmetric Risk Sensitive reinforcement learning model, which differentially weights rewards and losses, provides a superior explanation for human choice patterns and response times in decision-making under risk compared to symmetric learning approaches.

The Gist

The Big Idea: How We Learn from Wins and Losses

Imagine you are playing a game where you have to guess if a hidden card is higher or lower than the one you are holding. Sometimes you win a dollar, and sometimes you lose a dollar.

Scientists have long debated a simple question: When we learn from these games, do we treat winning and losing exactly the same way?

• The Old Theory (Symmetric Learning): This theory says our brains are like a perfectly balanced scale. If you win $1, your brain says, "Great, do that again!" If you lose $1, your brain says, "Bad, don't do that!" The weight of the win and the loss is identical.
• The New Theory (Asymmetric Learning): This paper suggests our brains are more like a biased scale. We might learn much faster from a win than from a loss, or vice versa. We might ignore small losses but get super excited about big wins.

This study set out to find out which theory is actually true.

---

The Experiment: The "Starling" Card Game

The researchers created a new game called the Starling Task. Here's how it worked:

1. The Setup: You see a card (say, a 4). You have to guess if a hidden opponent's card is higher or lower.
2. The Twist: The deck of cards isn't always fair.
   • Uniform Deck: All numbers (1–9) are equally likely.
   • Low Deck: Mostly low numbers (1, 2, 3).
   • High Deck: Mostly high numbers (7, 8, 9).
3. The Challenge: Sometimes the deck stays the same for a long time (so you learn the pattern). Other times, the deck changes every single turn, and you have to pay attention to a color clue to know which deck you are in.

47 people played this game (some were healthy volunteers, and some were patients with epilepsy who were already in the hospital for brain monitoring). They played hundreds of rounds, earning or losing fake money.

---

The Detective Work: Testing the Models

The researchers didn't just watch the people; they built five different computer "brains" (mathematical models) to see which one could best predict what the humans would do next.

1. The "Win-Stay, Lose-Shift" Robot: A simple robot that just repeats a move if it wins and changes it if it loses. (Like a toddler learning to walk).
2. The "Greedy" Robot: Always picks the option it thinks is best, but occasionally tries something random just to be safe.
3. The "Smooth" Robot: Picks the best option but mixes in a little bit of randomness, like a smooth curve.
4. The "Double-Tracker" Robot: Keeps two separate scorecards: one for "How much money did I make?" and one for "How risky was this?"
5. The "Risk-Sensitive" (RS) Robot: This is the star of the show. It learns asymmetrically. It has two different "learning speeds": one speed for when it wins, and a different speed for when it loses.

The Results: The "Risk-Sensitive" Robot Wins

After running the numbers, the Risk-Sensitive (RS) Robot was the clear winner. It predicted human behavior better than any other model.

What does this mean?
It means that when humans make decisions under risk, we do not treat wins and losses equally. We update our expectations differently depending on whether the outcome was good or bad.

• The Analogy: Imagine you are learning to cook.
  • If you burn a steak (a loss), you might think, "Okay, I'll lower the heat next time," but you might not remember the exact temperature perfectly.
  • If you cook a perfect steak (a win), you might think, "I'm a genius! I'll definitely do this again!" and remember the exact temperature very clearly.
  • The study suggests our brains work like this: we are asymmetric learners. We don't just add and subtract points on a scoreboard; we weigh the emotional impact of the win differently than the loss.

Why Does This Matter?

1. It explains why we make "weird" choices.
Sometimes people take huge risks because they remember the big wins vividly but forget the small losses. This model explains that behavior perfectly.

2. It helps us understand mental health.
The paper mentions that people with gambling disorders or addiction often have "broken" learning systems. Maybe their "loss learning speed" is too slow, so they keep playing even after losing money because they aren't updating their brain fast enough to realize it's a bad idea. This new model gives doctors a better tool to understand and treat these conditions.

3. It works for everyone.
Interestingly, the study found that people with epilepsy played the game just as well as healthy people. The only difference was that the epilepsy patients were slightly slower to press the buttons. This tells us that the logic of how we learn (the "software") is the same for everyone, even if the speed of our reaction (the "hardware") varies.

The Takeaway

Human decision-making isn't a cold, mathematical calculation where +1 and -1 cancel each other out. Instead, it's a dynamic process where wins and losses hit us with different weights.

We are not perfect calculators; we are Risk-Sensitive Learners. We learn faster from some outcomes than others, and that asymmetry is actually the key to understanding how we navigate a risky world.

Technical Summary

1. Problem Statement

Human decision-making under uncertainty is a core topic in cognitive and neural science, yet the computational mechanisms translating experience into choice remain debated. While Reinforcement Learning (RL) is a standard framework, it is unclear whether human behavior in risky environments is best explained by:

• Symmetric updating: Where gains and losses update value estimates at the same rate.
• Asymmetric learning: Where rewards and losses are weighted differently (e.g., learning faster from losses than gains, or vice versa).

Existing literature suggests that rare events are often underweighted and that individuals vary in their reliance on different strategies. However, a unified model that captures trial-by-trial choice patterns, response times (RT), and the specific impact of asymmetric learning rates in a controlled risk environment has been lacking. This study aims to identify which learning rule best explains human behavior in a novel risky decision-making task.

2. Methodology

Participants

• Total: 47 participants (37 non-epileptic controls, 10 patients with drug-resistant epilepsy).
• Setting: Non-epileptic participants completed the task online; patients completed it in a hospital neuro-acute care unit.

Task: The "Starling" Task

A novel static risk-taking paradigm where participants predict whether their card is higher or lower than an opponent's unseen card.

• Stimuli: Cards numbered 1–9.
• Feedback: Correct choices yield +$0.50; incorrect choices yield –$0.50.
• Experimental Design:
  • Fix Blocks (3 blocks): Participants learned a single deck distribution per block:
    • Uniform: Equal probability for cards 1–9.
    • Low Skewed: Higher probability of low numbers (1–3).
    • High Skewed: Higher probability of high numbers (7–9).
  • Mix Block (1 block): Distributions changed trial-by-trial. Participants used a color cue to identify the current deck distribution.
• Measures: Accuracy, Total Reward, Flip RT (time to reveal card), and Choice RT (time to decide).

Computational Models

Five candidate RL models were fitted to individual trial histories to predict choices and RTs:

1. Win-Stay/Lose-Shift (WSLS): A simple heuristic based on the previous outcome.
2. Rescorla-Wagner (RW) with $\epsilon$-Greedy: Symmetric learning with an exploration-exploitation policy.
3. Rescorla-Wagner (RW) with Softmax: Symmetric learning with a probabilistic policy based on temperature ($\tau$).
4. Dual-Q Model: Maintains separate Q-values for explicit Reward and Risk (uncertainty), updating them independently.
5. Risk-Sensitive (RS) Model: Extends RW with asymmetric learning rates ($\alpha_+$ for positive RPEs, $\alpha_-$ for negative RPEs) and a Softmax policy. This model tests if gains and losses are processed differently.

Analysis Pipeline

• Model Fitting: Parameters were estimated via grid search maximizing log-likelihood.
• Evaluation Metrics: Accuracy, Precision, Recall, Specificity, Bayesian Information Criterion (BIC), and Akaike Information Criterion (AIC).
• Behavioral Correlation: Regression analyses tested how well model-derived latent variables (specifically Q-value differences, $\Delta Q$) predicted human Choice RT and binary choices.
• Parameter Recovery: Simulations verified that fitted parameters were identifiable.

3. Key Results

Behavioral Findings

• Learning: Participants showed monotonic increases in total reward across trials.
• Context Effects: In "Fix" blocks, participants relied heavily on deck priors (base rates), shifting their decision boundaries outward. In the "Mix" block (high contextual uncertainty), decision boundaries collapsed toward the center (card 5), indicating a shift toward trial-specific evidence and base-rate neglect.
• Group Differences: Epileptic participants showed significantly slower Choice RTs but comparable accuracy and reward trajectories to non-epileptic controls, suggesting preserved decision policies despite motor/processing speed differences.

Model Comparison

• Superiority of the RS Model: The Risk-Sensitive (RS) model significantly outperformed all other models (WSLS, $\epsilon$-Greedy, Softmax, Dual-Q) across all metrics:
  • Highest Accuracy, Precision, Recall, and Specificity.
  • Lowest $\Delta$BIC and $\Delta$AIC scores.
  • Best fit to total reward trajectories and choice probability curves (Sigmoid fits).
• Failure of Symmetric Models: Symmetric models (RW variants) failed to capture the sharp transitions in choice probability observed in skewed decks. The Dual-Q model, which separates risk and reward explicitly, performed worse than the RS model, suggesting that the brain does not necessarily track risk as a separate value signal but rather weights outcomes asymmetrically.

Latent Variable Analysis

• $\Delta Q$ and Response Time: The RS model's derived value difference ($\Delta Q = Q_{up} - Q_{down}$) showed the strongest negative correlation with Choice RT. Larger value separations predicted faster decisions, and the RS model captured this relationship better than other models.
• Asymmetry: The RS model estimated a loss learning rate ($\alpha_-$) that was frequently near zero or significantly lower than the gain learning rate ($\alpha_+$). This implies that participants in this task underweighted losses relative to gains, or that losses generated weak value updates, leading to a "stickiness" to high-reward options.

4. Key Contributions

1. Identification of Asymmetric Learning: The study provides robust evidence that human decision-making under risk is better explained by asymmetric learning rates (different weights for gains vs. losses) rather than symmetric updating or explicit separate risk tracking.
2. Novel Task Design: The introduction of the Starling Task allows for the precise manipulation of distributional uncertainty (Uniform vs. Skewed) and the transition from stable priors to volatile contexts (Mix block).
3. Mechanistic Link to RT: The research demonstrates that the latent variables of the RS model (specifically $\Delta Q$) not only predict what people choose but also how fast they decide, offering a unified account of choice and timing.
4. Clinical Generalizability: The findings suggest that the computational mechanism of asymmetric value updating is preserved even in clinical populations (epilepsy) where processing speed is impaired, separating the "policy" from the "execution."

5. Significance

• Computational Psychiatry: The results support the use of asymmetric RL models in understanding disorders like gambling and substance use, where altered sensitivity to rewards and losses is a hallmark. The finding that losses may be underweighted ($\alpha_- \approx 0$) offers a mechanistic explanation for why individuals might persist in risky behaviors despite negative feedback.
• Theoretical Advancement: The study challenges the sufficiency of symmetric RL models and the necessity of complex Dual-Q architectures. It suggests that a simple modification to the learning rule (asymmetry) is sufficient to capture complex human behaviors like base-rate neglect and risk sensitivity.
• Future Directions: The authors propose that these latent variables (asymmetric RPEs) can serve as targets for neural encoding models (e.g., iEEG, fMRI) to locate the neural substrates of asymmetric learning, potentially in the striatum or prefrontal cortex.

In conclusion, the paper establishes that asymmetric reinforcement learning is the most parsimonious and accurate framework for explaining human trial-by-trial choices in risky environments, highlighting that humans do not treat gains and losses as equal inputs to the learning process.