Dynamic distortion of inferred reward probability shapes choice over time

This study reveals that when choices depend on inferring latent variables like elapsed time and reward probability without continuous sensory evidence, human behavior approaches optimality through a dynamic, linear distortion of inferred reward probabilities and reward-modulated temporal precision, rather than following traditional Weber-law scaling.

The Gist

Imagine you are playing a high-stakes game of "Red Light, Green Light," but with a twist. You don't know exactly when the light will change, and the rules of the game shift depending on when you decide to move.

This paper, titled "Dynamic distortion of inferred reward probability shapes choice over time," by Grabenhorst and Maloney, explores how our brains make decisions when we have to guess two things at once:

1. How much time has passed? (We can't see a clock; we have to feel it internally).
2. What is the best move right now? (The odds of winning change every second).

Here is the story of what they found, explained simply.

The Game: A Shifting Target

The researchers set up a computer game.

• The Start: A "Warning" light flashes.
• The Wait: A random amount of time passes.
• The Go: A "Go" light flashes.
• The Choice: You must instantly press either a Left button or a Right button.

The Catch: The reward isn't fixed.

• If you press Left early, you win big. But as time goes on, the chance of winning with Left drops.
• If you press Right early, you lose. But as time goes on, the chance of winning with Right rises.
• Somewhere in the middle, there is a "Crossover Point" where the odds are exactly 50/50.

To win the most money, you need to guess exactly when that crossover point happens and switch your strategy instantly.

The Two Big Surprises

The researchers expected people to be bad at this because guessing time is hard. Instead, they found that humans are actually quite clever, but they use two specific "mental shortcuts" (or distortions) to get it right.

1. The "Magnifying Glass" Effect (Time is Flexible)

The Old Idea: Scientists used to think that our internal clock gets fuzzier the longer you wait. It's like a rubber band: the longer you stretch it, the more it wobbles. This is called Weber's Law. If you wait 10 seconds, your guess is much worse than if you wait 1 second.

The New Discovery: The study found that time doesn't get fuzzy because of how long you wait; it gets fuzzy because of how boring the moment is.

• Analogy: Imagine you are waiting for a bus. If you know the bus is very likely to arrive in the next 30 seconds, your brain sharpens its focus. You are hyper-aware of every second. But if the bus is unlikely to come for a long time, your brain relaxes and loses track of the seconds.
• The Result: The brain zooms in (high precision) when the reward is high and zooms out (low precision) when the reward is low. We don't track time; we track importance.

2. The "Squeeze" Effect (The Probability Distortion)

The Old Idea: If the odds of winning are 80%, you should act like it's 80%. If it's 20%, act like it's 20%.

The New Discovery: Our brains don't treat probabilities linearly. They "squeeze" them.

• Analogy: Imagine a rubber band representing the odds of winning.
  • If the odds are very low (10%), our brain thinks, "That's basically zero!" and acts even more cautiously.
  • If the odds are very high (90%), our brain thinks, "That's basically a guarantee!" and acts even more confidently.
  • The Sweet Spot: The brain exaggerates the difference between "maybe" and "almost certainly." It pushes low probabilities lower and high probabilities higher.
• Why do this? This "squeeze" helps us make a clear decision. Instead of wavering back and forth when the odds are close to 50/50, the brain pushes us to pick a side decisively. It turns a blurry "maybe" into a sharp "go."

The "Goldilocks" Zone

The most fascinating part is why we do this.

If we were perfect robots, we would switch our choice the exact millisecond the odds flip. But that requires perfect time-keeping, which is impossible.
If we were too lazy, we would just guess randomly.

Instead, humans find a Goldilocks Zone.

• We distort the probabilities just enough to make a confident choice.
• We sharpen our time-keeping just enough to catch the big rewards.
• We don't try to be perfect (which is too hard), but we don't settle for average (which loses money). We find the "high-yield" spot where we get almost all the possible rewards with the least amount of mental effort.

The Takeaway

This paper tells us that when we are making decisions in a changing world, our brains aren't just passive observers. They are active editors.

1. We edit our sense of time: We pay close attention only when it matters (high reward).
2. We edit our sense of risk: We exaggerate the odds to force ourselves to make a clear choice.

It's like a skilled driver navigating a foggy road. They don't try to see every single pebble (perfect time); they focus intensely only on the curves where a crash is likely (high reward), and they steer decisively rather than hesitating. This "dynamic distortion" is actually a feature, not a bug—it's how our brains stay efficient and successful in a chaotic world.

Technical Summary

1. Problem Statement

The paper addresses a gap in existing computational frameworks for decision-making. While models exist for temporal inference (estimating elapsed time) and reward inference (estimating value), few address the regime where reward probability is a latent, time-dependent function that must be inferred without informative sensory evidence between events.

• The Core Challenge: In many natural settings (e.g., predator evasion, social timing), the probability of reward for an action changes systematically over time. Agents must jointly infer two latent variables:
  1. Elapsed time ($t$): Where they are in the interval.
  2. Reward probability ($P(R|A, t)$): The likelihood of reward for a specific action at that time.
• Unresolved Questions:
  • How do agents map dynamically inferred probabilities to choices under temporal uncertainty?
  • Does temporal uncertainty scale with elapsed duration (Weber's Law) or is it modulated by the expected reward structure?
  • Do systematic distortions in probability weighting (common in static risk) apply to dynamically inferred, time-indexed probabilities?

2. Methodology

Experimental Task: Set-Go Paradigm

• Participants: 12 healthy adults.
• Procedure:
  • A Set cue initiates a trial.
  • After a variable Go time ($t$) drawn from a uniform distribution $[0.4s, 1.4s]$, a Go cue appears.
  • Participants must immediately choose between Left (A) or Right (B) buttons.
  • Reward Structure: The probability of reward for Left decreases over time, while the probability for Right increases. They sum to 1 at every time point.
  • Conditions: Four distinct conditions were tested, varying the slope and crossover point of the reward probability curves.
• Optimal Strategy: A step-function policy: switch from Left to Right exactly at the time point where $P(R|Left) = P(R|Right) = 0.5$.

Computational Modeling

The authors formalized choice as an inference problem under dual uncertainty. They tested three main hypotheses regarding how temporal uncertainty and probability mapping interact:

1. Temporal Uncertainty Hypotheses:

   • Temporal Blurring (Weber's Law): Uncertainty ($\sigma$) scales linearly with elapsed time ($\sigma = \phi \cdot t$).
   • Probabilistic Blurring: Uncertainty scales inversely with reward probability (higher reward $\rightarrow$ higher precision), independent of elapsed time.

2. Probability Mapping Hypothesis (DLLO Model):

   • The authors proposed a Dynamic Log-Odds Linear Operator (DLLO) to model the transformation of objective reward probability ($p(t)$) into subjective choice probability ($\pi(p(t))$).
   • Equation: $DLLO(\pi(p(t))) = \gamma \cdot Lo(p(t)) + (1 - \gamma) \cdot Lo(p_0)$
     • $Lo(x) = \log(\frac{x}{1-x})$ (Log-odds transformation).
     • $\gamma$: Slope parameter (governs distortion).
     • $p_0$: Fixed point (crossover parameter).
   • Interpretation:
     • $\gamma = 1$: Veridical mapping (mimicking objective probabilities).
     • $\gamma \to \infty$: Optimal step-function policy (deterministic switching).
     • $1 < \gamma < \infty$: Sigmoidal distortion (systematic bias toward optimality but not fully deterministic).

Analysis

• Fitting: Models were fitted to group and individual choice data using Ordinary Least Squares (OLS) regression.
• Comparison: The authors compared the fit of the DLLO model combined with Temporal Blurring vs. Probabilistic Blurring.

3. Key Results

A. Behavioral Performance

• Participants significantly exceeded chance level ($P(R) \approx 0.70-0.72$ vs. chance $0.5$), indicating they learned the dynamic structure.
• Behavior approximated the optimal step-function strategy but did not achieve it perfectly.
• Systematic Deviations:
  • Crossover Shift: Participants shifted their subjective crossover point toward the mean Go time (contraction bias).
  • Probability Distortion: Choices were not a 1:1 match to objective probabilities. Small probabilities were overestimated, and large probabilities were underestimated, creating a sigmoidal curve relative to the objective reward.

B. The DLLO Model Fit

• The DLLO model provided an excellent fit to the data (Adjusted $R^2 \approx 0.98 - 0.99$).
• Slope Parameter ($\gamma$): Estimated values ranged from 1.71 to 2.20.
  • This indicates a systematic distortion where participants "sharpened" the probability curve (moving toward the optimal step function) but stopped short of a hard switch.
  • Reward Efficiency: The relationship between $\gamma$ and expected reward is non-linear. Small increases in $\gamma$ from 1 yield large reward gains, while further increases yield diminishing returns. Participants operated in a "high-yield" region ($\gamma \approx 1.8$) that maximized reward without requiring infinite temporal precision.
• Crossover Parameter ($p_0$): Varied slightly around 0.5 but had minimal impact on overall reward compared to $\gamma$.

C. Temporal Uncertainty Mechanism

• Rejection of Weber's Law: The "Temporal Blurring" model (where uncertainty scales with time) failed to predict the data, specifically misestimating crossover points and choice dynamics.
• Support for Reward-Contingent Scaling: The Probabilistic Blurring model (where uncertainty is inversely related to reward probability) provided the best account of the data.
  • Finding: Temporal precision was highest when expected reward was high and lowest when expected reward was low. This contradicts the classic assumption that time estimation precision degrades solely as a function of duration.

4. Key Contributions

1. Formalization of Time-Contingent Choice: The paper establishes a framework for decision-making where reward is a latent function of time, requiring joint inference of time and value, distinct from standard evidence accumulation or static reinforcement learning.
2. Discovery of Dynamic Log-Odds Distortion: It identifies that the brain applies a linear transformation in log-odds space to dynamically inferred probabilities, distinct from the static probability weighting seen in Prospect Theory.
3. Reward-Modulated Temporal Precision: It provides strong behavioral evidence that temporal uncertainty is not governed by elapsed duration (Weber's Law) but is adaptively modulated by the behavioral relevance (reward probability) of the time point.
4. Computational Efficiency: The study demonstrates that agents adopt a "good enough" strategy (moderate $\gamma$) that balances the cost of temporal uncertainty against the diminishing returns of perfect optimality.

5. Significance

• Theoretical Impact: This work bridges the gap between interval timing, reinforcement learning, and perceptual decision-making. It suggests that the brain does not treat time and reward as separate modules but integrates them such that reward expectations actively shape the precision of temporal representations.
• Neuroscientific Implications: The findings suggest that neural mechanisms for timing (e.g., dopamine signaling) may be directly coupled to reward prediction errors, sharpening temporal resolution when high-value outcomes are anticipated.
• Practical Application: Understanding these principles is crucial for modeling human behavior in dynamic environments (e.g., autonomous driving, financial trading, social interaction) where timing and reward are inextricably linked and uncertain.

In summary, the paper reveals that human choice in dynamic environments is shaped by two interacting principles: a log-odds transformation that biases inferred probabilities toward optimality, and a reward-contingent temporal precision that allocates cognitive resources to high-value moments in time.