Not all uncertainty is alike: volatility, stochasticity, and exploration

Technical Summary: Not all uncertainty is alike: volatility, stochasticity, and exploration

Problem Statement
Adaptive decision-making in both biological and artificial intelligence requires balancing the exploitation of known outcomes with the exploration of uncertain alternatives. While prior work generally posits that increased uncertainty drives increased exploration (e.g., via Upper Confidence Bound or Thompson Sampling), this view treats all sources of uncertainty as equivalent. This paper addresses environments where uncertainty arises from two distinct sources: volatility (latent reward states that drift over time) and stochasticity (noisy observations of those states). Both sources increase posterior uncertainty, yet standard algorithms fail to distinguish between them, prescribing more exploration in both cases. The core problem is that this uniform prescription is suboptimal: volatility creates opportunities for information gain, whereas stochasticity degrades the quality of information.

Methodology
The authors approach this problem through two primary theoretical lenses:

1. Extension of the Gittins Index Framework:
   The authors extend the classic Gittins index framework, traditionally applied to independent and identically distributed (iid) Gaussian bandits, to Gaussian state-space bandits with latent dynamics. In this setting, each arm has a latent reward state $x_t$ evolving as a Gaussian random walk with innovation variance $v$ (volatility) and is observed through noise with variance $s$ (stochasticity).

   • They formalize the exploration bonus as the value of information derived from the "retirement problem" (an agent choosing between pulling an arm or retiring for a fixed salary).
   • They prove that the optimal exploration bonus is monotonically increasing in volatility ($v$) and monotonically decreasing in stochasticity ($s$). This is established via coupling arguments: a noisier observation channel restricts the set of available stopping rules (reducing value), while higher volatility increases the spread of future posterior means, enhancing the option value of pulling (increasing value).

2. Derivation of CAUSE (Control-as-Inference):
   Recognizing that the Gittins index is optimal only for "rested" bandits (where unselected arms do not evolve) and is intractable for "restless" settings, the authors derive a practical, closed-form policy using the control-as-inference framework.

   • They cast action selection as posterior inference in a probabilistic graphical model, conditioning on future optimality.
   • By employing a sigmoid ansatz for the backward message and using probit approximations for Gaussian-sigmoid convolutions, they derive CAUSE (Cause-Aware Uncertainty-Sensitive Exploration).
   • CAUSE decomposes into an exploitation term (current mean) and an exploration bonus that explicitly depends on both $v$ and $s$, inheriting the opposing monotonicities established by the Gittins analysis.

Key Contributions

1. Theoretical Asymmetry: The paper formally proves that optimal exploration bonuses increase with volatility but decrease with stochasticity. This challenges the standard assumption that "more uncertainty equals more exploration."
2. CAUSE Index: The derivation of CAUSE, the first known closed-form index policy for restless bandits with continuous-state Gaussian dynamics. It is derived via control-as-inference and does not rely on the approximations typical of Whittle's index or dynamic programming on belief-state MDPs.
3. Unified View of Learning and Exploration: The authors demonstrate that the opposing effects of volatility and stochasticity on exploration parallel their known effects on learning rates. Both phenomena stem from the same normative properties of the generative model: volatility makes new observations relevant, while stochasticity makes them unreliable.
4. Lesion Analysis and Pathological Inference: The paper models "lesioned" agents that fail to infer one noise source correctly. It predicts that such failures produce reversed (rather than merely impaired) behavioral patterns. For instance, an agent insensitive to stochasticity will over-explore in high-noise environments (mistaking noise for drift), while an agent insensitive to volatility will under-explore in changing environments.

Experimental Results
The authors evaluate CAUSE against Thompson Sampling, UCB, Predictive Sampling, and a numerical Gittins-per-arm baseline across various regimes:

• Heterogeneous Restless Bandits: In mixed, stochasticity-dominant, and volatility-dominant regimes, CAUSE achieves the lowest cumulative regret. Notably, in the stochasticity-dominant regime, CAUSE and the Gittins-per-arm baseline correctly down-weight exploration of high-noise arms, whereas UCB and Thompson Sampling incur significantly higher regret by treating posterior variance as a single quantity.
• Bonus Structure: CAUSE's closed-form bonus closely tracks the shape of the optimal Gittins bonus across orders of magnitude of stochasticity, whereas UCB remains insensitive to stochasticity.
• Rested Limit Validation: In the limit where volatility is zero ($v=0$), CAUSE's performance is statistically indistinguishable from the numerically computed optimal Gittins index, validating the accuracy of the control-as-inference approximations.
• Lesion Analysis: Simulations confirm that agents with miscalibrated noise priors exhibit the predicted reversed exploration behaviors (e.g., stochasticity-blind agents over-explore as true stochasticity increases).

Significance and Claims
The paper claims to provide a unified normative account of how different sources of uncertainty should drive exploration. Its primary significance lies in:

• Correcting a Fundamental Flaw: It demonstrates that standard exploration strategies are structurally suboptimal in environments with heterogeneous noise structures because they conflate volatility and stochasticity.
• Computational Psychiatry: The framework offers a concrete behavioral target for investigating psychiatric conditions (e.g., anxiety, psychosis, depression). It suggests that pathological behaviors may not just be "impaired" exploration but "reversed" exploration, where the direction of the error depends on which noise inference channel (volatility vs. stochasticity) is miscalibrated.
• Methodological Advance: It establishes control-as-inference as a viable route to deriving tractable, closed-form indices for restless bandit problems, a setting where optimal policies are generally intractable.

The authors remain modest regarding the scope, noting that the closed-form CAUSE index relies on Gaussian conjugacy and specific approximations (probit/sigmoid), and while the directional predictions may generalize, the exact form does not transfer mechanically to non-Gaussian or change-point dynamics.