🧬 biology

State-Dependent Observation Noise Reintroduces Epistemic Value in Linear-Gaussian Active Inference

This paper demonstrates that introducing state-dependent observation noise into linear-Gaussian active inference models restores epistemic value by making the posterior covariance and effective Kalman gain action-dependent, thereby re-establishing a drive for information-seeking behavior without requiring multiplicative control dynamics.

Original authors: Daniel Corva

Published 2026-07-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Daniel Corva

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

The Science of Curiosity: Why Getting Lost Can Be the Best Way to Learn

Imagine you are trying to teach a robot how to navigate a dark, foggy maze. In the world of artificial intelligence, there are two main ways to teach a machine to act. The first is the "reward hunter": the robot is told, "Go get the gold coin!" and it will do whatever it takes to grab the prize, ignoring everything else. The second is the "curious explorer": the robot is programmed to want to learn, to reduce its confusion, and to figure out where it is, even if there is no gold coin in sight. This second type of behavior is called Active Inference. It's a theory that suggests intelligent agents don't just chase rewards; they also chase information because being uncertain feels "expensive" to their internal brain.

For a long time, scientists have been testing this idea using simple, predictable worlds called Linear-Gaussian models. Think of these as a perfectly smooth, straight hallway where the robot moves in a straight line, and its sensors give it a slightly blurry but consistent picture of the world. In these simple worlds, a famous problem was discovered: the robot's "curiosity switch" gets stuck in the off position. No matter what the robot does, the amount of new information it expects to gain stays exactly the same. It's like a student who thinks, "Reading this book won't teach me anything new, no matter which page I turn to," so they stop trying to learn and just focus on getting the right answer. This paper asks a simple but profound question: Is there a tiny, realistic change we can make to the robot's world that turns the curiosity switch back on?

The Discovery: When Sensors Get "Range-Dependent"

The authors of this paper, Daniel Corva, found a surprisingly simple way to bring curiosity back to these "boring" linear robots. They discovered that the problem wasn't the robot's goal or its movement; it was how the robot's eyes worked.

In the old, "broken" models, the robot's sensors were assumed to be equally blurry everywhere. Whether the robot was standing right next to a wall or far away in the corner, the camera's fuzziness (called noise) was the same. Because the fuzziness never changed, the robot realized that moving closer to an object didn't make its picture any clearer. It had no reason to move just to learn; it only moved to get the gold coin.

The authors proposed a tiny tweak: what if the sensor's blur depends on where the robot is? Imagine a camera that gets super sharp when you are close to an object but gets very grainy and blurry when you are far away. This is called state-dependent observation noise. In the real world, this is exactly how our senses work: a sign is easy to read when you are close to it, but impossible to read from across the street.

The "Dual Effect": Moving to See Better

When the authors added this "range-dependent" blur to their robot's model, something magical happened. The robot suddenly realized that its actions could change how well it saw the future.

If the robot moved closer to a wall, the noise in its sensor dropped, and its internal map of the world became much sharper. If it stayed far away, the map stayed fuzzy. This created a dual effect: the robot's action (moving) did two things at once. First, it changed where the robot was (the obvious part). Second, and this is the new part, it changed how certain the robot was about its position.

Because the robot could now choose to move to a spot where it would learn more, the "curiosity term" in its brain woke up. The robot started to act like a curious explorer again. It would choose to walk closer to a wall, not just because it wanted to touch the wall, but because walking closer would make its vision clearer and help it understand the maze better.

What This Means for AI

The paper proves mathematically that this tiny change is enough to break the "stuck" behavior. Before this, scientists thought that in simple, straight-line worlds, curiosity was impossible unless you made the robot's movements incredibly complicated or changed its goals. This paper shows that you don't need to change the goals or the movement rules. You just need to admit that sensors get worse the further away you are.

The authors didn't just write equations; they built a software tool called cpomdp to prove it. They created a virtual robot and showed that when they gave it a "smart" sensor (one that gets blurry with distance), the robot started making choices to gather information. When they gave it a "dumb" sensor (one that is always equally blurry), the robot stopped being curious and just followed the path to the reward.

The Takeaway

The main finding is that curiosity in simple AI agents comes from the relationship between where you are and how well you can see. If your ability to learn depends on your location, you will naturally want to move to the best spots to learn. If your ability to learn is the same everywhere, you have no reason to explore.

This paper rules out the idea that simple, linear robots are doomed to be uncurious. It argues that the "linear-Gaussian collapse" (the moment curiosity dies) only happens if we pretend our sensors are perfect and unchanging. Once we admit that sensors degrade with distance, the drive to learn returns. The authors are very sure about this because they provided a mathematical proof and a working computer program that demonstrates it. They show that "acting is attending"—moving your body to pay attention is a natural, built-in feature of any agent that knows its sensors aren't perfect.

Technical Summary: State-Dependent Observation Noise Reintroduces Epistemic Value in Linear-Gaussian Active Inference

Problem Statement
Recent literature (specifically Koudahl, Kouw, and de Vries [1]) established that in linear-Gaussian state-space models under active inference, the epistemic drive—the incentive to act to gain information—is absent "under any circumstances." In these standard models, the epistemic term of the Expected Free Energy (EFE) becomes constant regardless of the policy chosen. Consequently, the agent behaves indistinguishably from a Kalman filter coupled with a Kullback–Leibler (KL) controller: it seeks preferred outcomes but has no intrinsic motivation to reduce uncertainty. The known routes to restoring this drive involve multiplicative control (where actions alter the transition dynamics) or re-expressing the objective via constrained factor graphs. However, the "observation side" of this boundary—specifically, whether modifying the observation model alone could restore epistemic value—remained unexplored.

Methodology and Model
The paper investigates a minimal departure from the standard linear-Gaussian regime: state-dependent observation noise. The authors propose a generative model where:

Dynamics: The state evolves linearly with additive control ( $x_{k+1} = Ax_k + Bu_k + w_k$ ).
Observations: The observation map is linear ( $o_k = Cx_k + v_k$ ), but the observation noise covariance $R$ is a function of the state, $R(x_k)$ .
Inference Rule: The agent employs a first-order Gaussian filter where the noise covariance is evaluated at the predicted mean, $R_k = R(\mu_k^-)$ .

The core mechanism relies on the fact that while additive control shifts the state mean, it does not shift the state covariance in standard models. However, if $R$ depends on the state, the action $u_k$ shifts the predicted mean $\mu_k^-$ , which in turn changes the value of $R$ used in the update. This coupling makes the posterior covariance (and thus the Kalman gain) dependent on the action taken.

The authors define two concepts of "flattening" (reducing the agent to a fixed linear-Gaussian filter):

Fixed Linear-Gaussian Reduction: A policy-independent sequence of noise covariances that reproduces the agent's beliefs for all observation sequences.
Planning Reduction: A policy-independent sequence that reproduces the agent's open-loop planning covariances.

Key Contributions

Non-Reducibility Theorem: The paper proves that coupling state-dependent noise $R(x)$ to a controllable latent mean prevents the agent from admitting any fixed linear-Gaussian reduction. No pre-specified sequence of noise covariances can reproduce the agent's beliefs because the required noise value at any step is uniquely "pinned" by the policy-dependent predicted mean.
Restoration of Epistemic Value: Under mild rank and non-degeneracy conditions (specifically that $R$ is non-constant on the reachable set of predicted means and that this variation is visible to the epistemic functional), the per-step epistemic value is no longer constant across policies. The agent can distinguish between policies based on the information gain they promise.
Bridge to Dual Control: The result is framed as a minimal constructive instance of the Bar-Shalom–Tse dual effect. The paper demonstrates that state-dependent observation noise is a distinct source of the dual effect (where control influences the conditional error covariance), separate from the nonlinear observation maps or multiplicative dynamics usually associated with it. This establishes an equivalence: reintroducing epistemic value in this Gaussian setting is mathematically identical to introducing a second-order dual effect.
Executable Witness ($cpomdp$): The authors provide an open-source library, cpomdp, which implements the model class. The library includes a structural check that raises an IncompatibleLinearizationError if a user attempts to "flatten" a model with state-dependent noise and mean-shifting couplings. This serves as a runtime witness to the theorem, refusing to generate an incorrect fixed Kalman filter.

Results

Theoretical: Theorem 1 establishes that under assumptions of positive definiteness, full observation rank, and reachable non-constancy of $R(x)$ , the posterior covariance and Kalman gain become policy-dependent. Consequently, the epistemic term of the EFE varies with the policy, allowing curiosity to re-emerge.
Scalar Case: For scalar observations, the condition for non-constant epistemic value is unconditional; any state-dependence in $R(x)$ that is reachable by the control is sufficient.
Counter-Example (Example 1): The authors identify a "knife-edge" case in higher dimensions where the posterior covariance varies with the policy (dual effect exists), but the epistemic value (a scalar determinant) remains constant because the policies move the covariance along a level set of the determinant function. This highlights that covariance motion is necessary but not sufficient for epistemic value; the movement must be "visible" to the epistemic functional.
Empirical/Simulation: Using a continuous T-maze task, the authors demonstrate that an agent with state-dependent noise (Agent B) exhibits different behavior than a collapsed agent with fixed noise (Agent A). Agent B actively moves to a location where the sensor is most reliable (the "cue") to resolve uncertainty, whereas Agent A commits to a prior belief without seeking information. The epistemic value for Agent B varies with the action, while it remains flat for Agent A.

Significance and Claims
The paper claims to provide a precise, observation-side characterization of curiosity in Gaussian agents. Its primary significance lies in identifying that curiosity in the linear-Gaussian world is a property of reachable heteroscedasticity, not a modification of the objective function or the dynamics.

"Acting is Attending": The paper formalizes the intuition that acting to improve sensor reliability (e.g., moving closer to read a sign) is the mechanism that restores epistemic drive.
Minimal Departure: It shows that the "collapse" of active inference to KL control is fragile; it breaks with the minimal addition of state-dependent noise, without requiring multiplicative control or complex objective constraints.
Modesty: The authors are explicit about limitations. The theorem applies to finite-dimensional, single-chain models (though the demonstration uses coupled trees). The claim regarding epistemic value requires the stronger condition (H3') that the noise variation is visible to the determinant; while this is automatic for scalar observations, it is not guaranteed for all high-dimensional cases. Furthermore, the inference rule is a first-order approximation (evaluating $R$ at the mean), and the paper does not quantify the distance to the exact non-Gaussian posterior.

In summary, the paper argues that the "epistemic term" in active inference is not inherently broken in linear-Gaussian models but is rendered inert by the assumption of constant observation noise. By allowing the noise covariance to depend on the state, the agent regains the ability to influence the quality of its future estimates, thereby reintroducing the dual effect and the drive for curiosity.