📊 statistics

Auditing the Risk Claims of Distributional Reinforcement Learning

This paper presents an audit framework demonstrating that the risk-sensitive claims of trained distributional reinforcement learning agents are largely structural training artifacts rather than accurate reflections of environmental stochasticity, rendering their risk advice uninformative and often detrimental to decision-making.

Original authors: Hari Prasad

Published 2026-07-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hari Prasad

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you've built a super-smart robot gamer that doesn't just guess how many points it will get, but actually predicts the entire range of possible futures. It's like a weather forecaster that doesn't just say "it might rain," but draws a detailed map showing exactly where the rain will fall, how hard it will hit, and where the sun might peek through. This robot, called a Distributional Reinforcement Learning agent, is supposed to be a risk-taker's best friend. It's designed to spot when one move is a safe, boring bet and another is a wild gamble, even if they promise the same average score.

But here's the twist: The robot's "risk map" is mostly a hallucination.

The Great Audit

The authors of this paper decided to play detective. They asked a simple question: When this robot claims, "Hey, this move is risky and that one is safe," is it actually telling the truth?

To find out, they didn't just trust the robot's word. They built a "truth machine." They took snapshots of the game world, froze time, and then replayed the same moment thousands of times with different random seeds (like rolling dice over and over) to see what actually happens. They compared the robot's fancy predictions against this mountain of real-world data.

The Shocking Result: 40% to 95% of the "Risk" is Fake

The audit revealed a massive problem. In the games they tested (MinAtar, a mini-version of the classic arcade games), 40% to 95% of the robot's strongest claims about risk were completely false.

Think of it like this: The robot is standing in a hallway, pointing at two doors. It screams, "Door A is safe! Door B is a trap!" But when the auditors kicked the doors open and ran through them 2,000 times, they found that both doors led to the exact same room. The robot wasn't seeing a difference; it was just making one up.

The "Top" Claims are the Worst: The more confident the robot was about a risk difference, the more likely it was to be lying. In the top 2% of its "riskiest" moments, up to 95% of the claims were refuted.
It's Not Just a Glitch: This isn't a bug that happens when the robot is tired or learning. The authors checked the robot at different stages of its life. The fake risk claims were fully formed after just 500,000 steps of training and stayed that way even as the robot got twice as good at the game.
It's Not the Game's Fault: The robot wasn't confused by a tricky game. When the authors tested it on a special, custom-made game where the risks were real and clearly defined (a "RISKYGRID"), the robot got it right 96–100% of the time. This proves the robot can see risk, but in the real arcade games, it's seeing ghosts.

Why Does This Happen?

The paper rules out a few things that people might guess:

It's not because the robot is "bad" at the game. Even a near-perfect, pre-trained robot (trained on 10 million frames of data) fell for the same trap.
It's not because the robot is "miscalibrated" (like a thermometer that's always 5 degrees off). If you tried to "fix" the robot by recalibrating it, the only way to make it pass the test was to make it say nothing at all. The robot isn't just slightly wrong; it's uninformative. It's like a compass that spins wildly; you can't just "adjust" it to point North because it has no magnetic sense to begin with.
It's not a lack of data. The robot saw thousands of examples, but the "risk" it learned was a structural artifact of how it was trained, not a feature of the game itself.

The Danger: Acting on the Lies

Here is the scary part. If you were a human player using this robot's advice to make decisions, you'd be in trouble.

In one game (Breakout), the robot's advice was actually helpful.
In another (Seaquest), following its advice made you three times worse than just ignoring risk entirely.
In a third (Asterix), it was basically a coin flip.

The worst part? There is no way to tell which game is which. The robot gives you a "risk score," but that score tells you nothing about whether the advice is a lifesaver or a death sentence. It's like having a weather app that sometimes predicts a tornado and sometimes predicts a sunny day, but the app gives you no clue which prediction is real.

The Bottom Line

The paper concludes that for the robots they tested, the "risk" they see is a training artifact—a ghost in the machine. It's a pattern the robot learned to draw because of its math, not because the world is actually dangerous in that way.

The authors are very clear: Do not trust the robot's risk maps at face value. If you want to use these agents for safety-critical tasks (like self-driving cars), you need to audit them first. Until then, the robot's "risk" is just a very convincing, very confident lie.

Technical Summary: Auditing the Risk Claims of Distributional Reinforcement Learning

Problem Statement

Distributional Reinforcement Learning (RL) replaces scalar value functions with full return distributions ( $Z^\pi(s, a)$ ). While these distributions have driven state-of-the-art performance and are increasingly used for interpretability, risk-sensitive control (e.g., via Conditional Value-at-Risk, CVaR), and safety monitoring, they are often read at "face value." The core assumption is that if a learned distribution for one action differs in shape from another (e.g., promising a safe small return versus a gamble of equal mean), the environment actually contains that specific risk structure.

However, theoretical work suggests these distributions need not coincide with the true return distribution (e.g., Quantile Temporal-Difference learning converges to non-unique fixed points). Furthermore, distributional methods often improve performance even when only the mean is utilized, suggesting the learned distributions may be useful representations without being factually accurate. The critical gap identified is the lack of direct measurement: Are the specific risk claims made by a trained distributional agent true? Aggregate divergence metrics fail to distinguish between errors that matter for decision-making and those that do not. This paper addresses the question of whether the "risk" reflected in a trained agent's head is a genuine environmental feature or a training artifact.

Methodology

The authors propose a decision-relevant audit framework combining a screening metric, ground truth generation, and a statistical harness to avoid false conclusions.

1. Decision-Relevant Audit Metric

The audit focuses on the excess Wasserstein gap ( $\Delta_x(s)$ ) between the top two actions at a given state $s$ , ranked by their mean returns.
$\Delta_x(s) = W_1(Z(s, a_1), Z(s, a_2)) - |\mu_1(s) - \mu_2(s)|$
Where $W_1$ is the 1-Wasserstein distance between the quantile representations of the two actions, and $\mu$ represents their mean returns.

Significance: $\Delta_x(s) > 0$ occurs if and only if neither action first-order stochastically dominates the other. This is the precise condition under which a risk-sensitive policy would deviate from a mean-greedy policy. Thus, $\Delta_x$ isolates claims that carry decision content.

2. Ground Truth via Snapshot-Restart Monte Carlo

To verify claims, the authors generate ground truth by:

Collecting unique states from the trained agent.
Storing deep copies (snapshots) of the environment at these states.
Performing 2,000 rollouts per action from each snapshot, forcing the specific action and following the greedy policy to termination.
Crucially, each restarted copy is independently reseeded to ensure the rollouts capture true environmental stochasticity (aleatoric uncertainty) rather than replaying identical futures.

3. Statistical Harness

To prevent the audit from manufacturing false conclusions, the authors employ:

Permutation Nulls: Establishing a baseline for estimation noise by splitting a single action's rollouts.
Bootstrap Refutation: A claim is refuted if the learned $\Delta_x(s)$ exceeds the 97.5th percentile of the Monte Carlo (MC) excess distribution.
FDR Control: Using the Benjamini-Hochberg procedure to control the False Discovery Rate across states.
Aliasing Index: A metric to flag states where partial observability might hide the true environment state.

4. Addressing Silent Pitfalls

The paper documents two specific failure modes in restart-based evaluations that previously produced convincing but wrong audits:

P1 (RNG Cloning): Using deepcopy on environments often clones the internal random number generator, causing "random" rollouts to replay identical futures. This creates a false sense of determinism.
P2 (Empirical Wasserstein Noise Floor): For finite samples from the same distribution, the empirical Wasserstein distance converges to a non-zero value ( $O(n^{-1/2})$ ). Naive thresholding can "confirm" pure noise as a risk trade-off. The audit calibrates against per-state nulls to account for this.

Key Results

1. Strongest Claims Are Mostly False

Across QR-DQN, C51, and IQN on MinAtar (Breakout, Seaquest, Asterix) and a pretrained SB3-zoo QR-DQN on full Atari:

Refutation Rates: In the top 2% of the agent's own risk rankings (the states a practitioner would flag first), 40–95% of claimed risk trade-offs are refuted at 95% confidence.
Zero Confirmations: Across 735 audited states in MinAtar, essentially no claim is confirmable after FDR correction.
Truth-Blind Indistinguishability: The placement of the strongest claims is statistically indistinguishable from a "truth-blind" head (where claim magnitudes are preserved but randomly assigned to states). The correlation between learned excess and true excess is near zero ( $r \approx 0.02$ ).

2. Structural Nature of the Artifact

Early Formation: The overclaiming artifact is fully formed by 500k training steps and persists even as the agent's score doubles.
Seed Idiosyncrasy: The specific false claims are unique to each random seed and uncorrelated with final performance.
Doomed States: In states where the ball is irrecoverably lost (every future yields the same return), the agent's head still reports wide, crossing distributions with high $\Delta_x$ .

3. Decision-Level Impact

Acting on the agent's CVaR advice at its most-flagged states yields mixed and unpredictable results:

Breakout: The advice is genuinely informative (81% safer pick vs. 44% for mean-greedy).
Seaquest: The advice is anti-predictive, choosing the truly safer action only 24–34% of the time (worse than chance), tripling regret compared to ignoring risk.
Asterix: The advice is effectively a coin flip.
No observable signal exists to tell a practitioner which regime they are in.

4. Failure of Repairs

Standard remedies do not fix the issue:

Risk-Sensitive Training: Training an agent to be CVaR-greedy results in the same overclaiming signature.
Ensembling: Averaging quantile functions attenuates the artifact but does not calibrate the top claims.
Recalibration: Recalibration passes the audit only by nullifying the claims (collapsing them to noise), proving the head is uninformative, not merely miscalibrated.

5. Positive Controls

To ensure the audit itself is not broken, the authors tested on RISKYGRID, an environment with known risk trade-offs.

The audit correctly confirmed 96–100% of real claims with high correlation ( $r \approx 0.92$ ).
This confirms the audit measures the agents' inability to capture risk, not a failure of the audit methodology or the small effect sizes in MinAtar.

Significance and Claims

The paper concludes that for the agents tested (QR-DQN, C51, IQN), the learned "risk" reflects a structural training artifact rather than environment stochasticity.

The "Winner's Curse" of Risk: The states where the agent is most confident about a risk trade-off (highest $\Delta_x$ ) are precisely where the claims are least reliable.
Uninformative Heads: The distributional heads are not just miscalibrated; they are uninformative regarding risk structure. They conflate aleatoric and epistemic uncertainty in a way that produces spurious risk signals.
Implications for Practice:
1. Interpretability: Visualized return distributions should not be trusted without ground-truth validation.
2. Safety & Control: Risk-sensitive policies derived from raw quantile heads may act on spurious risk at critical states, potentially leading to worse outcomes than mean-greedy baselines.
3. Future Work: Any method claiming to estimate uncertainty or risk must be evaluated against a decision-level audit. The authors release a toolkit and a standard: an audit-passing head must confirm its top claims.

The authors emphasize modesty: their claims cover specific algorithms and environments (MinAtar, specific Atari seeds). They acknowledge that other calibrated variants or agents trained under different conditions (e.g., sticky actions at full scale) remain to be tested, but the current evidence suggests the "risk" in standard distributional RL is largely an artifact of the learning process.