🤖 machine learning

Performance Variation in Deep Reinforcement Learning

This paper addresses the challenge of low run-to-run robustness in deep reinforcement learning by critiquing conventional uncertainty estimates and proposing percentile-based tools (min-max IPR and run-wise percentile highlighting) to better characterize performance variation, which are then used to demonstrate how specific architectural choices and algorithmic designs differentially impact stability across PPO, SAC, TD-MPC, and DQN variants.

Original authors: Haruto Tanaka, A. Rupam Mahmood

Published 2026-06-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Haruto Tanaka, A. Rupam Mahmood

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a coach training a team of athletes (the AI algorithms) to run a specific obstacle course (the video game or robot task). You tell them, "Run this course 100 times, and I'll tell you how good you are."

In the world of Deep Reinforcement Learning (RL), there's a frustrating problem: even if you give the exact same instructions to 100 identical athletes, their results can be wildly different. One might run a perfect race, while another trips over their own feet. This is called performance variation.

This paper argues that scientists have been using the wrong tools to measure this inconsistency, and it proposes two new, simpler tools to see what's really happening.

The Problem: The "Average" Lie

For a long time, researchers have tried to measure how consistent an algorithm is by looking at the average result and drawing a "shaded band" around it (like a margin of error).

The authors say this is like looking at a weather forecast that says, "The average temperature this week is 70°F," with a tiny shaded band. It sounds nice, but it hides the fact that Monday was a freezing 30°F and Friday was a scorching 100°F.

The Flaw: These "shaded bands" (statistical uncertainty) shrink as you run more experiments, making the algorithm look more consistent than it actually is. They also fail to capture the "outliers"—the runs where the AI completely fails.
The Analogy: Imagine a dartboard. If you throw 100 darts, and 90 hit the bullseye but 10 hit the ceiling, the "average" might look great. But if you are a safety inspector, you care about those 10 darts hitting the ceiling. The old methods ignore the ceiling hits.

The Solution: Two New Tools

The authors propose two new ways to visualize and measure this chaos:

1. The "Min-Max IPR-90" (The 90% Safety Net)

Instead of looking at the average, this tool looks at the middle 90% of the results.

How it works: Imagine you line up all 100 athletes from worst to best. You chop off the bottom 5% (the total disasters) and the top 5% (the lucky miracles). You then measure the distance between the worst of the "good" runners and the best of the "good" runners.
Why it's better: This number tells you the "spread" of the performance. A small number means the athletes are all running at a similar speed. A huge number means the team is unpredictable. It's a simple percentage that doesn't get fooled by weird math tricks.

2. RPH (Run-Wise Percentile Highlighting) (The "Highlight Reel")

Instead of drawing a messy cloud of lines or a shaded band, this method highlights just three specific lines on a graph:

The 5th percentile (the "worst" of the reliable runs).
The 50th percentile (the "median" or typical run).
The 95th percentile (the "best" of the reliable runs).
The Visual: Imagine a race track. Instead of showing every single runner's path as a blurry gray smudge, you highlight the slowest reliable runner, the average runner, and the fastest reliable runner in bright colors. You instantly see how wide the gap is between them. If the gap is huge, the algorithm is unstable.

What They Discovered (The Case Studies)

The authors tested these new tools on three different scenarios to see what they revealed:

1. The "Normalization" Experiment (Fixing the Engine)
They tried adding "LayerNorm" (a technique to stabilize the AI's internal math) to two popular algorithms: PPO and SAC.

The Result: For PPO, the new tools showed that adding these stabilizers made the runners much more consistent (the gap between the 5th and 95th percentile shrank). For SAC, the tools showed that the stabilizers did almost nothing; the runners were still all over the place.
The Takeaway: What works for one engine doesn't necessarily fix the other.

2. The "Superstar" Comparison (PPO, SAC, TD-MPC, TD-MPC2)
They compared four different algorithms on 48 different robot tasks.

The Result: One algorithm, TD-MPC, was the clear winner. It didn't just win the race (get high scores); it was also the most consistent. Its "spread" was tiny. The others were either slower or very unpredictable.
The Takeaway: Even the best modern AI still has a high "failure rate" (about 35% of the time, the top 5% of runs still failed to reach a decent score), but TD-MPC was the most reliable of the bunch.

3. The "Atari" Showdown (DQN vs. Rainbow)
They compared two classic algorithms playing old-school video games (like BattleZone and Qbert*).

The Result: Rainbow was much better at winning than DQN. However, the new tools revealed a surprise: both algorithms were just as "jittery" and unpredictable as each other. Rainbow just happened to be a "jittery" winner, while DQN was a "jittery" loser.
The Takeaway: Winning more often doesn't mean you are more stable.

The Bottom Line

The paper concludes that we need to stop hiding behind "average" numbers and "shaded bands" that make AI look more stable than it is. By using the Min-Max IPR-90 (to measure the spread) and RPH (to visualize the worst/best reliable runs), researchers can finally see the true "brittleness" of their AI.

It's like moving from a weather report that only gives an average temperature to a report that clearly shows you: "It might be a perfect day, or it might be a disaster. Here is the range of what to expect."

Technical Summary: Performance Variation in Deep Reinforcement Learning

Problem Statement

Deep Reinforcement Learning (RL) algorithms are known for their ability to learn complex behaviors but are notorious for "lottery-like" performance sensitivity across independent runs of identically configured agents. This run-to-run performance variation hinders scientific rigor by complicating result reproduction, fair algorithm comparison, and hyperparameter tuning. Furthermore, in practical applications, such variability poses safety risks, as ineffective behaviors can occur unpredictably.

Despite the importance of this issue, empirical deep RL studies often overlook performance variation in favor of aggregated performance metrics. Current evaluation practices rely heavily on:

Uncertainty estimates (e.g., bootstrapped confidence intervals of the Interquartile Mean, standard error), which measure the discrepancy between sample statistics and true population parameters rather than the spread of the performance distribution itself. These metrics vanish as the number of runs increases ( $O(1/\sqrt{n})$ ), failing to capture inherent data variability.
Standard deviation, which assumes a Gaussian distribution. However, deep RL performance distributions are frequently non-Gaussian (skewed or bimodal), leading to the "underreporting" of risk where standard deviation fails to cover significant portions of the distribution, such as failed runs or heavy tails.

Existing rigorous methods like Tolerance Intervals (TIs) or Performance Profiles are often computationally costly or difficult to interpret for single-task analysis. There is a need for evaluation tools that accurately capture single-task performance variation while remaining interpretable and scalable.

Methodology

The authors propose a suite of quantification and visualization methods designed to balance accuracy and interpretability:

1. Min-Max IPR-90 (Quantification)

The paper introduces the Min-Max Inter-Percentile Range (IPR) from the 5th to the 95th percentiles, normalized by the theoretical or empirical min-max range of the task.

Definition: $IPR = \frac{U(95) - U(5)}{M_T - m_T} \times 100\%$ , where $U(x)$ is the $x$ -th percentile of the performance set, and $M_T, m_T$ are the task's maximum and minimum possible returns.
Rationale: Unlike standard deviation, percentiles do not assume a specific data distribution. The 5th–95th range (90% coverage) is chosen to assess risk (capturing the spread of most runs) while avoiding the extreme sensitivity to outliers that the full range (0–100%) would introduce.
Normalization: Min-max normalization allows for direct comparison of variation across different algorithms and tasks with different reward scales.

2. Run-wise Percentile Highlighting (RPH) (Visualization)

To visualize learning curves without the misrepresentation risks of time-wise aggregation (e.g., mean $\pm$ standard deviation bands), the authors propose Run-wise Percentile Highlighting.

Technique: Instead of aggregating statistics at each timestep, RPH highlights the specific learning curves corresponding to the 5th, 50th (median), and 95th percentiles of the final performance.
Implementation: The 5th and 95th percentile curves are highlighted (e.g., with dotted lines) to emphasize the variation envelope, while the 50th percentile represents the central tendency. Other curves are rendered with transparency.
Advantage: This preserves the shape of individual learning curves and avoids the "underreporting" of variation common in shaded error bands. It is computationally simpler and more interpretable than Functional Boxplots (FB).

Key Contributions and Case Studies

The paper validates these methods through three case studies using 100 independent runs per task:

Case Study 1: Effect of Normalization on PPO and SAC

The authors analyzed the impact of LayerNorm and penultimate-layer normalization (PNorm) on PPO and SAC.

Findings: In PPO, applying these normalization techniques (individually or jointly) significantly narrowed the performance variation (reduced Min-Max IPR-90) and often improved median performance. In contrast, for SAC, these modifications generally left performance variation unchanged or increased it.
Implication: Normalization benefits are algorithm-specific; what stabilizes PPO does not necessarily stabilize SAC.

Case Study 2: Comparison of Continuous Control Algorithms (PPO, SAC, TD-MPC, TD-MPC2)

A systematic comparison was conducted across 48 DeepMind Control Suite (DMC) tasks.

Findings: TD-MPC and TD-MPC2 demonstrated superior performance across all metrics (mean, median, 5th/95th percentiles) compared to PPO and SAC. Crucially, TD-MPC exhibited the lowest performance variation (average Min-Max IPR-90 $\approx$ 10%) while maintaining high data efficiency.
Observation: Despite TD-MPC's low variation, an average IPR-90 of 10% remains significantly higher than the $\approx$ 2% observed in supervised learning baselines, indicating that deep RL still struggles with consistent low-variation performance.

Case Study 3: Comparison of Discrete Control Algorithms (DQN vs. Rainbow)

The study compared DQN and Rainbow on five Atari environments (ALE).

Findings: Rainbow significantly outperformed DQN in terms of median and percentile performance. However, both algorithms exhibited similar, high levels of performance variation (Min-Max IPR-90 $\approx$ 14–23%).
Implication: While Rainbow improves the aggregate performance over DQN, it does not resolve the underlying issue of high run-to-run variability.

Significance and Claims

The paper claims that Min-Max IPR-90 and RPH provide a necessary shift in how deep RL performance is evaluated:

Misalignment Correction: They correct the misuse of uncertainty estimates (like confidence intervals) as proxies for variation.
Robustness: By relying on percentiles rather than parametric assumptions (like Gaussianity), these methods provide a more accurate summary of performance spread, particularly for non-Gaussian distributions common in RL.
Interpretability: RPH offers a clear visual representation of run-to-run behavior that is easier to interpret than Functional Boxplots and more informative than standard error bands.
Practical Insight: The proposed tools reveal that algorithmic improvements (e.g., in TD-MPC or Rainbow) do not automatically equate to reduced variation, highlighting that "robustness" remains a distinct and challenging objective in deep RL research.

The authors conclude that while these methods are effective, they should be used alongside other statistical tools to provide complementary insights, as relying on a single statistic can still be misleading. The work aims to aid future research by providing a more rigorous framework for assessing the reliability and robustness of deep RL agents.