🤖 machine learning

Learning the Supports for Categorical Critic in Reinforcement Learning

This paper proposes a novel actor-critic reinforcement learning approach that dynamically learns the support bounds for Gaussian Histogram Loss, thereby eliminating the need for pre-defined intervals while providing a tighter theoretical error bound and achieving performance comparable to or better than existing methods on continuous control tasks.

Original authors: Jen-Yen Chang, Takayuki Osa, Tatsuya Harada

Published 2026-07-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jen-Yen Chang, Takayuki Osa, Tatsuya Harada

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to walk, run, or balance. To do this, the robot needs a "scorecard" (called a Value Function) to predict how good a specific move will be in the future.

Traditionally, robots calculate this score as a single number, like a precise temperature reading (e.g., "This move is worth 42.5 points"). However, the future is messy and uncertain. A move might be worth 40 points, or it might be worth 80 points depending on luck.

The Problem: The "Fixed Box" Trap

To handle this uncertainty, some advanced robots use a method called Distributional RL. Instead of guessing one number, they guess a whole range of possibilities.

The paper discusses a specific technique called HL-Gauss. Imagine you have a long, empty ruler (a "support interval") that you use to measure these scores. You divide this ruler into 128 tiny boxes (bins). The robot learns to say, "There is a 10% chance the score falls in Box 1, a 20% chance in Box 2," and so on.

The Catch: In the old way, you had to pre-pick the length of the ruler before the robot started learning.

If the ruler is too short: The robot's future scores might shoot off the end of the ruler. The robot loses that information, like trying to measure a 10-foot pole with a 1-foot ruler. It gets cut off (truncated), and the robot learns the wrong lessons.
If the ruler is too long: To fit a 10-foot pole, you might use a 1-mile long ruler. But if you only have 128 boxes to cover that mile, each box becomes huge. The robot can't tell the difference between a score of 50 and 51 because they both fall in the same giant box. The "resolution" is too blurry.

The big problem is that in real life, we don't know how long the ruler needs to be. The robot's skills change as it learns, so the range of possible scores changes too. A ruler that fits a beginner might be useless for an expert.

The Solution: The "Smart, Stretchy Ruler" (DySEL)

The authors propose a new algorithm called DySEL (Dynamic Support Endpoint Learning). Instead of a fixed ruler, they give the robot a stretchy, self-adjusting ruler.

Here is how it works, using a simple analogy:

Imagine the robot is trying to fit a pile of sand (the possible future scores) into a bucket (the ruler).

The Goal: The robot wants the bucket to be as small as possible so the sand is packed tightly (high resolution), but it must be big enough to hold all the sand without spilling over the sides.
The Conflict:
- If the bucket is too small, sand spills out (Truncation Error).
- If the bucket is too big, the sand is spread out too thin (Low Resolution).
The Game: The authors set up a "tug-of-war" (a min-max game) inside the robot's brain:
- Player A (The Optimizer): Tries to shrink the bucket to make the measurement precise.
- Player B (The Enforcer): Acts like a safety guard. If Player A shrinks the bucket too much and sand starts spilling out, Player B yells "Stop!" and forces the bucket to expand just enough to catch the spill.

This tug-of-war allows the robot to automatically find the perfect size for the ruler at every stage of learning. If the robot is just starting and scores are small, the bucket stays small. As the robot gets better and scores get huge, the bucket stretches to accommodate them, all without the human programmer having to guess the size.

What Did They Find?

The researchers tested this "stretchy ruler" on various robot tasks, like making a virtual cheetah run or a humanoid walk.

It works: The robot with the stretchy ruler performed just as well as the best robots using fixed rulers on most tasks.
It shines in hard cases: On very difficult tasks (like the "humanoid" walking tasks), the stretchy ruler actually performed better. This is because these tasks have wild, unpredictable score ranges that a fixed ruler simply couldn't handle well.
No more guessing: The biggest win is that humans no longer need to spend time guessing the right ruler size for each new robot. The robot figures it out on its own.

In a Nutshell

The paper introduces a way for AI to stop guessing the "range" of its future rewards and start learning the range itself. By turning the problem into a balancing act between "keeping the range tight" and "catching all the data," the robot learns more efficiently and avoids the errors caused by using a ruler that is either too short or too blurry.

Technical Summary: Learning the Supports for Categorical Critic in Reinforcement Learning

Problem Statement

In actor-critic deep reinforcement learning (RL), value functions are traditionally trained as regression tasks minimizing the Mean Squared Error (MSE) against bootstrapped targets. Distributional RL offers an alternative by modeling the full probability distribution of returns. A recent approach, Gaussian Histogram Loss (HL-Gauss), reframes value estimation as a classification problem by encoding scalar Bellman targets as Gaussian-smoothed categorical targets over a pre-defined set of bins.

However, HL-Gauss and similar histogram-based methods suffer from a critical limitation: the requirement to pre-define a fixed support interval $[\nu_{min}, \nu_{max}]$ . In RL, target values are non-stationary and stochastic; the range of returns induced by early exploratory policies often differs significantly from that of converged policies. A fixed interval creates a trade-off:

Too narrow: Leads to boundary truncation, discarding informative extreme returns and introducing truncation bias.
Too broad: Dilutes the representational resolution of fixed bins, weakening the learning signal and introducing quantization bias.

Current literature typically assumes a sufficiently broad, static interval, which is often suboptimal for specific tasks or training phases. There is no single fixed interval that remains optimal across all policies encountered during training.

Methodology: Dynamic Support Endpoint Learning (DySEL)

The authors propose Dynamic Support Endpoint Learning (DySEL), an algorithm that dynamically learns the lower and upper bounds of the support interval rather than assigning them beforehand.

Theoretical Foundation

The core theoretical insight is that the Mean-Squared Bellman Error (MSBE) is upper-bounded by the HL-Gauss loss. The tightness of this upper bound is directly correlated with the absolute values of the support interval. Specifically, the bound includes a term proportional to $\max(|\nu_{min}|, |\nu_{max}|)^2$ (the absolute width) and a distribution-matching term (related to KL divergence).

Tension: Minimizing the upper bound requires a narrow support (to tighten the bound), but minimizing truncation bias requires a wide support (to capture the full probability mass $Z$ ).

Formulation as Constrained Optimization

To resolve this tension, the authors formulate the problem as a constrained optimization task:

Objective: Minimize the support width to maintain high resolution and a tight theoretical bound.
Constraint: Ensure the support interval captures sufficient probability mass of the target distribution (minimizing the "leaked mass" $1-Z$ ) to prevent truncation bias.

The authors transform this into an adversarial min-max game using a Lagrangian multiplier $\lambda$ . The final objective function $L(\theta, \phi, \lambda)$ consists of three components:

Width Penalty: $\alpha \cdot \max(|\nu_{min}|, |\nu_{max}|)$ , penalizing excessive width.
Cross-Entropy Loss: The standard HL-Gauss classification loss.
Mass Constraint: $\lambda \cdot ((1 - Z) - \epsilon)$ , where $Z$ is the covering mass (integral of the truncated Gaussian over the support).

The parameters are updated via:

$\theta, \phi$ (Critic and Support networks): Minimize the Lagrangian.
$\lambda$ (Lagrange multiplier): Maximize the Lagrangian to enforce the coverage constraint.

This approach allows the neural network to automatically expand the support to capture "leaked" mass when necessary, while the width penalty prevents unbounded expansion.

Key Contributions

Theoretical Upper Bound: The authors demonstrate that the MSBE is upper-bounded by the HL-Gauss loss, with the bound's tightness correlated to the support interval's absolute values. This motivates the search for the narrowest feasible support.
Dynamic Support Formulation: They eliminate reliance on pre-defined intervals by formulating dynamic support learning as a constrained optimization problem, balancing bound tightness against distributional coverage.
Adversarial Min-Max Objective: They derive a tractable learning objective using a Lagrangian multiplier to enforce probability mass coverage, enabling the network to adaptively expand or contract supports.
Empirical Validation: The method is evaluated on continuous control tasks, showing it is competitive with fixed-support HL-Gauss baselines and superior on specific tasks (notably humanoid tasks) without requiring manual interval tuning.
Evidence of Adaptation: The paper provides evidence that a dynamically adapted support outperforms any single fixed support, as no single fixed interval is optimal across all training policies.

Experimental Results

The authors evaluated TD3+DySEL against TD3 and TD3+HL-Gauss (with a fixed support of $[-100, 100]$ ) on 11 tasks from the DeepMind Control Suite.

Performance: TD3+DySEL was competitive with TD3+HL-Gauss on most tasks and demonstrated substantial performance gains on humanoid tasks (e.g., humanoid-run, humanoid-stand). The authors attribute this to the finer granularity provided by the dynamically learned intervals, allowing the critic to discern subtle return variations obscured by fixed supports.
Support Evolution: The learned support intervals exhibited two behaviors:
- Stable: Some tasks (e.g., finger-turn_hard) converged to a stable interval quickly.
- Adaptive: Others (e.g., cheetah-run, humanoid-run) showed gradual expansion as training progressed.
Ablation Studies: Removing the width penalty caused the support to diverge to infinity (exploiting the cross-entropy loss), while removing the mass constraint led to insufficient coverage for tasks requiring expansion. Both components were deemed critical for stability.
Comparison to Fixed Baselines: Using the final learned support interval from DySEL as a fixed input for vanilla HL-Gauss improved performance on some tasks but failed to match DySEL on tasks requiring dynamic expansion, confirming the necessity of the adaptation mechanism.

Significance and Claims

The paper claims that DySEL addresses a fundamental limitation of classification-based value learning in RL: the reliance on static, hand-picked support intervals. By theoretically linking the MSBE to the support width and formulating a constrained optimization problem, the authors provide a method that:

Automates Hyperparameter Tuning: Removes the need for practitioners to manually specify or schedule support intervals.
Improves Stability and Performance: Achieves stable adaptation of the support interval, matching or exceeding fixed-support baselines, particularly in complex, high-dimensional environments like humanoid control.
Theoretical Justification: Offers a theoretical framework showing that dynamic adaptation yields a tighter bound on the Bellman error than fixed supports.

The authors acknowledge limitations, noting that the saddle-point optimization is sensitive to the dual learning rate and that the learned supports are currently symmetric. They suggest future work could explore asymmetric supports, adaptive Gaussian $\sigma$ , and broader benchmarks beyond DM-Control.