Technical Summary: Behavior-Consistent Deep Reinforcement Learning

Problem Statement

Reinforcement learning (RL) algorithms frequently exhibit high variance across independent training runs. While final performance metrics (returns) may appear similar, the underlying policies often diverge significantly in their behavioral distributions. This lack of consistency poses challenges for scientific evaluation, which requires averaging over numerous seeds, and for practical deployment, where iterative processes like reward design, debugging, and safety validation become unreliable if identical conditions yield qualitatively different behaviors. Existing work has largely treated this variability as noise to be averaged out, rather than a property to be explicitly controlled.

Methodology

Theoretical Foundation

The authors formalize Behavior-Consistent RL (BRL), an objective to learn policies that are both high-performing and distributionally similar across independent runs. The core theoretical insight relies on Maximum Entropy (MaxEnt) RL. By regularizing policies toward a common prior (specifically, a uniform prior in MaxEnt), the policy space is constrained.

The paper establishes a theoretical link between policy divergence and Q-function disagreement:

1. Theorem 4.1: For Boltzmann policies, if the temperature parameter $\alpha(s)$ is set proportional to the pointwise disagreement between two Q-value vectors ($\|Q^{(1)} - Q^{(2)}\|_\infty$), the Kullback-Leibler (KL) divergence between the induced policies is bounded. Specifically, setting $\alpha(s) = \max(\alpha_{min}, \frac{\|Q^{(1)} - Q^{(2)}\|_\infty}{\kappa})$ ensures $D_{KL}(\pi^{(1)} \|\pi^{(2)}) \leq 2\kappa$.
2. Theorem 4.2: Using this disagreement-scaled temperature in soft value iteration allows coupled runs to converge to a neighborhood of the optimal unregularized Q-function, with the error bound controlled by the temperature floor $\alpha_{min}$ and the initial disagreement.

Practical Algorithm: Q-value Expectile Disagreement (QED)

Directly accessing Q-functions from other independent runs is impractical. To address this, the authors propose Q-value Expectile Disagreement (QED), which uses a single-run proxy for cross-run disagreement.

• Double-Critic Proxy: QED leverages the "double Q" trick, utilizing two neural network critics ($Q_{\theta_1}, Q_{\theta_2}$) trained in parallel. The authors hypothesize that the disagreement between these two critics within a single run serves as a representative proxy for the disagreement that would exist between two independent runs.
• Expectile Aggregation: To avoid the adversarial nature of maximizing over all actions (which could highlight extrapolation errors on out-of-distribution actions), QED computes an upper expectile of the absolute differences between the two critics over actions sampled from the current policy.
• Adaptive Temperature: The temperature $\alpha_{QED}(s)$ is dynamically adjusted based on this expectile disagreement:
  $$\alpha_{QED}(s) = \text{clip}\left( \frac{e\Delta(s)}{k \cdot d}, \alpha_{min}, \alpha_{max} \right)$$
  where $e\Delta(s)$ is the expectile disagreement, $k$ is a hyperparameter controlling the trade-off, and $d$ is the action dimension.
• Handling Off-Policy Instability: The paper notes that naively increasing entropy can amplify off-policy extrapolation errors, leading to training collapse. To mitigate this, QED is evaluated on top of algorithms specifically designed to handle distribution shifts: SAC-LN (Soft Actor-Critic with Layer Normalization) and MAD-SAC (Model-Augmented Data SAC).

Key Contributions

1. Formalization of BRL: Introduced the setting of Behavior-Consistent Reinforcement Learning, framing behavioral consistency as a core objective alongside return maximization.
2. Theoretical Connection: Proved a relationship between policy divergence and critic disagreement in MaxEnt RL, showing that scaling temperature by disagreement bounds the KL divergence between policies.
3. Identification of Off-Policy Challenges: Illustrated how high-entropy MaxEnt RL can exacerbate off-policy extrapolation errors, leading to functional policy collapse if not managed with stable base algorithms.
4. QED Algorithm: Proposed QED, a practical method for adaptive entropy tuning that uses single-run critic disagreement to induce cross-run behavioral consistency.

Experimental Results

The authors evaluated QED on 18 continuous-control tasks from the dm_control suite using 10 random seeds per task.

• Reduction in Divergence: QED reduced pairwise policy divergence (measured by symmetric KL) by two orders of magnitude compared to standard entropy tuning (e.g., SAC with target entropy).
• Performance Preservation: In most cases, QED maintained competitive normalized returns. For very low KL divergences, a minor reduction in return (at most 5%) was observed, attributed to a temporary loss in sample efficiency as the algorithm explores longer to reduce Q-function uncertainty.
• Variance Reduction: On the MAD-SAC baseline, QED reduced the variance of returns (measured by the width of 95% bootstrap confidence intervals) by approximately 50%.
• Behavioral Consistency: Qualitative visualizations (e.g., cheetah_run) and quantitative metrics (cumulative L2 action distance) confirmed that QED produces policies with visibly and statistically similar behaviors across runs, whereas standard training yields distinct behavioral modes (e.g., different hopping or walking patterns).
• High-Variance Tasks: On the notoriously unstable hopper_hop task, QED not only improved performance but aligned all training runs to converge to an identical performance level, whereas the baseline exhibited high variance.

Significance and Claims

The paper argues that controlling behavioral consistency is both feasible and beneficial for the stability and reproducibility of RL systems. By treating variability as an explicit property to be controlled rather than noise to be averaged, the authors suggest that behavioral consistency is a missing ingredient for scalable and reliable RL.

The significance lies in enabling more reliable:

• Scientific Evaluation: Reducing the need for massive averaging over seeds to detect true algorithmic improvements.
• Deployment: Ensuring that iterative processes like reward tuning and safety validation are not confounded by stochastic idiosyncrasies.
• Debugging: Making it easier to distinguish between algorithmic failures and random variance.

The authors modestly note limitations, particularly that the method relies on pointwise Q-estimates which may take time to converge, and that a trade-off between performance and similarity becomes more pronounced in high-dimensional tasks. They conclude that while QED offers a principled approach to stability, future work is needed to address long-horizon consistency and the balance between diversity and uniformity in valid behavior strategies.