Uncertainty-Aware and Temporally Regulated Expert Advice in Reinforcement Learning for Autonomous Driving

Technical Summary: Uncertainty-Aware and Temporally Regulated Expert Advice in Reinforcement Learning for Autonomous Driving

Problem Statement
End-to-end reinforcement learning (RL) offers a promising paradigm for autonomous driving (AD) by enabling agents to adapt to complex, rare, and occluded scenarios through direct environmental interaction. However, the inherent exploration required for RL to learn effective policies poses significant safety risks, as agents must experience novel behaviors that can lead to collisions or off-road driving. Furthermore, extensive exploration results in poor sample efficiency. Existing safety mechanisms, such as Constrained Markov Decision Processes (CMDPs) or Lyapunov functions, often rely on handcrafted risk definitions and complex optimization without actively guiding policies toward safer actions. While expert demonstrations can accelerate learning, static pre-training leaves agents vulnerable to unsafe exploration, and human-in-the-loop systems lack scalability. Current methods that query expert advice based on state uncertainty often neglect the risk induced by the agent's specific actions and lack mechanisms to regulate the frequency and duration of guidance, leading to either over-reliance or insufficient correction.

Methodology
The authors propose an uncertainty-aware framework that integrates expert advice into an off-policy Implicit Quantile Network (IQN) backbone. The core of the approach involves three interconnected components:

1. Uncertainty Estimation: The framework distinguishes between two types of uncertainty to trigger advice:

   • Epistemic Uncertainty (Limited Knowledge): Measured using an ensemble of IQN heads trained on different subsets of the replay buffer. The authors propose two formulations: one based on the Wasserstein distance ($W_1$) between return distributions across ensemble heads, and a risk-sensitive formulation based on the variance of Conditional Value at Risk (CVaR) estimates across heads.
   • Aleatoric Uncertainty (Environmental Risk): Derived from the spread of the predicted return distribution for the specific action the agent intends to execute. This captures inherent environmental randomness, such as sensor noise or occlusions.

2. Adaptive Query Mechanism: Instead of using fixed thresholds, the system employs adaptive thresholds derived from rolling buffers of past uncertainty values. Advice is triggered when either epistemic or aleatoric uncertainty exceeds the empirical $\beta$-percentile of its respective buffer. This ensures the triggering mechanism evolves with the agent's confidence and training dynamics.

3. Commitment–Cooldown Strategy: To prevent over-reliance and ensure the agent internalizes coherent maneuvers, the framework regulates advice duration. Once triggered, the agent follows the expert policy for a "commitment" period rather than a single step. This is followed by a "cooldown" period where the agent operates independently. Additionally, a stochastic early-stop heuristic allows the agent to disengage from the expert if statistical evidence suggests its own policy yields higher safety-adjusted returns, preventing unnecessary dependence.

4. Integration: Expert and agent experiences are stored in a shared replay buffer. Unlike approaches that separate buffers or force high rewards on expert actions, this method treats expert transitions as standard samples evaluated under the same reward function, allowing the policy to naturally surpass the expert as it gains competence.

Key Contributions

• Uncertainty-Aware Guidance: A framework where the agent requests advice based on high epistemic or aleatoric uncertainty, utilizing a shared replay buffer for efficient reuse of expert trajectories.
• Dual Uncertainty Estimation: A method that derives aleatoric uncertainty from return variance and epistemic uncertainty via bootstrapped ensembles, explicitly accounting for action-related risk.
• Temporal Regulation: A commitment–cooldown strategy with a stochastic early-stop heuristic that balances effective guidance with the prevention of long-term dependence on the expert.
• Risk-Sensitive Integration: The use of IQN to model the full return distribution, enabling risk-sensitive decision-making and the detection of both risky and underexplored states.

Experimental Results
Experiments were conducted in the CARLA simulator using unsignalized intersection navigation tasks with randomized traffic densities. The proposed method was benchmarked against a standard IQN baseline and various ablation configurations.

• Performance Gains: The best configuration (CVaR-based uncertainty, 50% expert budget, and 5-step commitment/cooldown periods) outperformed the IQN baseline, improving the success rate by 5–7% and reducing failure rates across all traffic densities.
• Ablation Insights:
  • Commitment/Cooldown: Temporal consistency (e.g., 5-step commitment) significantly stabilized policy learning compared to single-step interventions.
  • Expert Budget: A 50% budget provided the optimal balance; higher budgets (75%) led to over-reliance and degraded generalization.
  • Uncertainty Formulation: The CVaR-based epistemic uncertainty formulation consistently outperformed the Wasserstein-based approach by better filtering distributional noise and emphasizing risk-sensitive quantiles.
  • Runtime Safety: Using aleatoric uncertainty as a runtime safety guard (triggering deceleration-to-stop when uncertainty exceeds the 90th percentile) further improved success rates in dense traffic.
• Statistical Robustness: Analysis using RLiable metrics confirmed that the proposed method not only achieved higher mean performance but also demonstrated greater statistical stability (higher Interquartile Mean) and a reduced optimality gap compared to the baseline.

Significance
The paper claims that coupling risk-sensitive uncertainty estimation with regulated expert integration enables safer and more efficient exploration for sensor-based RL policy learning. By addressing the limitations of static safety constraints and unregulated expert intervention, the framework allows agents to internalize safety within the learned policy without creating a lasting dependency on external guidance. The results demonstrate that this approach effectively navigates the trade-off between exploration safety and sample efficiency in complex, unsignalized intersection scenarios.