Beyond Scalar Rewards: Distributional Reinforcement Learning with Preordered Objectives for Safe and Reliable Autonomous Driving

1. Problem Statement

Autonomous driving (AD) requires balancing multiple, often conflicting objectives such as safety, efficiency, comfort, and risk mitigation. Current Reinforcement Learning (RL) approaches typically address this by collapsing these objectives into a single scalar reward via weighted summation. This approach has significant limitations:

• Loss of Priority: It obscures the relative importance of objectives, often leading to policies that sacrifice safety for efficiency or comfort.
• Tuning Difficulty: Finding the correct weights is context-dependent and prone to error.
• Inadequate Hierarchical Handling: Existing Multi-Objective RL (MORL) methods often still rely on scalar aggregation or rigid lexicographic ordering, failing to capture flexible priority structures (partial orders) or disentangle the contribution of each objective during decision-making.

The core problem is the lack of a framework that can semantically represent multiple objectives with explicit priority relations and enforce these priorities directly within the learning and inference processes without collapsing them into a single scalar.

2. Methodology

The authors propose a novel framework that integrates Preordered Multi-Objective MDPs (Pr-MOMDP) with Distributional Reinforcement Learning (DRL).

A. Preordered Multi-Objective MDP (Pr-MOMDP)

The authors extend the standard MOMDP formulation by introducing a preorder relation ($\succeq$) over reward components.

• Structure: Instead of a vector of rewards $R = \{r_1, ..., r_N\}$, the system defines a hierarchy where $r_i \succeq r_j$ indicates that objective $i$ has higher priority than $j$.
• Flexibility: This allows for both total orders (strict hierarchy) and partial orders (where some objectives are incomparable), represented as directed graphs.
• Action Relations: This preorder induces semantic relations between actions:
  • Dominance: Action $a$ dominates $a'$ if it performs better on a higher-priority objective, or if it performs better on a lower-priority objective while not violating higher ones.
  • Indifference/Incomparability: Actions are indifferent if neither dominates the other, or incomparable if they trade off on incomparable objectives.

B. Quantile Dominance (QD)

To make these relations actionable in a DRL setting, the authors introduce Quantile Dominance, a metric that compares full return distributions rather than scalar values.

• Distributional Representation: Using Implicit Quantile Networks (IQN), the agent estimates the full return distribution $Z_{r_i}(s, a)$ for each objective.
• Normalization: To ensure robustness against noise and scale differences, quantile estimates are normalized using Z-score normalization.
• Ideal Reference: An "ideal" distribution is constructed by taking the maximum return across all actions at each quantile.
• Scoring: The quality of an action is measured by its Wasserstein-1 distance to this ideal profile.
• Comparison: Two actions are compared based on the directional difference in their scores. A tolerance parameter ($\epsilon$) is introduced to define indifference, preventing overly strict comparisons due to estimation noise.

C. Pr-IQN Algorithm (Preorder-Informed IQN)

The authors propose Pr-IQN, an algorithm that integrates the preorder into both training and inference:

1. Architecture: A multi-head network where a shared encoder feeds into separate heads for each objective, preserving distinct value estimates.
2. Optimal Subset Extraction (Algorithm 1): The algorithm traverses the reward hierarchy (topologically sorted). It iteratively filters the action space:
   • It inherits dominance relations from higher-priority objectives.
   • It applies the QD metric to undecided action pairs at the current level.
   • It produces an Optimal Subset of non-dominated actions for each objective level.
3. Training & Inference:
   • Training: Temporal Difference (TD) targets are restricted to the Optimal Subset derived from the hierarchy. This prevents the agent from learning to maximize a lower-priority reward if it violates a higher-priority constraint.
   • Inference: The agent selects actions uniformly from the optimal subset associated with the leaf nodes of the hierarchy.

3. Key Contributions

1. Pr-MOMDP Formulation: A formal extension of MOMDPs that incorporates a preorder relation over rewards, enabling reasoning about actions based on a hierarchy of objectives rather than a scalar signal.
2. Quantile Dominance (QD): A novel distributional metric that compares action return distributions pairwise without reducing them to a single statistic, enabling robust action selection under uncertainty.
3. Optimal Subset Extraction: An algorithm that extracts non-dominated action subsets consistent with the reward preorder, which are then used to shape both training targets and inference decisions.
4. Pr-IQN Implementation: A concrete instantiation of the framework using Implicit Quantile Networks, demonstrating compatibility with existing distributional RL methods.

4. Experimental Results

The framework was evaluated in the CARLA simulator on urban driving tasks involving unsignalized intersections (T-junctions and four-way intersections) with varying traffic densities.

• Baselines: Compared against standard IQN, Ensemble-IQN (same capacity but no hierarchy), and Mean-Aggregated IQN (MA-IQN).
• Performance Metrics: Success Rate (SR), Collision Rate (CR), Off-road Rate (OR), and Route Progress (RP).
• Key Findings:
  • Safety & Reliability: Pr-IQN significantly outperformed baselines, reducing collisions and off-road events. At high traffic density (1.0), it improved success rates by 20.3% over standard IQN and 13.9% over Ensemble-IQN.
  • Objective Optimization: Pr-IQN achieved up to 61% improvement in safety violations and 41% in risk exposure compared to baselines, proving that the preorder successfully prevents the agent from trading safety for efficiency.
  • Robustness: Statistical analysis (using RLiable metrics like Interquartile Mean and Optimality Gap) showed Pr-IQN is more stable and reliable across different random seeds.
  • Ablation: Removing the preorder during training or collapsing distributions into scalar metrics (like CVaR) resulted in performance degradation, confirming the necessity of the full distributional and hierarchical approach.

5. Significance

This work addresses a critical gap in safe autonomous driving by moving beyond scalar reward engineering.

• Safety-Critical Guarantees: By enforcing reward priorities directly in the learning process, the system ensures that safety-critical constraints (like collision avoidance) are never compromised for secondary goals (like comfort or speed).
• Interpretability: The use of preorders and multi-head architectures makes the decision-making process more interpretable, as the hierarchy of objectives is explicit.
• Scalability: The framework is designed to handle complex, real-world scenarios with partial orders and can be extended to include external components (e.g., Car2X) in the future.
• Generalizability: While tested on AD, the Pr-MOMDP and QD framework offers a generalizable solution for any RL application requiring strict adherence to hierarchical or conflicting objectives.