Offline Reinforcement Learning using Human-Aligned Reward Labeling for Autonomous Emergency Braking in Occluded Pedestrian Crossing

Technical Summary: Offline Reinforcement Learning using Human-Aligned Reward Labeling for Autonomous Emergency Braking in Occluded Pedestrian Crossing

Problem Formulation

The paper addresses a critical bottleneck in the deployment of Autonomous Driving (AD) systems: the scarcity of meaningful reward labels in real-world datasets required for Offline Reinforcement Learning (Offline RL). While real-world datasets (e.g., Waymo, nuScenes, A2D2) offer rich, high-fidelity data, they typically lack the reward signals necessary to train agents to distinguish between desirable and undesirable behaviors. Existing solutions often rely on simulation, which suffers from the "sim-to-real" gap, or use low-dimensional state representations that ignore the perceptual richness of image-based observations.

The specific task investigated is vehicle longitudinal control within an occluded pedestrian crossing scenario. In this high-risk environment, an autonomous vehicle (AV) must navigate a zebra crossing where pedestrians may be hidden by other vehicles (occlusion) and suddenly emerge. The challenge lies in generating reward labels that align with human safety values (prioritizing safety over efficiency in potential collision scenarios) without direct human annotation of every frame, thereby enabling the training of perception-based Offline RL agents directly from static datasets.

Methodology

The proposed solution introduces a novel reward labeling pipeline that transforms unlabeled AD datasets into training-ready data for Offline RL. The methodology consists of three primary stages:

1. Semantic Segmentation and Risk Assessment:

   • Semantic Mapping: Raw camera images are processed through a ResNet18 UNet model to generate pixel-level semantic segmentation maps. These maps identify 28 classes, including pedestrians, vehicles, zebra crossings, and road markings.
   • Adaptive Safety Component ($c_t$): The system analyzes specific layers of the semantic map to compute an adaptive safety flag ($c_t$). This flag is determined by three risk factors:
     • Pedestrian Presence ($F_p$): Detects if a pedestrian is on the road versus the sidewalk.
     • Zebra Crossing Occlusion ($F_c$): Assesses if a zebra crossing is partially occluded by traffic, creating uncertainty.
     • Pedestrian History ($F_h$): Uses a memory bank to track pedestrians that appear and then disappear (indicating occlusion), flagging them as medium risk.
   • Decision Logic: These factors are summed and passed through a scaled sigmoid function. If the resulting probability exceeds a threshold ($\psi = 0.75$), $c_t$ is set to 1, activating the safety reward component and deactivating the efficiency component.

2. Reward Function Construction:
   The final reward signal ($R_{final}$) is a weighted sum of three components, dynamically switched based on $c_t$:

   • Safety ($g_{safe}$): Penalizes high speeds near pedestrians and collisions. Activated when $c_t=1$.
   • Efficiency ($g_{efficient}$): Encourages higher speeds when no collision is imminent. Activated when $c_t=0$.
   • Smoothness ($g_{smooth}$): Penalizes jerky acceleration, always active.
   • Sensor Integration: For real-world data, distance to pedestrians is inferred using the pinhole camera model combined with semantic segmentation, while speed and acceleration are derived from vehicle CAN bus and IMU data.

3. Offline RL Training with Spatial Attention:
   To train the agent, the paper proposes a perception-based pipeline that leverages the semantic maps to guide feature extraction:

   • Feature Extraction: Camera images are passed through a ResNet18 autoencoder to produce latent feature embeddings.
   • Spatial Attention: The task-relevant semantic layers (pedestrian, zebra crossing, vehicle) are isolated and used to apply spatial attention to the latent features. This is achieved via weighted cross-wise multiplication, where pedestrian features receive the highest weight ($\omega_{ped}=1.0$), followed by zebra crossings ($\omega_{cross}=0.75$) and vehicles ($\omega_{veh}=0.50$). This mechanism forces the agent to focus on salient objects relevant to collision avoidance.

Key Contributions

The paper outlines three main contributions:

1. Human-Aligned Reward Labeling: A novel approach to generate reward labels for existing AD datasets, enabling their use in end-to-end Offline RL without requiring new data collection.
2. Adaptive Safety Mechanism: A reward function incorporating an adaptive safety component derived from semantic segmentation maps. This allows the agent to dynamically prioritize safety over efficiency based on real-time risk factors like pedestrian location and occlusion.
3. Spatial Attention via Semantic Maps: A method to utilize semantic segmentation layers to provide varying levels of spatial attention to different object classes within the image observation, improving the agent's decision-making by focusing on salient scene elements.

Experimental Results

The method was evaluated in two settings: a CARLA simulation and a subset of the real-world Audi Autonomous Driving Dataset (A2D2).

• Simulation (CARLA):

  • Algorithms Tested: The labeled data was used to train CQL, IQL, and BPPO.
  • Baselines: The proposed method ("Gen") was compared against Unlabeled Data Sharing (UDS) (assigning zero rewards) and a Vision Language Model (VLM) approach.
  • Performance: The IQL (Gen) setup achieved the highest performance, with success rates of 98-100% in partial occlusion scenarios. The proposed method generally produced more stable and reliable behaviors than UDS or VLM baselines.
  • Limitations: Performance degraded in highly congested, fully occluded scenarios, particularly for CQL and BPPO, suggesting the front-facing camera view has limitations in dense traffic.

• Real-World (A2D2):

  • Comparison: The generated reward labels were compared against those manually annotated by a human.
  • Findings: While the overall distribution of rewards was similar, there was a moderate disparity in specific safety classifications. Notably, the method flagged certain unsafe states (e.g., heavily occluded pedestrians) that the human annotator missed. Conversely, the method sometimes failed to identify clearly visible pedestrians due to limitations in the specific risk factors used (e.g., confusion between road markings and road surfaces).

Significance and Claims

The paper claims that its primary significance lies in bridging the gap between large-scale, unlabeled real-world AD datasets and the requirements of Offline RL. By automating the generation of human-aligned reward labels, the method enables the training of perception-based agents without the prohibitive cost of manual annotation or the fidelity issues of simulation.

The authors emphasize that while the method successfully generates meaningful reward signals and improves policy performance in simulation, it is not a perfect replacement for human judgment. The A2D2 results highlight that the method can detect hazards humans miss (due to its systematic analysis of semantic maps) but also lacks the nuanced contextual reasoning of human drivers. The paper concludes that this approach provides a viable starting point for Offline RL in complex scenarios, though future work must address temporal reasoning and multi-view integration to handle dense traffic environments more robustly.