Robust and Safe Multi-Agent Reinforcement Learning with Communication for Autonomous Vehicles: From Simulation to Hardware

This paper introduces RSR-RSMARL, a robust and safe multi-agent reinforcement learning framework that integrates communication, robust policy training for zero-shot sim-to-real transfer, and Control Barrier Function-based safety shields, successfully demonstrating enhanced coordination and safety on hardware autonomous vehicles.

The Gist

Imagine a group of self-driving toy cars trying to race around a track together. In the perfect world of computer simulations, these cars can talk to each other instantly, like telepathic twins. If one car sees a pothole, it tells the others immediately, and everyone reacts at the exact same time.

But in the real world, that's not how it works. Real cars talk over Wi-Fi, and that signal takes time to travel. Sometimes it's fast, sometimes it's slow, and sometimes the message arrives a split second late. If you train your cars to expect instant telepathy, they will crash when you put them on a real track because they are reacting to information that is already old news.

This paper introduces a new training method called RSR-RSMARL (a mouthful, but think of it as "Real-Sim-Real Smart Driving") to solve this exact problem. Here is how it works, broken down into simple concepts:

1. The "Real-World Delay" Training

Instead of pretending the cars can talk instantly, the researchers measured how long it actually takes for their toy cars to send messages to each other. They found it takes about 10 to 20 milliseconds (a blink of an eye, but a long time for a fast car).

They then built this "lag" directly into the computer simulation. They taught the AI cars to drive while knowing that their friends' messages might be a little late. It's like training a basketball team where the coach yells instructions with a slight delay, forcing the players to learn how to anticipate and react even when the signal isn't perfect. This way, when the cars go from the computer to the real world, they are already used to the delay.

2. The "Safety Guard" (The Bouncer)

Even with good training, AI can sometimes make a risky move. To prevent crashes, the researchers added a "Safety Shield." Think of this as a strict bouncer at a club or a safety net for a gymnast.

• The Coach (The AI): The AI decides what the car should do (e.g., "Change lanes now!").
• The Bouncer (The Safety Shield): Before the car actually moves, the Safety Shield checks the plan. It asks, "Is this safe given where the other cars are right now and where they might be if their message was late?"
• The Result: If the AI's plan is too risky, the Safety Shield gently nudges the car to do something safer (like slowing down) instead of crashing. This happens in real-time, every single second.

3. The "Plug-and-Play" Brakes

The system is designed to be flexible. The AI can talk to different types of "low-level" controllers (the parts that actually press the gas or brake).

• PID Controller: Like a simple, fast reflex. Good for quick, light reactions.
• MPC Controller: Like a chess player. It thinks a few steps ahead to make the ride smoother, though it takes a tiny bit more brainpower.
  The researchers showed their system works great with both types, proving it's a versatile framework.

4. The Big Test: From Simulation to Reality

The team tested this in two ways:

1. In the Computer (CARLA Simulator): They ran thousands of races with different levels of "lag" and obstacles.
2. On Real Hardware: They put the trained AI onto a fleet of 1/10th-scale autonomous cars (about the size of a large shoebox) equipped with cameras and lasers.

The Results:

• No Crashes: The cars trained with the "Real-World Delay" method and the "Safety Shield" completed the tracks without hitting anything, even when the other cars were moving unpredictably.
• The "Telepathy" Fail: Cars trained without accounting for delays (assuming instant communication) crashed much more often when put on the real track.
• The "No-Talk" Fail: Cars that couldn't talk to each other at all were slower and more likely to bump into things.
• The "Time-Varying" Winner: The best results came from training the cars with changing delays (sometimes fast, sometimes slow), just like real Wi-Fi. This made them the most adaptable and safe.

The Bottom Line

This paper proves that to make self-driving cars safe in the real world, you can't just train them in a perfect, instant-communication simulation. You have to teach them to deal with the messy reality of delayed messages and give them a "safety guard" that overrides bad decisions. By doing this, they can learn in a computer and then immediately drive safely on a real track without needing extra practice.

Technical Summary

Technical Summary: Robust and Safe Multi-Agent Reinforcement Learning with Communication for Autonomous Vehicles

Problem Statement
Deep Multi-Agent Reinforcement Learning (MARL) has shown promise for connected autonomous driving in simulation; however, a significant gap exists between simulated performance and real-world hardware deployment. Prior work often assumes instantaneous, perfectly synchronized inter-agent communication, which fails to account for the inherent latency and asynchrony of Vehicle-to-Vehicle (V2V) systems. In practical environments, shared information is subject to measurable delays and transmission variability. Furthermore, existing testbeds often lack the necessary components to validate end-to-end multi-agent frameworks with formal safety guarantees, and standard sim-to-real transfer techniques (like domain randomization) often fail to address the specific combined challenges of communication latency, sensing uncertainty, and safety-critical constraints.

Methodology: RSR-RSMARL
The authors propose RSR-RSMARL (Robust and Safe Real-Sim-Real Multi-Agent Reinforcement Learning), a framework designed to bridge the gap between simulation and hardware by explicitly modeling communication constraints. The architecture consists of three core components:

1. Communication-Aware MARL Formulation:

   • Real-to-Sim Design: Instead of assuming ideal information exchange, the framework incorporates measured real-world V2V latency statistics directly into the state representation.
   • Delayed State Modeling: Agents receive local observations and shared observations from neighbors ($N_i$) that are treated as delayed and potentially asynchronous. The shared state includes complementary perception features (e.g., obstacle presence, lane boundaries) derived from onboard sensors (LiDAR, cameras).
   • Training Strategy: The policy is trained under both fixed and time-varying delay models. This exposes the agents to temporally misaligned neighbor information, forcing the policy to learn robust coordination strategies that do not rely on synchronous data.

2. CBF-Based Safety Shield:

   • To ensure formal safety guarantees during both training and deployment, a modular Safety Shield is integrated. This shield utilizes Control Barrier Functions (CBFs) enforced via a Quadratic Program (QP).
   • Mechanism: High-level actions generated by the MARL policy are passed through the Safety Shield, which filters out unsafe commands before they reach the low-level controller.
   • Delay-Aware Safety: The CBF constraints are conservatively inflated to account for state uncertainty caused by communication delays and asynchronous updates. Neighboring vehicles are modeled as dynamic obstacles with bounded delays, ensuring collision avoidance guarantees even when neighbor state estimates are stale.
   • Controller Agnosticism: The framework supports pluggable low-level controllers, specifically PID (for lightweight execution) and Model Predictive Control (MPC) (for smoother, optimization-based trajectories), without altering the learning formulation.

3. Real-Sim-Real Pipeline:

   • The system employs a Centralized Training with Decentralized Execution (CTDE) approach. Policies are trained in the CARLA simulator using measured latency distributions and then transferred zero-shot to physical hardware.
   • Hardware Platform: Validation is performed on a fleet of 1/10th-scale F1TENTH autonomous vehicles equipped with LiDAR, cameras, and IMUs, communicating via Wi-Fi with empirically measured latencies (10–20 ms in the testbed, with training sampling from broader distributions to mimic production variability).

Key Contributions

1. Framework Proposal: Introduction of RSR-RSMARL, a robust MARL framework specifically designed for Real-Sim-Real policy transfer that integrates delay-aware training and a CBF-based Safety Shield to address communication latency, model uncertainties, and state estimation errors while maintaining formal safety guarantees.
2. Latency Modeling Strategy: Development of a communication-aware training strategy that explicitly injects stochastic latency (both fixed and time-varying) into the MARL learning process. This enables delay-robust coordination prior to deployment, moving beyond the assumption of ideal communication.
3. Extensive Validation: Comprehensive evaluation in CARLA and on physical hardware platforms. The study includes structured ablation studies under increasing communication delays and demonstrates successful zero-shot transfer of simulation-trained policies for safe real-world operation.

Experimental Results
The framework was evaluated in two scenarios: a 3-lane miniature highway and a 2-lane circular highway, with varying obstacle densities.

• Safety and Zero-Shot Transfer: The proposed RSR-RSMARL variants (using both Time-Varying (TV) and Fixed-delay (F-2) models) achieved zero collisions across all obstacle levels on both hardware and simulation. In contrast, baselines without the Safety Shield (RSR-MARL) or without communication (No-Comm) exhibited increasing collision rates as obstacle density increased.
• Efficiency vs. Safety: While the TV delay model resulted in slightly longer completion times compared to some baselines, it maintained a collision-free record. The Time-Varying model consistently outperformed fixed-delay models in terms of coordination efficiency and stability, suggesting that training with stochastic delays better prepares agents for real-world jitter.
• Controller Performance: The framework successfully operated with both PID and MPC backends. The integration of MPC further enhanced trajectory smoothness, though at a higher computational cost.
• Ablation Studies: Removing the Safety Shield led to unstable learning dynamics and significantly higher collision rates, confirming that delay modeling alone is insufficient for safety. Comparisons with Domain Randomization (MARL-DR) showed that MARL-DR suffered from higher collision rates and lower efficiency, indicating that explicit delay modeling is superior to generic noise injection for this specific problem.
• Communication Impact: Experiments demonstrated that V2V communication significantly reduces the need for reactive safety overrides (CBF interventions), lowering the intervention rate from 18.7% (no communication) to 11.4% (with communication).

Significance and Claims
The paper claims that RSR-RSMARL provides a structured and reliable pathway for transferring MARL-based cooperative decision-making from simulation to physical systems. The authors emphasize that their approach addresses the critical "sim-to-real" gap by grounding training in experimentally observed delay statistics and enforcing formal safety constraints.

The significance of this work lies in its demonstration that hardware-grounded communication modeling is essential for scalable and reliable multi-agent autonomous systems. By integrating delay-aware learning with formal safety enforcement, the framework supports zero-shot transfer to real-world platforms, maintaining strong safety-performance trade-offs even under realistic, asynchronous, and delayed communication constraints. The authors position this as a step toward the practical deployment of robust, communication-aware multi-agent CAV autonomy.