Traffic Co-Simulation Framework Empowered by Infrastructure Camera Sensing and Reinforcement Learning

Technical Summary: Traffic Co-Simulation Framework Empowered by Infrastructure Camera Sensing and Reinforcement Learning

Problem Statement
Traffic congestion remains a critical urban challenge, necessitating advanced Traffic Signal Control (TSC) strategies beyond traditional fixed-time or actuated systems. While Multi-Agent Reinforcement Learning (MARL) offers promising potential for adaptive signal control, existing simulation studies often rely on "perfect" vehicle detection data, overlooking real-world limitations such as sensor reliability, infrastructure availability, and the noise inherent in computer vision systems. Furthermore, standard traffic simulators like SUMO lack the high-fidelity synthetic sensor data required to realistically test vision-based control strategies. There is a gap in frameworks that integrate realistic, ground-level camera sensing with large-scale network traffic simulation to evaluate the robustness of MARL agents under imperfect sensing conditions.

Methodology
The study proposes a co-simulation framework integrating CARLA (for high-fidelity 3D environment and sensor simulation) and SUMO (for microscopic traffic flow and signal control).

1. Framework Architecture:

   • Environment: The framework utilizes the "Town04" map from CARLA, featuring a downtown grid with 12 intersections.
   • Sensing: Virtual RGB cameras are programmatically spawned on traffic light poles within CARLA. The system automatically calculates camera yaw and pitch angles based on the coordinates of the lane stopping lines in SUMO to ensure proper alignment.
   • Detection: A YOLO-based computer vision system (specifically YOLOv5 and YOLOv8) processes camera images to detect and count vehicles in approaching lanes. To manage computational load, the system processes images in batches every 5 seconds (every 100 simulation ticks).
   • State Translation: A custom module translates raw visual detection data (vehicle counts) into the structured state vectors (lane occupancy and queue length) required by the MARL agents. This bridges the gap between raw image data and the Q-learning input format.

2. Reinforcement Learning:

   • Agents: Multi-Agent Q-learning agents are trained to control individual traffic signals. Each agent operates independently without inter-agent communication.
   • State Space: Defined by the current phase index, minimum green time status, and vectors for lane occupancy and queue length.
   • Action Space: The selection of the next green phase index.
   • Reward Functions: Four distinct reward structures were tested to train the agents:
     1. Difference in waiting times.
     2. Average speed of vehicles near the intersection.
     3. Negative queue length.
     4. Pressure (difference between vehicles leaving and approaching).

3. Experimental Setup:

   • Agents were trained in SUMO using ground-truth data and then deployed in the CARLA-SUMO co-simulation.
   • Experiments were conducted under medium and heavy traffic demand scenarios.
   • Performance was evaluated by comparing MARL agents using camera-based detection against static timing, actuated control, and agents using ground-truth observations.

Key Contributions

• Novel Co-Simulation Framework: The study presents the first framework to explicitly model and evaluate a network of infrastructure-mounted cameras for MARL-based TSC, integrating high-fidelity 3D rendering (CARLA) with large-scale traffic flow (SUMO).
• Automated Camera Deployment: A technical contribution involves the automated spawning of virtual cameras on traffic poles and the calculation of their orientation (yaw/pitch) based on lane stopping line coordinates, enabling scalable deployment across complex networks without manual configuration.
• Robustness Evaluation: The framework explicitly tests MARL robustness against imperfect sensing. It compares the performance of YOLOv5 and YOLOv8 and analyzes how detection errors (false positives/negatives) impact agents trained with different reward functions.
• Vision-to-State Translation: The development of a translation module that converts raw visual detection counts into the specific occupancy and queue metrics required by pre-trained Q-learning agents.

Results

• Performance Improvement: MARL agents significantly outperformed static timing and actuated control methods in reducing total travel time and average waiting times, particularly when using the average speed reward function.
• Impact of Detection Accuracy: While higher detection accuracy (YOLOv8 vs. YOLOv5) improved system performance, the study found that MARL agents could still achieve significant improvements even with imperfect detection.
• Reward Function Sensitivity:
  • Agents trained with average speed rewards demonstrated the highest robustness to sensing noise, showing less than a 10% performance drop compared to ground-truth scenarios in medium traffic.
  • Agents trained with queue length and pressure rewards were highly sensitive to detection errors. In heavy traffic, the queue-length agent failed to learn effectively because it consistently observed "full queues" due to spillback outside the camera's field of view, leading to suboptimal phase selection.
• Computational Trade-offs: While YOLOv8 offered better accuracy (lower Mean Absolute Error), it doubled the simulation wall-clock time compared to real-time, highlighting a trade-off between detection fidelity and simulation speed.

Significance and Claims
The paper claims that this work provides a critical step toward the practical deployment of infrastructure-assisted reinforcement learning for automated vehicles. By demonstrating that vision-driven control remains effective despite imperfect sensing, the study addresses a major barrier to real-world implementation. The framework serves as a virtual test-bed for evaluating control strategies under realistic conditions (weather, lighting, sensor noise) that are difficult to replicate in single-simulator environments.

The authors position this co-simulation approach as a foundational step toward Digital Twin technologies for urban traffic. They argue that such frameworks are essential for training robust control algorithms that can handle noisy data and infrastructure variability before deployment. The study concludes that while detection accuracy is beneficial, the choice of reward function is paramount for ensuring system robustness in the face of real-world sensing limitations.