Human-Centric Traffic Signal Control for Equity: A Multi-Agent Action Branching Deep Reinforcement Learning Approach

Technical Summary: Human-Centric Traffic Signal Control for Equity

Problem Statement
Coordinating traffic signals along multimodal corridors presents significant challenges. Existing Multi-Agent Deep Reinforcement Learning (DRL) approaches are predominantly vehicle-centric, often neglecting pedestrians, cyclists, and transit passengers. Furthermore, these methods struggle with high-dimensional discrete action spaces inherent in multi-intersection coordination. Traditional adaptive systems (e.g., SCATS, SCOOT) often require manual tuning and fail to adequately accommodate non-motorized users. The core problem addressed is the lack of a scalable, human-centric framework that explicitly optimizes for traveler-level equity (fairness across all transport modes) while effectively managing the complexity of discrete action spaces in a multi-agent environment.

Methodology: MA2B-DDQN
The authors propose MA2B-DDQN (Multi-Agent Action-Branching Double Deep Q-Network), a cooperative multi-agent DRL framework designed for medium-scale corridor networks. The methodology is built on five key components:

1. Environment and State Representation:

   • The simulation utilizes PTV VISSIM modeling a real-world corridor in Melbourne, Australia, featuring three intersections with mixed traffic (cars, buses, pedestrians).
   • The state vector $S_t$ combines three elements: traffic conditions (vehicle occupancy, velocity, acceleration, and passenger counts per lane cell), pedestrian information (waiting counts per zone), and current signal phases (one-hot encoded).

2. Action Branching Architecture:

   • To address the "curse of dimensionality" in discrete multi-agent spaces, the action space is decomposed into $n+1$ dimensions (where $n$ is the number of intersections).
   • Local Actions ($n$ dimensions): Each agent independently determines the green time allocation ratio ($p_{i,1}$) between the next two phases.
   • Global Action (1 dimension): A single global action selects the total duration ($C$) of the upcoming two phases for the entire corridor.
   • This decomposition allows the model to handle discrete decisions (phase selection and duration) more efficiently than standard joint-action DRL.

3. Human-Centric Reward Mechanism:

   • Unlike traditional rewards based on vehicle queue length or throughput, the reward function penalizes the total number of delayed individuals.
   • It aggregates delays for pedestrians, vehicle occupants, and transit passengers. A delay is triggered when vehicle speed drops below a threshold (5 km/h) or when pedestrians wait at crossing zones.
   • The reward is formulated as $r_t = -(w_{veh} \sum N^{veh}_{i,t} + w_{ped} \sum N^{ped}_{j,t})$, prioritizing the reduction of impacted travelers rather than just vehicle flow.

4. Network Architecture:

   • The model employs a Double DQN (DDQN) framework to mitigate Q-value overestimation.
   • The network uses a shared feature extractor (two dense layers with ReLU and Layer Normalization) feeding into $n+1$ parallel output heads (one per action dimension).
   • Actions are selected via $\arg\max$ over the Q-values of each branch.

5. Learning Algorithm:

   • The system uses Centralized Training with Decentralized Execution (CTDE). Agents share global information during training to update policies but execute actions locally.
   • The loss function is the average of the losses across all $n+1$ action dimensions, optimized using the Adam optimizer.

Key Contributions

1. Human-Centric Equity: The study introduces a forward-looking TSC approach that treats all transportation network participants (private vehicle occupants, public transit passengers, cyclists, and pedestrians) equally, explicitly optimizing for the reduction of delayed individuals.
2. Cooperative Multi-Agent Framework: The authors present a framework that adaptively determines a global action (total phase duration) using shared information while simultaneously optimizing local signal timings (green time splits) via individual agents.
3. Action Branching for Multi-Agent DRL: This is the first application of an action-branching architecture to multi-agent TSC. It effectively handles the discrete action space and multi-agent setting by decomposing the control problem, thereby reducing the complexity of joint decision-making.

Experimental Results
The model was evaluated across seven realistic traffic scenarios in Melbourne, ranging from off-peak and rush hour to school periods and near-saturation conditions. It was benchmarked against nine baselines, including fixed-time signals (Webster's Formula, Green Wave), and various DRL models (MADQN, MADDQN, MADDPG, CMRM, MASAC, MAA2C, MA2B-DQN).

• Performance: MA2B-DDQN consistently achieved the lowest Aggregate Individual Delay (AID) and Per Capita Delay (PCD) across all scenarios.
  • In Scenario 1 (off-peak), it reduced AID by 36.75% compared to the best fixed-signal baseline and 11.94% compared to the best DRL baseline (MAA2C).
  • In Scenario 2 (rush hour), it improved performance by 11.73% over the next-best DRL model.
  • In Scenario 4 (afternoon school), it achieved a 35.24% improvement over MASAC.
• Robustness: The model demonstrated the lowest standard deviation in performance metrics across repeated trials, indicating high stability compared to baselines like MADDPG, which showed high variance.
• Ablation Studies:
  • Removing the action branching mechanism significantly degraded performance (e.g., a 16.09% increase in AID in Scenario 1), confirming its critical role in learning efficiency.
  • Incorporating the Double DQN (DDQN) architecture improved performance over standard DQN by 2.93% to 16.40% depending on the scenario, validating the mitigation of overestimation.
• Trade-offs: While MA2B-DDQN excelled in overall equity and vehicle flow, some baselines (e.g., CMRM, MASAC) occasionally achieved lower pedestrian-specific delays (APDS/AWTP) at the cost of significantly higher vehicle delays. MA2B-DDQN maintained a more balanced optimization across modes.

Significance and Claims
The paper claims that MA2B-DDQN provides a scalable, robust solution for multimodal traffic signal control that explicitly addresses equity. Its significance lies in:

• Shifting the paradigm from vehicle-centric optimization to individual-level equity, ensuring fair access for all road users regardless of travel mode.
• Demonstrating that action branching is a viable and effective strategy for handling high-dimensional discrete action spaces in multi-agent environments, outperforming both traditional fixed-time controls and other state-of-the-art DRL methods.
• Offering a practical framework adaptable to varied urban conditions, validated on a real-world corridor with realistic multimodal traffic data.

The authors conclude that while DRL generally outperforms fixed-time methods, the specific combination of action branching, DDQN, and human-centric rewards is necessary to achieve both high efficiency and fairness in complex, multimodal networks. They note that in near-saturation conditions, physical lane capacity becomes the limiting factor, but their method still maintains superior performance relative to baselines.