A Robust and Efficient Multi-Agent Reinforcement Learning Framework for Traffic Signal Control

This paper proposes a robust Multi-Agent Reinforcement Learning framework for traffic signal control that integrates turning ratio randomization, an exponential phase duration adjustment action space, and a neighbor-based MAPPO observation scheme to significantly reduce average waiting time and improve generalization in dynamic traffic scenarios.

The Gist

Imagine a busy city intersection as a giant, chaotic dance floor. The traffic lights are the DJs, and the cars are the dancers. Right now, most traffic lights are like DJs playing a pre-recorded playlist on a loop. They switch songs (phases) at the exact same time every day, regardless of whether the dance floor is packed or empty. This works okay when the crowd is predictable, but if a sudden rush of dancers arrives, the DJ keeps playing the slow song, causing a massive pile-up.

This paper introduces a new, super-smart DJ system using Artificial Intelligence (AI) to manage traffic lights. The goal is to make the lights "dance" in real-time with the cars, reducing how long people sit in traffic.

Here is how their new system works, broken down into three simple tricks:

1. The "Surprise Party" Training (Turning Ratio Randomization)

The Problem: Imagine you train a robot to play soccer only on a sunny day with a flat field. If you suddenly put it on a rainy, muddy field with a bumpy slope, it will fail miserably. Similarly, most AI traffic systems are trained on "perfect" data where traffic patterns never change. When real life hits (like a sudden detour or a different rush hour), these systems get confused because they just memorized the "perfect" pattern instead of learning how to react.

The Solution: The authors decided to train their AI like a "surprise party." Every time they practiced, they randomly changed the rules of the game. Sometimes, more cars would turn left; sometimes, fewer would go straight. They didn't let the AI memorize a schedule. Instead, they forced it to learn how to adapt to chaos.

• The Analogy: It's like training a chef not just for a specific recipe, but by throwing random ingredients at them every day. By the time they open the restaurant, they can cook a delicious meal no matter what ingredients the customers bring in.

2. The "Zoom Lens" Control (Exponential Phase Duration Adjustment)

The Problem: Traffic lights need to be stable (so drivers don't get confused) but also fast to react to jams. Old systems were like a light switch: either you change the light by a tiny bit (too slow to fix a jam) or a huge bit (too jarring and unsafe). It was a "one size fits all" approach.

The Solution: The new system uses a "zoom lens" approach. It can make tiny, precise adjustments when traffic is calm (like turning a volume knob slightly), but it can also make giant, rapid jumps when a massive traffic jam appears (like slamming the volume up).

• The Analogy: Think of driving a car. When you are cruising on the highway, you make tiny steering adjustments to stay in the lane. But if a deer jumps out, you make a huge, sharp turn instantly. This system does both: it's gentle when things are calm and aggressive when things get dangerous.

3. The "Neighborhood Watch" (Scalable Coordination)

The Problem: To control a whole city perfectly, you'd need one giant brain that sees every single car in the entire city at once. But that's impossible; the computer would crash from trying to process so much data. On the other hand, if every traffic light only looks at its own intersection, they act selfishly and cause gridlock.

The Solution: The authors created a "Neighborhood Watch" system. Each traffic light (agent) only looks at itself and its immediate neighbors (the next few intersections). However, they use a special training trick called CTDE (Centralized Training, Decentralized Execution).

• The Analogy: Imagine a group of firefighters. During training, they all stand in one room with a giant map of the whole city, learning how to work together as a team (Centralized Training). But when the fire actually happens, they split up. Each firefighter only sees the fire in front of them and the firefighters right next to them, but because of their training, they know exactly what to do without needing to talk to the Chief in the tower (Decentralized Execution). They act locally but think globally.

The Results

The team tested this system in a super-realistic computer simulation of a real road in Taiwan (using a program called Vissim that mimics real human driving behavior, not just simple math).

• The Outcome: Their new AI system reduced the average time people spent waiting in traffic by over 10% compared to standard methods.
• The Best Part: It didn't just work on the days it was trained for. Because of the "Surprise Party" training, it handled new, unseen traffic patterns much better than the old systems, which completely broke down when the traffic changed.

In a Nutshell

This paper presents a traffic light system that is flexible (learns from chaos), agile (can make small or big changes instantly), and smart (works together with neighbors without needing a supercomputer). It's a step toward a future where traffic lights don't just follow a clock, but actually "feel" the traffic and dance along with it.

Technical Summary

1. Problem Statement

The paper addresses the critical challenges in deploying Deep Reinforcement Learning (DRL) for Traffic Signal Control (TSC) in real-world scenarios. While DRL offers adaptability, existing approaches face three fundamental hurdles:

• Lack of Generalization: Agents trained on static traffic patterns often overfit to specific demand profiles (e.g., fixed turning ratios) and fail to adapt to the non-stationary, stochastic nature of real-world traffic.
• Action Space Instability: Existing action spaces struggle to balance safety and responsiveness. Fixed-duration methods lack flexibility, while binary switching or linear adjustment methods often cause signal oscillation or cannot react quickly enough to sudden congestion surges.
• Scalability vs. Coordination: Centralized control (global observation) is theoretically optimal but computationally unscalable for large networks. Conversely, decentralized control (local observation) is scalable but "myopic," failing to coordinate with upstream/downstream traffic (preventing "green waves").

Additionally, the paper notes a gap in high-fidelity simulation; most RL-TSC research uses simplified simulators (SUMO, CityFlow), whereas this study utilizes PTV Vissim, the industry standard for microscopic traffic modeling, to bridge the "sim-to-real" gap.

2. Methodology

The authors propose a robust Multi-Agent Reinforcement Learning (MARL) framework validated in Vissim. The framework integrates three core technical innovations:

A. Turning Ratio Randomization (Robustness)

To prevent overfitting to static traffic patterns, the authors introduce a training strategy where turning probabilities are dynamically perturbed at the start of every training episode.

• Mechanism: For each movement $m$, a noise factor $\epsilon_m$ is sampled from a uniform distribution $U(-\delta, \delta)$. The original ratio $r_m$ is scaled to $\hat{r}_m = r_m(1+\epsilon_m)$ and then re-normalized to ensure probabilities sum to 1.
• Goal: This forces the agent to learn state-based reactivity rather than memorizing fixed timing schedules, ensuring the policy remains robust under unseen traffic distributions.

B. Exponential Phase Duration Adjustment (Stability & Responsiveness)

The framework adopts a cyclic control scheme (Green $\to$ Yellow $\to$ Red) to ensure driver safety and sequence consistency. Instead of setting absolute durations, the agent selects an adjustment $\Delta t$ for the next phase's green duration.

• Mechanism: The action space uses an exponential scale: $\Delta t \in \{0, \pm\lambda^0, \pm\lambda^1, \pm\lambda^2, \pm\lambda^3\}$.
• Benefit: This provides "coarse-to-fine" control. Small steps (e.g., $\pm 1s$) allow precise tuning during stable traffic, while large steps (e.g., $\pm 8s$ or more) enable rapid reaction to sudden congestion surges, overcoming the trade-off inherent in linear adjustment methods.

C. Neighbor-Based Observation via CTDE (Scalability)

To resolve the scalability-coordination dilemma, the framework employs Centralized Training with Decentralized Execution (CTDE) using the MAPPO (Multi-Agent Proximal Policy Optimization) algorithm.

• Observation Scope: During execution, agents only observe their local intersection and directly connected neighbors ($O_{neighbor}$).
• Training: A centralized critic utilizes global network information to evaluate the joint impact of local actions, guiding the decentralized actors toward cooperative behaviors.
• Result: This achieves global-level coordination performance while maintaining a constant state space size independent of network size.

3. Experimental Setup

• Environment: PTV Vissim microscopic simulator using the Wiedemann car-following model.
• Testbed: A calibrated digital twin of Zhongzheng East Road in Taoyuan City, Taiwan, featuring five consecutive signalized intersections.
• Data: Real-world 24-hour traffic data was used.
  • Training: Strictly on Peak Hour data (high load, ~4800 veh/hr).
  • Testing: Evaluated on both Peak and Off-Peak hours (~1800 veh/hr) to test generalization.
• Baselines: Fixed-time plans, MaxPressure heuristic, and standard RL variants (Static training, Local/Global/Neighbor observation, Non-CTDE).

4. Key Results

The proposed framework ($M^{randomized}_{neighbor}$) demonstrated superior performance across all metrics:

• Generalization: Standard RL models trained with static ratios failed significantly in off-peak scenarios (overfitting). The proposed Turning Ratio Randomization allowed the model to reduce Average Travel Time (ATT) by over 10% in unseen off-peak scenarios compared to baselines.
• Performance Metrics:
  • Peak Hour: Achieved an ATT of 230.58s, significantly outperforming the MaxPressure heuristic (265.79s).
  • Off-Peak Hour: Achieved an ATT of 124.37s, outperforming MaxPressure (126.57s) and approaching the performance of the theoretically optimal but unscalable Global-View agent (119.32s).
• Ablation Studies:
  • CTDE vs. Non-CTDE: MAPPO (CTDE) significantly outperformed IPPO (decentralized critic), proving the necessity of a global critic for stable credit assignment in multi-agent systems.
  • Action Space: The Exponential Adjustment outperformed both Small-Scale and Large-Scale Linear Adjustments, particularly in off-peak scenarios where linear methods caused instability or slow reaction times.

5. Significance and Contributions

1. Real-World Viability: By utilizing PTV Vissim and real-world traffic data, the study bridges the gap between theoretical RL research and practical deployment, addressing the "sim-to-real" challenge.
2. Robustness Strategy: The Turning Ratio Randomization technique provides a novel regularization method to prevent overfitting in non-stationary environments, a common failure point in RL-TSC.
3. Balanced Control: The Exponential Phase Duration Adjustment offers a mathematically grounded solution to the stability-vs-reactivity trade-off, ensuring safety while maintaining agility.
4. Scalable Coordination: The framework demonstrates that high-level coordination can be achieved without global observation, making it feasible for large-scale urban networks.

In conclusion, this paper presents a practical, robust, and scalable MARL framework that significantly improves traffic flow efficiency and adaptability, offering a viable path toward autonomous, adaptive traffic signal control systems.