VissimRL: A Multi-Agent Reinforcement Learning Framework for Traffic Signal Control Based on Vissim

Technical Summary: VissimRL: A Multi-Agent Reinforcement Learning Framework for Traffic Signal Control Based on Vissim

Problem Statement
Traffic congestion poses significant economic and environmental challenges, necessitating effective Traffic Signal Control (TSC). While Reinforcement Learning (RL) has demonstrated potential in adaptive TSC, its application in high-fidelity industrial simulators remains limited. Existing research predominantly utilizes open-source simulators like SUMO and CityFlow due to their accessibility and standardized RL frameworks (e.g., SUMO-RL). However, these tools often rely on rule-based driver behavior models that oversimplify traffic dynamics, limiting their relevance to industry and policy. Conversely, Vissim, a commercial-grade simulator widely adopted by transportation agencies for its high-fidelity driver behavior modeling and 3D visualization, is underutilized in RL research. This gap stems from Vissim's complex Component Object Model (COM) interface and the absence of standardized, open-source frameworks for seamless RL integration.

Methodology
The paper proposes VissimRL, a modular RL framework designed to bridge the gap between Vissim's high-fidelity simulation capabilities and modern RL algorithms. The architecture consists of two primary components:

1. Vissim Wrapper: A Python-based encapsulation of Vissim's COM interface. This module abstracts the complexity of the underlying COM API (which includes thousands of attributes and object types) into a user-friendly, high-level API. It handles simulation control, signal operations, and performance evaluation. Crucially, the wrapper employs performance-oriented mechanisms such as batching and caching of repeated queries to minimize the overhead of COM calls, thereby maintaining runtime efficiency comparable to or better than direct COM usage.
2. RL Environment Framework: This component implements standardized interfaces compatible with Gymnasium (for single-agent) and PettingZoo (for multi-agent) APIs. It facilitates the integration of mainstream RL libraries (e.g., Stable-Baselines3, RLlib). The framework adopts a modular design for the core RL elements:
   • Observation: Supports both local (single intersection) and global (multi-intersection) state representations, including features like queue length, phase state, density, and waiting times.
   • Action: Extends existing capabilities by supporting three distinct action types: Choose Next Phase, Switch Next or Not, and Set Phase Duration.
   • Reward: Computes multi-objective weighted rewards based on metrics such as internal/boundary waiting time, delay, throughput, and travel time.

The framework supports both single-agent and multi-agent training, allowing agents to learn coordinated strategies in complex, high-fidelity environments without requiring additional "glue code."

Key Contributions

• First Multi-Agent RL Framework for Vissim: VissimRL is presented as the first framework specifically designed for Vissim that supports multi-agent RL, enabling research in high-fidelity traffic simulation environments previously inaccessible to the RL community.
• Standardized and Modular Architecture: The framework provides standardized inheritance interfaces for states, actions, and rewards, significantly lowering the barrier to entry for researchers. It expands the supported performance metrics from six (in SUMO-RL) to eleven and introduces three distinct action types.
• Efficiency Optimization: By implementing batching and caching within the Vissim Wrapper, the framework reduces the number of COM API calls by up to 21% in complex workloads, resulting in lower latency and higher throughput compared to raw COM implementations.
• Development Efficiency: The wrapper reduces the development effort required to implement equivalent control tasks by approximately 84.5% (reducing Source Lines of Code from 207 to 32) compared to manual COM implementation.

Experimental Results
The authors evaluated VissimRL through two levels of assessment:

1. Wrapper Performance: Benchmarks comparing the Wrapper against Raw COM across three workloads (Step-only, Step+Signal Control, Step+Signal+Evaluation) demonstrated that the Wrapper introduces negligible overhead (4% increase) in simple stepping tasks but actually improves performance (6.8% to 12.3% lower latency) in interactive tasks due to reduced API call frequency.
2. RL Effectiveness: Experiments were conducted on synthetic networks (Single-Intersection and Arterial-3) and a real-world scenario (Dayuan Interchange in Taoyuan, Taiwan) using the Proximal Policy Optimization (PPO) algorithm.
   • Convergence: All three action designs showed consistent convergence and rapid improvement in delay, travel time, and waiting time across both single and multi-intersection scenarios.
   • Emergent Coordination: In the Arterial-3 multi-agent scenario, the system successfully learned a "Green Wave" phenomenon, where agents coordinated signal offsets to allow vehicles to pass through multiple intersections without stopping. The learned offsets and phase durations closely matched theoretical optima.
   • Real-World Application: In the Dayuan Interchange case study, PPO-based control reduced average delay by 63% (from 559.7s to 209.0s), travel time by 56%, and waiting time by 38% compared to fixed-time control.

Significance and Claims
The paper claims that VissimRL serves as a critical bridge between academic RL research and practical, industry-grade traffic simulation. By encapsulating the complexities of Vissim's COM interface while maintaining high runtime efficiency, the framework enables the application of RL in high-fidelity environments that better reflect real-world traffic dynamics. The authors assert that the framework demonstrates the feasibility of applying RL in such simulations and supports consistent traffic performance improvements and emergent multi-agent coordination. The work is positioned as an extended version of prior research, with the implementation made publicly available to foster reproducibility and further development in intelligent traffic signal control.