💻 computer science

Action-masked deep Q-learning for optimizing full-length and short-turn urban rail services

This paper proposes an action-masked deep Q-learning framework to optimize urban rail services by dynamically selecting between full-length and short-turn patterns, which significantly reduces passenger waiting times and increases service capacity compared to traditional fixed or threshold-based policies while ensuring operational feasibility.

Original authors: Yibo Wang, Zhengfeng Ma, Rongjie Chen

Published 2026-08-12

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Yibo Wang, Zhengfeng Ma, Rongjie Chen

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the conductor of a massive, invisible orchestra. Your instruments are trains, your sheet music is the schedule, and your audience is thousands of commuters. But here's the twist: the audience doesn't show up in a neat, predictable line. Sometimes, a whole stadium of people rushes to the center of the city at 8:00 AM, while the outer neighborhoods are empty. Other times, the crowd shifts to a specific subway station, leaving the rest of the line quiet. If you keep playing the same song (running the same number of trains on the same route) all day, you'll either have empty trains wasting fuel or a chaotic crush of people waiting on the platform. This is the daily puzzle of urban rail transit: how to match the music to the crowd without breaking the rules of the orchestra.

To solve this, scientists use a branch of artificial intelligence called Deep Reinforcement Learning. Think of this as a video game where a computer agent learns to play by trial and error. It tries different moves, gets points for good ones (like keeping passengers happy), and loses points for bad ones (like making people wait too long). Over time, it learns the best strategy. However, in the real world, you can't just try anything. You can't run a train that doesn't exist, or turn a train around at a station that doesn't have the space to do so. This is where Action Masking comes in. Imagine a video game controller that physically blocks your thumb from pressing buttons that would make your character fall off a cliff. The AI is only allowed to press the "legal" buttons, ensuring it never tries to do something impossible. This paper asks: Can we teach a digital conductor to juggle two types of train services—long ones that go the whole way and short ones that turn around early—while strictly obeying these "no-fall" rules?

The Digital Conductor's New Trick

In this study, researchers Yibo Wang, Zhengfeng Ma, and Rongjie Chen built a digital simulation of a 12-station subway line to test a new kind of AI conductor. They wanted to see if an Action-Masked Deep Q-Network (DQN) could dynamically decide whether to run full-length trains (going from start to finish) or short-turn trains (turning back early to serve the busy middle section) based on how many people were waiting.

The AI agent acts like a smart dispatcher. At every step, it looks at the crowd, the number of available trains, and the physical limits of the tracks (like how fast a train can turn around). It then chooses from four possible moves: keep the current service, switch to a short-turn, increase the frequency of trains, or decrease it. The "Action Mask" is the safety guard that steps in before the AI makes a move. If the AI tries to pick a move that violates a rule—like trying to turn a train around when the station is full, or running more trains than the fleet has—the mask blocks that choice instantly. The AI is forced to pick only from the list of "legal" moves.

The Results: A Smarter, Faster Ride

When the researchers pitted their new AI against three other strategies (a fixed schedule that never changes, a fixed short-turn schedule, and a simple rule-based system), the results were striking. In a simulation of a typical weekday, the Action-Masked DQN achieved a mean reward of 93.22, while the other strategies all scored negative numbers (meaning they performed poorly).

The improvements were massive:

Waiting Time: The AI reduced the waiting-time metric by 83.12% compared to a fixed full-length schedule, 62.78% compared to a fixed short-turn schedule, and 76.84% compared to the rule-based system.
Passengers Served: It managed to serve 19.42% more passengers than the fixed full-length plan, 17.39% more than the fixed short-turn plan, and 5.08% more than the rule-based plan.

The AI learned to be a shapeshifter. During peak hours, it deployed more full-length trains to handle the rush to the city center. During quieter times, it switched to more short-turn trains to keep the core section busy without wasting energy on empty outer tracks. It successfully served 36,923.33 passengers in the simulation, with zero violations of the safety rules.

The Twist: Safety vs. Speed

Here is where the story gets interesting. The researchers wanted to know if the "safety guard" (the action mask) was actually helping the AI learn better, or if it was just a rule-enforcer. To find out, they ran a test without the mask, letting the AI try any move, even the illegal ones, and then punishing it heavily for doing so.

Surprisingly, the unmasked AI actually performed slightly better in terms of reward and waiting time in this specific test. The unmasked version had a reward of -1577.80 compared to the masked version's -1870.01, and it reduced the waiting-time metric by 13.55% more. The unmasked AI also served a tiny bit more traffic.

So, what's the point of the mask? The paper suggests that while the mask didn't make the AI "smarter" in terms of raw points in this specific simulation, it guaranteed that the AI never tried to break the rules. The unmasked AI had an invalid-action rate of 0.2274 (meaning it tried to do illegal things about 22% of the time during training), whereas the masked AI had a rate of 0. The authors conclude that the mask is crucial for feasibility—ensuring the plan can actually be built and run in the real world—rather than just maximizing points. It forces the AI to stay within the "playground" of reality.

Testing the Limits

The researchers didn't stop at a normal day. They threw curveballs at the AI:

Sudden Surges: When a massive crowd appeared at the interchange station, a fixed short-turn plan actually did slightly better than the AI, suggesting that sometimes simple, location-specific rules beat complex learning.
Broken Tracks: When they simulated reduced capacity for turning trains around, the AI adapted perfectly, while the fixed plans struggled.
Wrong Guesses: Even when the AI was given a 20% error in its demand predictions, it still outperformed the other methods.

However, the study also found limits. When the number of available trains was cut down to 16, the waiting time more than doubled compared to having 18 trains, showing that no amount of AI magic can fix a shortage of physical trains.

The Takeaway

This paper doesn't claim to have solved the problem of urban rail forever. Instead, it suggests that Action-Masked Deep Q-Learning is a powerful tool for creating service plans that are both responsive to crowds and strictly safe. The AI learned to balance the trade-off between serving more people and keeping trains from running empty, all while obeying the strict laws of the railway. While the "safety guard" didn't always make the AI score higher points in the simulation, it ensured that every decision the AI made was something a real human dispatcher could actually execute. In the chaotic, crowded world of city transit, that guarantee of feasibility might be just as important as the speed of the ride.

Technical Summary: Action-Masked Deep Q-Learning for Optimizing Full-Length and Short-Turn Urban Rail Services

Problem Formulation
Urban rail operators face the challenge of adapting service plans to time-varying passenger demand while adhering to strict operational constraints. Passenger demand is unevenly distributed across time and space, creating capacity imbalances where central sections are congested during peaks while peripheral sections suffer from underutilization. While combining full-length and short-turn services offers a solution to balance service frequency and efficiency, jointly accounting for dynamic demand, turn-back capacity, rolling-stock availability, and minimum service-switching intervals significantly increases decision complexity.

The authors formulate this optimization task as a Constrained Markov Decision Process (MDP). The agent (dispatching module) operates within an environment comprising passenger arrivals, train movements, queues, and constraints. The state space includes time periods, station-level demand, platform queues, current service patterns, headways, and constraint statuses. The action space consists of four discrete decisions: retaining/adopting a full-length service, retaining/adopting a short-turn service, increasing frequency, or decreasing frequency.

Methodology: Action-Masked Deep Q-Network (DQN)
The core methodological contribution is an Action-Masked Deep Q-Network. Unlike conventional DQNs that estimate Q-values for all actions and may select infeasible decisions during exploration, this approach explicitly restricts the action space at every decision step.

Feasibility Masking: Before action selection, the environment constructs a set of feasible actions based on current operational limits:
- Headway limits: Minimum and maximum intervals between trains.
- Turn-back capacity: Limits on the number of trains that can reverse at intermediate stations.
- Rolling-stock availability: The number of available trains.
- Switching intervals: Minimum time required between changing service patterns.
  Both random exploration and greedy exploitation are restricted to this feasible set, ensuring that the agent never attempts to execute an invalid operation.
Reward Function: The reward function is a weighted sum designed to balance passenger service, operating costs, and feasibility. It includes terms for:
- Passengers served (weight: 0.30)
- Waiting time (weight: 0.25)
- Transfer effects (weight: 0.20)
- Underutilization (weight: 0.15)
- Service-pattern stability (weight: 0.10)
- Penalties for constraint violations ( $R_c$ ).
Training Architecture: The model utilizes a fully connected neural network with four outputs. Training employs experience replay, a target network, and the Adam optimizer to stabilize value estimation.

Experimental Setup
The study was simulated in Python 3.10 using PyTorch 2.0 on a 12-station urban rail line. The simulation included:

Passenger Dynamics: Poisson arrival processes and origin-destination matrices.
Train Parameters: 6-car trains (200 passengers/car) with specific headways for peak (4 min), inter-peak (6 min), and off-peak (8 min) periods.
Service Topology: Full-length services cover all 12 stations; short-turn services reverse after the 9th station (interchange).
Comparators: The proposed method was compared against three baselines: Fixed Full-Length, Fixed Short-Turn, and a Threshold Heuristic (rule-based).
Scenarios: Evaluations included a typical weekday, peak-demand surges, interchange-demand surges, reduced turn-back capacity, and 20% demand-prediction errors.

Key Results
Under a typical weekday demand profile, the Action-Masked DQN achieved a mean reward of 93.22, significantly outperforming all reference policies (which yielded negative rewards).

Performance Gains: Compared to Fixed Full-Length, Fixed Short-Turn, and Threshold policies, the proposed method reduced the waiting-time metric by 83.12%, 62.78%, and 76.84%, respectively. It increased the number of passengers served by 19.42%, 17.39%, and 5.08%, respectively.
Constraint Adherence: The method achieved zero constraint violations across all tested scenarios, including perturbations in demand and capacity.
Ablation Study: A critical finding from the ablation study (comparing masked vs. unmasked DQN) revealed that while the action mask successfully reduced the invalid-action rate during training from 0.2274 to 0, it did not automatically result in higher rewards or lower waiting times compared to the unmasked DQN under the specific training configuration used. The unmasked DQN actually achieved a slightly less negative reward and lower waiting time, though with a higher invalid-action rate.
Sensitivity: The method remained robust across various perturbations, though the fixed short-turn policy outperformed the DQN in a specific "interchange-demand surge" scenario, highlighting that spatial demand structure remains a critical factor.

Significance and Claims
The paper claims that the primary contribution of the action mask is guaranteeing operational feasibility rather than inherently maximizing reward. By removing infeasible actions before exploration, the method aligns the decision process with real-world operating rules, preventing the agent from learning policies that violate safety or capacity limits.

The authors modestly conclude that while action masking enables demand-responsive service selection, its impact on overall policy performance (reward/waiting time) depends heavily on training configurations, reward specification, and hyperparameters. The study demonstrates that the method can generate feasible service plans that adapt to demand, but it does not claim to be a universal superior optimizer over all baselines in all scenarios. The work establishes a framework for embedding constraints directly into the reinforcement learning action-selection process, though the authors note limitations regarding the use of synthetic data, the lack of uncertainty intervals in the evaluation, and the current restriction to a single line with discrete actions.