🤖 machine learning

RE-SAC: Disentangling aleatoric and epistemic risks in bus fleet control: A stable and robust ensemble DRL approach

This paper proposes RE-SAC, a robust ensemble deep reinforcement learning framework that explicitly disentangles aleatoric and epistemic uncertainties through IPM-based regularization and Q-ensemble diversification to achieve superior stability and performance in bus fleet holding control under volatile traffic conditions.

Original authors: Yifan Zhang, Liang Zheng

Published 2026-03-20

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yifan Zhang, Liang Zheng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the traffic controller for a busy city bus line. Your job is to decide when to make a bus wait at a stop (hold it) and when to let it go, so that all the buses stay evenly spaced apart.

If you get it right, passengers wait a consistent amount of time, and no two buses arrive at once (a phenomenon called "bunching"). If you get it wrong, buses clump together, leaving long gaps where no bus arrives, and the whole system collapses.

The problem is that the real world is chaotic.

Aleatoric Uncertainty (The "Noise"): Sometimes a bus is late because of a sudden rainstorm, a red light that stays red too long, or a crowd of passengers getting on. This is unavoidable randomness. You can't predict it perfectly, no matter how much data you have.
Epistemic Uncertainty (The "Ignorance"): Sometimes a bus faces a situation the controller has never seen before. Maybe a massive parade blocks the road, or a bus breaks down in a weird spot. This is lack of data. The controller doesn't know what to do because it hasn't happened yet.

The Problem: The "Confused Brain"

In the past, AI controllers (using Deep Reinforcement Learning) tried to handle these two types of uncertainty as if they were the same thing. They treated random noise (rain) and total ignorance (a parade) as a single "risk."

This caused a brain glitch:

The AI saw a noisy situation (like a rainy day) and thought, "I don't know what to do! This is dangerous!"
It became too pessimistic. It started thinking, "I should just stop the bus forever to be safe," or it gave up on good strategies it had learned before.
The result? The bus system crashed. The AI stopped making smart decisions because it was too scared of the noise.

The Solution: RE-SAC (The "Smart Team")

The authors of this paper created a new AI framework called RE-SAC. Think of it as a specialized team of bus controllers who have learned to separate "noise" from "ignorance."

Here is how they do it, using two simple tools:

1. The "Smooth Operator" (Handling Noise)

To deal with Aleatoric Uncertainty (the rain, the traffic jams), the team uses a mathematical trick called IPM Regularization.

The Analogy: Imagine a nervous driver who jerks the steering wheel every time a bird flies by. That's a bad driver. The "Smooth Operator" is a driver who knows that birds fly by all the time. They don't overreact.
How it works: The AI is forced to be "smooth." It learns that small, random changes in the environment shouldn't cause huge swings in its decisions. It ignores the static noise so it doesn't get scared.

2. The "Skeptical Committee" (Handling Ignorance)

To deal with Epistemic Uncertainty (the parade, the unknown), the team uses a Q-Ensemble.

The Analogy: Instead of one boss making a decision, imagine a committee of 10 experts.
- If they all agree, "Yes, this is a safe road," the AI is confident.
- If they all disagree (some say "Go!", some say "Stop!"), the AI knows, "Wait, we haven't seen this before. I don't know enough."
How it works: When the AI sees a situation it hasn't seen much in its training data, the committee gets confused. The AI uses this confusion as a signal to be cautious (pessimistic) only in those specific unknown areas, rather than being scared of everything.

Why This Matters

The paper tested this new "Smart Team" in a realistic simulation of a bus line.

The Old Way (Vanilla SAC): The AI got confused by the noise and the unknown, leading to a "value collapse" where it stopped working properly.
The "Skeptical Committee" Only: If you only used the committee without the "Smooth Operator," the AI got too scared of the normal noise (like rain) and thought it was a disaster, causing the system to crash.
The RE-SAC Way: By separating the two, the AI learned:
- "Oh, it's raining? That's just noise. Keep driving normally."
- "Oh, there's a parade? I've never seen that. Let's be very careful."

The Result

The RE-SAC system kept the buses running smoothly even in the worst traffic conditions. It didn't panic at the noise, and it didn't get overconfident in the unknown. It found the perfect balance, keeping the bus lines running on time and preventing the dreaded "bus bunching."

In short: The paper teaches AI how to tell the difference between "just a little chaos" and "I have no idea what's happening," so it doesn't freeze up when things get messy.

1. Problem Statement

The paper addresses the challenge of bus holding control in urban transit systems, specifically the phenomenon of bus bunching (where multiple buses cluster together due to positive feedback loops of delay and passenger accumulation). While Deep Reinforcement Learning (DRL) has been applied to this problem, standard Actor-Critic algorithms (like Soft Actor-Critic, SAC) suffer from Q-value instability and "value collapse" in highly stochastic environments.

The authors identify a critical bottleneck: the conflation of two distinct types of uncertainty in standard DRL:

Aleatoric Uncertainty: Irreducible environmental noise (e.g., random passenger arrivals, traffic speed fluctuations).
Epistemic Uncertainty: Lack of model knowledge due to data insufficiency in specific regions of the state-action space (e.g., rare extreme bunching scenarios).

Treating these as a single risk leads to pathological underestimation in well-explored but noisy states (mistaking noise for data gaps) and dangerous optimism in unvisited states. This results in "Q-value poisoning," where the agent fails to maintain service regularity despite having encountered optimal trajectories.

2. Methodology: RE-SAC Framework

The authors propose RE-SAC (Robust Ensemble Soft Actor-Critic), a framework that explicitly disentangles and addresses both uncertainty sources using a dual-mechanism approach grounded in Robust Markov Decision Processes (RMDP).

A. Theoretical Foundation

The method leverages the equivalence between robust optimization and function regularization. It decomposes the Robust Bellman Operator into two distinct penalty terms:
$T^{REV}(s, a) = R(s, a) + \gamma \left( \mathbb{E}[V(s')] - \lambda_{epi} \cdot \Gamma_{epi} - \lambda_{ale} \cdot \Gamma_{ale} \right)$

B. Mechanism 1: Aleatoric Risk Mitigation (IPM Regularization)

Goal: Handle inherent environmental noise without requiring expensive inner-loop state perturbations.
Approach: Uses Integral Probability Metric (IPM) duality to derive a lower bound for the robust Bellman operator.
Implementation: Instead of sampling perturbations, the method applies $\ell_1$ -norm weight regularization to the critic network. This enforces Lipschitz continuity, ensuring the value function is smooth and less sensitive to small input perturbations (aleatoric noise).
Effect: Prevents the Q-value from collapsing due to high-variance noise in well-explored states.

C. Mechanism 2: Epistemic Risk Mitigation (Q-Ensemble)

Goal: Handle data insufficiency and "unknown unknowns" (Out-of-Distribution states).
Approach: Employs a diversified Q-ensemble (multiple independent Q-networks).
Implementation: The variance among the ensemble members ( $\Gamma_{epi}$ ) serves as a proxy for epistemic uncertainty. A penalty is subtracted from the value estimate proportional to this variance.
Effect: Enforces "pessimism in the face of the unknown," discouraging the agent from selecting actions in sparsely sampled regions where the model is overconfident.

D. Synergy and Architecture

Disentanglement: By separating the penalties, RE-SAC prevents the ensemble from misinterpreting inherent noise as a data gap (a failure mode observed in "Epistemic-Only" ablations).
State Representation: Uses categorical embeddings for discrete features (Bus ID, Stop ID) combined with continuous features (headways, speed). This allows the ensemble variance to naturally scale with the sampling frequency of specific entities.
Algorithm: Integrates these components into the SAC architecture with a Lower Confidence Bound (LCB) objective for the actor to promote exploration in safe, well-explored regions.

3. Key Contributions

Theoretical Disentanglement: Provides a formal motivation and mathematical framework for decoupling aleatoric and epistemic risks in transit control, bridging risk-sensitive RL and service reliability.
Analytic Robustness: Introduces an IPM-based weight regularization as a computationally efficient alternative to conservative smoothing or inner-loop maximization, providing a structural prior against noise.
Empirical Stability: Demonstrates that the dual-mechanism approach prevents the "underestimation trap" and catastrophic value collapse, maintaining superior service regularity compared to state-of-the-art baselines.

4. Experimental Results

The framework was evaluated on a high-fidelity bidirectional bus corridor simulation (22 stops, 13-hour window, stochastic passenger demand and traffic).

Performance: RE-SAC achieved the highest cumulative reward (approx. $-0.4 \times 10^6$ ) compared to Vanilla SAC ( $-0.55 \times 10^6$ ) and BAC.
Ablation Studies:
- Epistemic-Only: Suffered catastrophic collapse (reward dropped to $-1.2 \times 10^6$ ), confirming that treating noise as data gaps leads to policy failure.
- Aleatoric-Only: Failed to detect OOD states, leading to suboptimal convergence.
- RE-SAC: Successfully balanced both risks, showing stable learning curves.
Robustness in Rare States (OOD):
- Using Mahalanobis rareness analysis, the authors measured Q-value estimation error against ground-truth Monte Carlo returns.
- RE-SAC reduced the Oracle Mean Absolute Error (MAE) by 62% in rare states compared to Vanilla SAC (1647 vs. 4343), demonstrating superior worst-case robustness.

5. Significance

This paper makes a significant contribution to the application of RL in safety-critical, high-variance domains like public transit.

Solves the "Value Collapse" Problem: It identifies and fixes a specific failure mode where standard robust RL fails in stochastic environments by misidentifying noise as uncertainty.
Scalability: By replacing expensive inner-loop robustness checks with analytic weight regularization, the method remains computationally feasible for real-world deployment.
Practical Impact: The results suggest that explicitly modeling the source of uncertainty (noise vs. lack of data) is crucial for maintaining reliable bus schedules, potentially reducing passenger wait times and improving system efficiency in real-world transit networks.