Competitive Multi-Operator Reinforcement Learning for Joint Pricing and Fleet Rebalancing in AMoD Systems

Imagine a bustling city where two rival companies, RideCo and GoGo, are launching fleets of self-driving taxis. In the past, researchers mostly studied what happens if one company controls all the taxis (a monopoly). But in the real world, it's a free-for-all. This paper asks a big question: What happens when two self-driving taxi companies try to outsmart each other using AI?

Here is the story of their competition, explained simply.

The Setup: A High-Stakes Chess Game

Think of the city as a giant chessboard.

The Pieces: The self-driving cars.
The Players: RideCo and GoGo.
The Goal: Make as much money as possible.
The Moves:
1. Pricing: Should we charge $10 or $15 for a ride?
2. Rebalancing: If all our cars are stuck in the downtown area, should we send empty ones to the suburbs to wait for new customers?

In a monopoly (one company), the AI just tries to be efficient. But in this paper, the AI has to play Chess against another AI. Every time RideCo lowers its price, GoGo loses customers. If GoGo moves its cars to a busy neighborhood, RideCo might lose out. They have to guess what the other guy is doing.

The Secret Weapon: The "Smart Shopper"

The researchers added a special rule to the game called the "Smart Shopper Model."

Imagine a passenger standing on a street corner. They see two apps: RideCo and GoGo. They don't just pick the first one; they calculate:

"How much does it cost?"
"How long will I wait?"
"How much money do I make an hour? (If I'm rich, I might pay more to save time. If I'm on a budget, I'll wait longer for a cheaper ride.)"

The AI learns that passengers are rational. If RideCo charges too much, the "Smart Shopper" switches to GoGo. This forces the AI to constantly adjust its strategy based on what the competitor is doing.

The Big Discovery: Competition Changes Everything

The researchers ran simulations in three real cities: San Francisco, Washington D.C., and New York City. Here is what they found:

1. The Price War (The "Race to the Bottom")
When there is only one company, it can charge high prices. But when two companies fight, they start slashing prices to steal customers.

Analogy: It's like two lemonade stands on the same street. If one lowers the price to 50 cents, the other has to drop to 45 cents to stay in business.
Result: Passengers win because rides become cheaper. But the companies make less profit overall.

2. The "Empty Car" Problem
In a monopoly, one company can perfectly position all its cars to cover the whole city. In a duopoly (two companies), they get in each other's way.

Analogy: Imagine two delivery drivers trying to cover the same neighborhood. They might both send a truck to the same house, leaving other houses empty.
Result: Wait times go up slightly because the fleet is "fragmented." The system is less efficient than if one big boss controlled it all.

3. Different Cities, Different Strategies
The AI learned that different cities need different tactics:

San Francisco (Chaotic & Unpredictable): The AI learned that moving cars around (rebalancing) is the most important weapon. You have to be where the demand is before it happens.
New York City (Dense & Stable): Here, the AI learned that price is the main weapon. Since demand is everywhere, you don't need to move cars as much; you just need to be slightly cheaper than the other guy to get the ride.

The "Magic" of the AI

The most impressive part of the paper is that these AI agents learned to play this game without being told the rules of competition.

They started with no idea how the other company would act.
They just tried things, got rewarded when they made money, and punished when they didn't.
The Result: They figured out complex strategies on their own. They learned to "undercut" the competitor in specific neighborhoods (slightly lowering prices just enough to steal a few customers) without starting a price war that hurts everyone.

The Takeaway

This paper proves that competition makes self-driving taxi systems cheaper for you (the passenger) but slightly messier for the system.

For the Passenger: You get lower prices.
For the City: You might wait a few minutes longer because the cars aren't perfectly organized.
For the Companies: They have to be smarter and more reactive than ever before.

The researchers concluded that even though the competition adds chaos (like two people trying to dance to the same song without hearing each other), the AI is robust enough to learn the steps and keep the dance going without tripping over. It's a win for the future of urban transport, showing that we can have a competitive market that still works efficiently.

Here is a detailed technical summary of the paper "Competitive Multi-Operator Reinforcement Learning for Joint Pricing and Fleet Rebalancing in AMoD Systems."

1. Problem Statement

The paper addresses the control of Autonomous Mobility-on-Demand (AMoD) systems in a competitive market environment. While existing research largely focuses on single-operator (monopolistic) settings where a central entity optimizes fleet rebalancing and pricing, realistic future AMoD markets will likely involve multiple operators competing for passengers.

The core challenge is that optimal policies in a competitive setting depend not only on demand and supply but also on competitor behavior (e.g., price cuts by one operator divert passengers from another). The authors investigate how competition impacts the learning of joint pricing and fleet rebalancing policies, specifically asking:

Can Reinforcement Learning (RL) agents converge to effective strategies when competing against another learning agent?
How does competition alter learned behaviors, market efficiency, and service quality compared to monopolistic settings?
How do factors like fleet size asymmetry and regional wage heterogeneity influence equilibrium outcomes?

2. Methodology

A. System Modeling

The authors model the environment as a directed graph $G=(V, E)$ representing spatial regions. The system involves two independent operators ( $o \in \{0, 1\}$ ), each controlling a fleet of autonomous vehicles ( $M_0, M_1$ ).

Time Horizon: Discrete time steps (3 minutes).
State Space ( $S$ ): Includes network topology, idle vehicle counts, vehicles en route, queue lengths, own/competitor prices, and historical demand. Operators do not share vehicle location or demand data but can observe competitor prices.
Action Space ( $A$ ): Each operator outputs:
1. Origin-based price scalars ( $\rho$ ): Multiplied by historical base prices to determine fares.
2. Desired idle-vehicle distribution ( $w$ ): A probability distribution used to solve a minimum-cost flow problem for rebalancing.

B. Demand and Choice Mechanism

A critical innovation is the integration of Discrete Choice Theory (Multinomial Logit model) to endogenously allocate demand.

Passengers choose between Operator 0, Operator 1, or an alternative mode based on Utility ( $U$ ).
Utility is a function of: Trip price, estimated travel time, and passenger hourly wage (capturing income effects and price sensitivity).
This creates a feedback loop where pricing strategies directly influence market share.

C. Reinforcement Learning Framework

The problem is formulated as a Multi-Agent Markov Decision Process (MMDP).

Algorithm: A2C (Advantage Actor-Critic).
Architecture: Each operator maintains independent Actor-Critic networks with no parameter sharing.
- Encoder: Graph Convolutional Networks (GCN) to capture spatial dependencies in the transportation network.
- Actor: Outputs parameters for stochastic policies. Pricing scalars are sampled from a Beta distribution, and vehicle distribution weights are sampled from a Dirichlet distribution.
- Critic: Aggregates node embeddings to estimate value.
Reward Function: Defined as Profit = (Trip Revenue) - (Operational Costs) - (Rebalancing Costs).

D. Simulation Setup

Experiments were conducted using real-world taxi trip data from three cities: San Francisco, Washington DC, and NYC Manhattan South. The simulation covers peak evening hours (19:00–20:00) with Poisson-distributed demand.

3. Key Contributions

Competitive Multi-Operator RL Framework: The first framework to jointly learn pricing and rebalancing policies for two competing AMoD operators, extending previous monopolistic RL models to competitive markets.
Endogenous Demand Allocation: Integration of a wage-dependent discrete choice model that allows passenger allocation to emerge naturally from utility maximization, rather than being fixed or exogenous.
Empirical Analysis of Competition: A comprehensive study demonstrating that competition fundamentally alters learned strategies, leading to lower prices and distinct fleet positioning patterns compared to monopoly.
Robustness Validation: Demonstration that learning-based agents successfully converge to effective policies despite the added stochasticity of competitor actions and partial observability.

4. Key Results

A. Monopoly vs. Competition

Monopoly: Joint control (pricing + rebalancing) consistently outperforms single-mode strategies (pricing-only or rebalancing-only) across all cities.
Competition:
- Price Reduction: Competition drives prices down significantly (up to 27% lower in San Francisco) compared to monopolistic settings.
- Strategy Divergence: In high-demand-variability environments (e.g., San Francisco), fleet positioning (rebalancing) becomes the primary competitive lever. In stable, high-density environments (e.g., NYC), pricing becomes the dominant lever.
- Profit Impact: Competition generally reduces total operator profits (e.g., 15.2% loss in SF, 7.1% in DC) due to fragmented fleet management and price wars, though in NYC, competitive pricing slightly stimulated demand, mitigating losses.

B. Service Quality and Efficiency

Wait Times: Wait times generally increase in competitive settings due to the inefficiency of fragmented fleet management (operators cannot coordinate to minimize global wait times).
Market Share: Profits are split relatively evenly between operators (differences typically <2%), indicating a balanced equilibrium.

C. Sensitivity Analyses

Fleet Size: As fleet size increases, operators dynamically lower prices to maintain utilization. Total reward peaks at a specific fleet size (1,050 vehicles in NYC) before diminishing returns set in due to high rebalancing costs.
Asymmetric Fleets: When fleet sizes are unequal (e.g., 1:9 split), the smaller operator raises prices to maximize margin, while the larger operator undercuts prices to capture volume. This reduces direct price competition compared to symmetric splits.
Wage Heterogeneity: In regions with varying incomes, operators reposition fleets toward high-income areas and increase prices to exploit higher willingness to pay.

D. Information Conditions

Experiments showed that system performance is robust whether operators can observe competitor prices or not. This suggests that in simultaneous learning from scratch, competitor prices act more as noise than actionable signals, and agents learn to adapt based on their own state and reward feedback.

5. Significance and Conclusion

This paper bridges a critical gap in AMoD research by moving from theoretical monopolistic control to realistic competitive markets.

Theoretical Impact: It validates that Multi-Agent RL can handle the non-stationarity introduced by competing learning agents in complex, spatial-temporal environments.
Practical Implications:
- For Operators: Joint control is essential, but the optimal mix of pricing vs. rebalancing depends on market volatility.
- For Policymakers: Competition benefits consumers through lower fares but may degrade service quality (higher wait times) due to lack of coordination.
- Future Directions: The authors suggest exploring waiting time trade-offs, asymmetric learning architectures, and the modeling of collusive behavior.

In summary, the study demonstrates that while competition introduces complexity and reduces operator profits, RL agents can successfully learn robust, adaptive strategies that balance pricing and fleet deployment in dynamic, multi-operator urban environments.