Excess demand in public transportation systems: The case of Pittsburgh's Port Authority

This paper proposes a framework using Poisson regression with censored data filtering to accurately estimate excess demand in public transportation systems, addressing the common issue of underestimation caused by unrecorded passengers left behind on full buses, and validates the approach using simulated data and real-world data from Pittsburgh's Port Authority.

The Gist

Imagine you are running a popular coffee shop. Every morning, a long line of people waits to buy a latte. You have a barista who can only serve 50 cups an hour.

If 60 people show up, 50 get their coffee, and 10 go home empty-handed. Now, imagine you are trying to figure out how popular your coffee shop is. You look at your sales log, and it says, "We sold 50 cups."

The Problem:
If you only look at that log, you might think, "Oh, only 50 people wanted coffee today." You wouldn't know that 10 people were actually turned away because you ran out of cups. In the world of public buses, this is exactly what happens. The bus company's computer records how many people got on the bus, but it has no way of knowing how many people were left standing at the bus stop because the bus was already full.

This paper is about figuring out how many people are being left behind at bus stops in Pittsburgh, even though the official records don't show them.

The "Censored" Mystery

The authors call this "censored data." Think of it like a movie where the screen goes black right before the hero gets shot. You know something happened, but you don't see the details.

In the bus system:

• The Reality: 100 people want to get on. The bus holds 50.
• The Record: The computer sees "50 people got on." It assumes only 50 people wanted to ride.
• The Consequence: If the bus company uses this bad data to plan for the future, they might think, "Hey, we only need 50 seats!" and send a tiny bus next time. But actually, they need a big bus for 100 people. This leads to more people being left behind, and eventually, people stop taking the bus altogether because it's unreliable.

The Detective Work: How They Solved It

The researchers (Tianfang, Robizon, Sera, and Konstantinos) built a framework to act like a detective. They couldn't see the people left behind, but they could look for clues.

The Clue:
They looked at two things:

1. How full was the bus when it arrived? (Was it packed to the brim?)
2. How many people got on? (Did the number drop to zero or stay low?)

The Analogy:
Imagine a bus arriving at a stop.

• Scenario A: The bus is half-empty, and nobody gets on. Conclusion: Nobody wanted to ride. (No excess demand).
• Scenario B: The bus is half-empty, and 10 people get on. Conclusion: 10 people wanted to ride. (Normal demand).
• Scenario C: The bus is completely full, and zero people get on. Conclusion: This is suspicious! Maybe 20 people were waiting, but the bus was too full to let them on. This is "Excess Demand."

The researchers created a filter to spot these "Suspicious Scenarios" (Scenario C). They realized that if they taught their computer model using these suspicious scenarios, the model would get confused and think, "Oh, nobody wants to ride during rush hour!" because it sees zero people getting on.

The Fix:
They told the computer: "Ignore the times when the bus was full and nobody got on. Don't use those to learn how many people usually want to ride." By removing these "censored" data points, the computer could learn the true demand.

The Simulation: Testing the Theory

Before looking at real Pittsburgh buses, they built a fake bus system in a computer (a simulation). They knew the exact truth: "We made 1,000 people wait."

They tested three ways to teach the computer:

1. The Perfect Teacher: Showed the computer the truth (1,000 people waited).
2. The Filtered Teacher: Showed the computer the data, but removed the "full bus" moments (like they planned to do with real data).
3. The Naive Teacher: Showed the computer everything, including the "full bus" moments.

The Result:
The Naive Teacher failed miserably. It thought demand was low because it saw empty buses during rush hour.
The Filtered Teacher did a great job. It was almost as good as the Perfect Teacher. This proved that their method of "filtering out the full bus moments" actually works to find the hidden demand.

The Real-World Findings in Pittsburgh

They applied this method to real data from Pittsburgh's Port Authority (PPA) for a whole year. Here is what they found:

• The "Hidden" Passengers: On average, about 1% of all passengers on the top 10 busiest routes were left behind at the stop because the bus was full.
• Rush Hour Reality: During the morning and evening rush, that number jumps to 8%. That means on a busy morning, 1 out of every 12 people waiting for the bus might be left behind.
• Seasonal Spikes: The problem gets worse in the Fall (when students return to school) and better in the Summer (when students are on break).
• The "Full Bus" Myth: They found that most of the time (98%), buses arrive at stops not full. The problem is very specific to certain stops, certain times, and certain routes. It's not a city-wide disaster, but a "pinch point" issue.

Why This Matters

This isn't just about math; it's about fairness and efficiency.

• For the City: If they know exactly where and when people are being left behind, they can send bigger buses or more frequent buses to those specific spots.
• For the Riders: It means a more reliable system. If you know the bus won't be full, you won't waste your time waiting.
• For the Environment: If the system is reliable, more people will choose the bus over driving their own cars, reducing traffic and pollution.

The Takeaway

The authors built a clever "X-ray" for bus data. Even though the official records are blind to the people left behind, this method allows us to see the invisible crowd. By filtering out the confusing data points where buses are full, they can predict the true demand and help Pittsburgh (and other cities) build a better, more reliable public transportation system.

As the former mayor of Bogota said, "An advanced city is not a place where the poor move about in cars, rather it's where even the rich use public transportation." But for the rich (and everyone else) to use the bus, the bus has to be able to fit them all. This paper helps make sure there's enough room.

Technical Summary

Here is a detailed technical summary of the paper "Excess demand in public transportation systems: The case of Pittsburgh's Port Authority."

1. Problem Statement

Public transportation reliability is often measured by schedule adherence, but a critical dimension is the system's ability to satisfy total passenger demand. When bus capacity is insufficient, passengers are left behind at stops, leading to a negative feedback loop where ridership drops due to poor service quality.

The core challenge addressed in this paper is quantifying "excess demand" (passengers unable to board due to full buses).

• Data Limitation: Standard transit data (specifically from bus systems using infrared sensors) only records passengers who successfully board. It does not record those left behind.
• Censoring Bias: An observation of "0 passengers boarding" does not necessarily mean "0 demand." It could mean the bus was full, or no one was waiting. Including these censored data points (where demand was truncated by capacity) in demand modeling leads to a systematic underestimation of true demand and excess demand.
• Goal: To design a framework that detects instances of potential excess demand, filters them from training data to prevent bias, and estimates the magnitude of the unmet demand.

2. Methodology

The authors propose a two-stage framework: Excess Demand Detection and Demand Modeling with Filtering.

A. Excess Demand Detection Mechanism

Since ground truth (actual number of people waiting) is unavailable in real-world data, the authors developed a heuristic binary classification mechanism to identify potential excess demand instances ($I_E$).

• Logic: An instance is flagged as potential excess demand if:
  1. The bus arrives at a stop with a load at or near its capacity limit.
  2. The number of passengers boarding is less than the number of passengers waiting (or zero boarding despite high load).
• Error Handling: The authors acknowledge this mechanism produces False Positives (bus full, but no one waiting) and False Negatives (bus full, driver allows a few to board). However, simulations suggest these errors tend to balance each other out regarding the total estimation of excess demand.

B. Data Filtering and Model Training

To avoid the bias introduced by censored data, the authors filter out the detected instances ($I_E$) from the training phase of the demand model.

• Simulated Validation: Using simulated data where ground truth is known, the authors compared three training scenarios:
  1. T1: Training on all data except true excess demand instances (Ideal).
  2. T2: Training on all data except detected excess demand instances ($I_E$) using the heuristic.
  3. T3: Training on all data (including censored points).
• Result: Models trained without filtering (T3) significantly underpredicted demand during peak hours because the model learned that "rush hour = low boarding" (due to capacity caps), whereas T1 and T2 correctly identified rush hour as high demand.

C. Model Selection

The authors evaluated four statistical models for count data to predict passenger boarding numbers ($Y_{ijt}$) based on features like time of day, inter-arrival times, and local passenger load:

1. Poisson Regression: Assumes mean equals variance.
2. Negative Binomial Regression: Handles overdispersion (variance > mean).
3. Zero-Inflated Poisson (ZIP): Handles excess zeros in the data.
4. Hierarchical Model: A two-step approach (Logistic for probability of excess demand + Poisson for rate).

Selection Criteria: The models were trained on 80% of the Pittsburgh Port Authority (PPA) data (excluding $I_E$) and validated on the remaining 20%. Performance was measured using Root Mean Squared Error (RMSE).

3. Key Contributions

1. Framework for Censored Data: The paper provides a novel methodology to handle censored transit data by identifying and filtering capacity-constrained observations before model training.
2. Bias Quantification: Through simulation, the authors demonstrated that failing to filter censored data leads to a negative bias in demand estimation, particularly misinterpreting peak hours as low-demand periods.
3. Empirical Estimation of Excess Demand: The study is one of the first to quantify excess demand in a real-world bus system (Pittsburgh) over a full year, moving beyond theoretical models or small-scale studies.
4. Model Selection for Transit: The authors rigorously tested multiple count data models, finding that Poisson regression consistently provided the lowest out-of-sample RMSE for the PPA dataset, despite the presence of overdispersion in some contexts.

4. Results

The framework was applied to the top 10 busiest routes in the Pittsburgh Port Authority system using data from 2018.

• Model Performance: The Poisson regression model outperformed Negative Binomial, ZIP, and Hierarchical models across low, medium, and high-demand stops.
• Excess Demand Magnitude:
  • Annual Average: Approximately 1% of total passengers on a route are left behind at stops due to full buses over the course of a year.
  • Peak Hours: During rush hours, this fraction rises significantly, reaching up to 8% of total passengers.
• Seasonality: Excess demand exhibits strong seasonality. It is lowest in summer (due to university recess) and December (holidays), and peaks in the fall months (September) when students return.
• Spatial/Temporal Heterogeneity: Peak hours vary significantly by specific stop and direction (inbound vs. outbound), challenging the notion of uniform "rush hours" across a route. For example, the 61D route showed different overload patterns for inbound (morning peak) vs. outbound (evening peak) stops.

5. Significance and Implications

• Operational Insight: The findings provide transit agencies with a data-driven method to identify specific routes and times where capacity is insufficient, justifying the need for increased frequency or larger vehicles (e.g., articulated buses) during specific windows (e.g., fall rush hours).
• Methodological Rigor: The paper highlights the danger of using raw transit data for demand forecasting without accounting for capacity constraints. It establishes a best practice: detect and filter capacity-constrained observations to build accurate demand models.
• Scalability: While the specific results (1% vs 8%) are unique to Pittsburgh, the framework is generic. It can be applied to other bus systems, provided they have load data (via sensors or ticketing) and capacity limits.
• Limitations: The authors note that the assumption that excess demand follows the same distribution as observed demand is an implicit assumption that requires field validation. Additionally, the detection mechanism relies on heuristic rules that may not perfectly capture driver behavior (e.g., drivers allowing some passengers to board even when full).

In conclusion, this study bridges the gap between theoretical demand modeling and the practical reality of capacity-constrained public transit, offering a robust statistical approach to quantify the "invisible" demand that is currently lost in standard transit data.