⚡ electrical engineering

Traffic-Aware Pedestrian Intention Prediction

This paper proposes a Traffic-Aware Spatio-Temporal Graph Convolutional Network (TA-STGCN) that integrates dynamic traffic signal states and bounding box features to significantly improve pedestrian intention prediction accuracy for autonomous vehicles, achieving a 4.75% gain over baseline models on the PIE dataset.

Original authors: Fahimeh Orvati Nia, Hai Lin

Published 2026-07-15

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Fahimeh Orvati Nia, Hai Lin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to guess what a friend is about to do next. Are they going to wave hello, or are they about to sprint across a busy street? To make a good guess, you don't just look at your friend's legs; you look at the whole picture. You check if the traffic light is red or green, you see how close they are to the curb, and you notice if a car is speeding toward them. This is the heart of a field called "pedestrian intention prediction." It's a branch of science dedicated to teaching computers, specifically self-driving cars, to understand human behavior. The goal is simple but life-saving: if a car can predict that a person intends to cross the road, it can slow down or stop before an accident happens. For years, computers have been trying to learn this by watching people move, but they often missed the most important clues: the traffic signals and the messy, dynamic context of a real city street.

This paper introduces a new, smarter way for computers to read these clues. The researchers, Fahimeh Orvati Nia and Hai Lin, built a model they call TA-STGCN (Traffic-Aware Spatio-Temporal Graph Convolutional Network). Think of this model as a super-observant detective that doesn't just watch a person; it watches the person and the traffic lights and the cars all at once, understanding how they all talk to each other over time.

The Problem: Computers Missing the Big Picture

For a long time, computers trying to predict pedestrian behavior were a bit like a student who only studied the textbook but never went outside. Early methods relied on simple math to track where a person was and how fast they were moving. Later, deep learning models (like CNNs and LSTMs) got better at spotting patterns in video, but they still often ignored the "scene." They might see a person walking toward a street, but they wouldn't necessarily notice that the traffic light is currently red, or that the person is standing right next to a crosswalk.

The authors argue that this is a big mistake. In the real world, a pedestrian's decision to cross is heavily influenced by their environment. If the light is green for cars, a pedestrian usually waits. If the light turns red for cars (and green for them), they might step out. Existing models often missed these dynamic signals, leading to guesses that were too slow or just plain wrong.

The Solution: A Detective with a Traffic Light Map

The team's solution, TA-STGCN, is designed to fix this by treating the street scene like a giant, living puzzle where every piece is connected. They use a "Spatio-Temporal Graph," which is a fancy way of saying they draw invisible lines connecting the pedestrian to everything else around them: the traffic lights, the cars, and the crosswalks.

Here is how the model works, broken down into simple parts:

The Two Streams of Information: The model looks at the scene in two different ways at the same time.
- The "Where" Stream: It tracks the exact location of the pedestrian and how their "box" (the digital outline around them) changes size. If the box gets bigger, it means the person is getting closer to the car.
- The "What" Stream: It looks at what the objects are and what state they are in. Crucially, this includes the traffic light status (Red, Yellow, or Green).
The Graph Connection: Imagine a spiderweb connecting the pedestrian to the traffic light. The model uses a special type of math (Graph Convolutional Networks) to let these connections "talk" to each other. It learns that a "Green Light" for cars usually means "Wait," while a "Red Light" for cars might mean "Go."
The Time Machine: The model doesn't just look at one snapshot; it watches a short video clip (15 frames, or 0.5 seconds). It uses a memory unit called an LSTM (Long Short-Term Memory) to remember what happened a split second ago. This helps it understand if a person is hesitating or if they are already stepping off the curb.

What They Found: Smarter Guesses

The researchers tested their new detective against older models using a dataset called PIE, which contains over 6 hours of real driving videos from a city. They asked the models to predict if a pedestrian would cross the street in the next few seconds.

The results were clear: the new model was better at guessing.

Accuracy: The TA-STGCN model got the right answer 84.65% of the time. This is a significant jump compared to the previous best models, which hovered around 79.00%.
Precision and Recall: The model was better at spotting people who actually intended to cross (Recall of 88.03%) and was less likely to cry wolf about people who were just standing still (Precision of 86.50%).
Trajectory: Not only did it guess if the person would cross, but it also predicted where they would be a moment later with less error than before (an Average Displacement Error of 0.43).

The authors explicitly showed that adding the traffic light information was the key. When they compared their model to a version without the traffic light data, the version with the lights performed better across every single metric. This proves that the model isn't just guessing; it's actually using the traffic signals to make smarter decisions.

The Catch: It's Still a Work in Progress

While the results are promising, the paper is careful not to claim this is a perfect, finished product ready for every car on the road today. The authors point out a few important limitations:

The "Eye" Problem: Right now, the model relies on perfect, pre-labeled data. It assumes someone has already drawn the boxes around the pedestrians and identified the traffic lights perfectly. In the real world, a self-driving car has to find these things itself using its own cameras and sensors. If the car's "eyes" make a mistake, the TA-STGCN model might get confused.
Speed: The model is smart, but it's also a bit heavy. It does a lot of math to connect all the dots in the graph. Running this on a small computer inside a car (an embedded system) might be slow. The authors suggest that future work needs to make the model faster and lighter so it can run in real-time without lagging.
Missing Scenarios: The data they used didn't cover every possible situation, like busy bus stops or shared spaces where people behave differently. The model might need more training to handle those tricky spots.

The Bottom Line

This paper suggests that if we want self-driving cars to be truly safe, they need to stop looking at pedestrians in isolation. They need to understand the whole stage: the traffic lights, the crosswalks, and the flow of the city. By teaching the computer to pay attention to the traffic signals, the TA-STGCN model shows a 4.75% improvement in accuracy over the previous best methods. It's a step forward in making autonomous vehicles not just smart, but context-aware, helping them make the right call when a pedestrian is deciding whether to cross the street.

Technical Summary: Traffic-Aware Pedestrian Intention Prediction (TA-STGCN)

Problem Definition
Accurate pedestrian intention estimation is a critical component for the safe navigation of autonomous vehicles (AVs), as pedestrians account for a significant portion of global road traffic fatalities. While existing deep learning models have advanced the field by capturing spatial and temporal dependencies, many fail to adequately incorporate dynamic contextual information, specifically traffic signal states (Red, Yellow, Green) and scene context. Traditional approaches relying on handcrafted features or classical filters (e.g., Extended Kalman Filter) struggle with generalization and nonlinear behaviors. Even recent deep learning models often neglect the specific influence of traffic lights and crosswalks on pedestrian decision-making, leading to potential inaccuracies in complex urban environments.

Methodology: TA-STGCN
To address these limitations, the authors propose the Traffic-Aware Spatio-Temporal Graph Convolutional Network (TA-STGCN). The model is designed to integrate dynamic traffic sign states and scene context directly into the intention prediction process.

Architecture Overview: The model employs a multi-stream architecture that processes two distinct feature streams:
1. Image-Class Stream: Combines visual data and object class labels.
2. Location-Class Stream: Combines spatial coordinates (bounding boxes) and object class labels.
  Both streams are processed through Spatio-Temporal Graph (ST-Graph) layers to encode spatial interactions and short-term temporal dependencies. The outputs are concatenated and fed into a Temporal Encoder consisting of Long Short-Term Memory (LSTM) cells to capture long-term dependencies before passing through a Fully Connected Network (FCN) for final classification.
Graph Construction: The system models interactions between pedestrians and traffic elements (e.g., traffic lights, vehicles) using a spatio-temporal graph $G = (V, E)$ . Nodes represent pedestrians and traffic elements, while edges represent interactions over time. The adjacency matrix is normalized to stabilize training, and a parametric matrix learns the relative importance of different interactions.
Feature Representation:
- Pedestrian Features: Extracted via CNN, including position, velocity, appearance, and bounding box size. The inclusion of bounding box size is a specific design choice to capture the pedestrian's proximity to the autonomous vehicle.
- Traffic Sign Features: Encoded as vectors containing type, state (Red, Yellow, Green), class, and position.
- Input Data: The model utilizes image data ( $I$ ), location data ( $L_f$ ), and class data ( $C_f$ ) derived from ground truth annotations to isolate intention estimation performance from object detection errors.
Objectives: The model outputs both the crossing probability ( $\hat{y}$ ) and future pedestrian positions ( $\hat{p}_{t+\Delta t}$ ). It is optimized by minimizing the binary cross-entropy loss function.

Key Contributions

Integration of Traffic Signals: Unlike prior models that may treat scene context generally, TA-STGCN explicitly integrates dynamic traffic signal states (Red, Yellow, Green) as key features, acknowledging their direct influence on pedestrian behavior.
Bounding Box Size as a Feature: The model introduces bounding box size as a critical feature to infer pedestrian proximity to the vehicle, enhancing the model's ability to detect crossing intent.
Spatio-Temporal Graph Modeling: The approach utilizes a dual-stream STGCN architecture to jointly model spatial interactions (between pedestrians and traffic elements) and temporal evolution (changes in these relationships over time).

Results and Evaluation
The model was evaluated on the PIE dataset, which contains over 6 hours of annotated urban driving videos. The dataset was split into training, validation, and test sets, with sequences of 15 frames (0.5 seconds) used as input.

Performance Metrics: The model was assessed using Accuracy, Precision, Recall, F1-Score, Average Displacement Error (ADE), and Final Displacement Error (FDE).
Comparative Performance: TA-STGCN outperformed traditional methods (SVM, RNN) and existing deep learning baselines (GCN, PIE Baseline Model).
- Accuracy: TA-STGCN achieved 84.65%, surpassing the PIE Baseline and GCN (both 79.00%) and the RNN (76.89%).
- Improvement: The authors report a 4.75% higher accuracy compared to the baseline model.
- Trajectory Prediction: The model achieved the lowest error rates, with an ADE of 0.43 and an FDE of 0.91, outperforming the baseline (ADE 0.47, FDE 0.92).
Component Analysis: A comparative study between a standard STGCN and the proposed TA-STGCN confirmed that the integration of traffic sign information improved performance across all metrics, including a boost in Precision (86.50%) and Recall (88.03%).

Significance and Limitations
The paper claims that TA-STGCN demonstrates the effectiveness of integrating dynamic traffic signal awareness and bounding box size into pedestrian intention prediction, leading to more robust performance in complex urban settings. The results suggest that explicitly modeling the state of traffic lights significantly enhances the reliability of AV safety systems.

However, the authors maintain a modest stance regarding real-world deployment, acknowledging several limitations:

Dependency on Ground Truth: The current evaluation relies on pre-annotated datasets for object detection and classification. In real-world scenarios, the model depends on the accuracy and speed of external object detection modules, which introduces potential noise and latency.
Computational Overhead: The construction of spatio-temporal graphs and the use of LSTM layers introduce computational costs that may challenge real-time deployment on resource-constrained embedded devices.
Dataset Diversity: The current dataset lacks coverage of specific critical scenarios, such as bus stops and shared spaces, which may limit the model's generalization to all traffic environments.

The authors conclude that while TA-STGCN significantly improves prediction accuracy, future work must focus on optimizing computational efficiency for real-time applications and expanding dataset diversity to ensure broader generalization.