Gesture Matters: Pedestrian Gesture Recognition for AVs Through Skeleton Pose Evaluation

Technical Summary: Gesture Matters: Pedestrian Gesture Recognition for AVs Through Skeleton Pose Evaluation

Problem Statement
As autonomous vehicles (AVs) are deployed in intelligent transportation systems, a critical gap remains in their ability to interpret non-verbal communication between pedestrians and drivers. While formal traffic rules govern right-of-way, real-world traffic often relies on informal, non-verbal cues—specifically gestures—to negotiate intentions and resolve deadlocks. Human drivers naturally perceive these social signals, but AVs currently lack the perceptual capacity to interpret them, relying instead on predefined rules that may not capture the uncertainty of human behavior. This limitation risks AVs becoming overly cautious or unresponsive in complex urban environments. Furthermore, existing research lacks a dedicated, real-world dataset specifically tailored for pedestrian gesture recognition, with current datasets often limited to simulated environments or specific gesture categories.

Methodology
To address these gaps, the authors propose a computer vision framework for pedestrian gesture recognition using real-world video sequences from the WIVW dataset. The methodology proceeds through the following stages:

1. Data Curation and Taxonomy:

   • The authors utilized the WIVW dataset, selecting 180 outdoor, front-view, pedestrian-centric videos (excluding indoor VR and third-person perspectives) to ensure relevance to pedestrian-AV interaction.
   • A four-class gesture taxonomy was established based on literature review: "Stop," "Go," "Thank & Greet," and "No Gesture." This aggregation was designed to improve robustness across different regions and cultures.
   • Videos were preprocessed to isolate gesture-positive frames, removing non-essential sequences.

2. Pose Estimation and Normalization:

   • The framework employs the RTMPose-m model (via MMPose) to extract 2D skeleton keypoints (17 points following COCO annotation) from the video frames.
   • A torso-based normalization method was implemented to mitigate scale and position variations. Keypoints were normalized relative to the mid-hip point and scaled by the torso size (calculated as the average Euclidean distance between shoulders and hips), ensuring the features remain invariant to the pedestrian's distance from the camera.

3. Feature Engineering:

   • Static Features (71 features): Derived from the normalized keypoints to capture pose configuration at individual timesteps. These include mean keypoint positions, maximum keypoint positions, and average distances between specific keypoints (e.g., hand-to-hand, hand-to-shoulder).
   • Dynamic Features (5 features): Derived to capture temporal evolution and motion. These include the mean velocity and acceleration of the left and right hands, as well as the velocity ratio between the two hands.
   • The final feature set comprises 76 features (71 static + 5 dynamic).

4. Classification and Analysis:

   • A Random Forest classifier was selected for its explainability and proven performance in skeleton-based activity recognition.
   • t-SNE (t-distributed Stochastic Neighbor Embedding) was used to visualize the clustering of the feature subsets (static-only, dynamic-only, and combined) to assess class separability.
   • Feature importance was evaluated using Gini importance scores to identify discriminative factors.

Key Results

• Classification Accuracy: The combined feature set (static and dynamic) achieved the highest classification accuracy of 87.04%. This represented an improvement over the static-only model (83.33%) and the dynamic-only model (66.67%).
• Class Performance:
  • The "No Gesture" class was perfectly classified.
  • The "Go" class achieved 88% accuracy in the static model.
  • The "Stop" class showed the most significant improvement when dynamic features were added, increasing from 59% (static-only) to 71% (combined).
• Feature Discriminators:
  • Static Analysis: The mean vertical ($y$) position of the right hand and the hand-to-hand distance were the most discriminative static features.
  • Dynamic Analysis: Hand velocity was the primary discriminator. Specifically, the velocity of the left hand was identified as the single most important feature in the combined model. The authors attribute this to pedestrians on the right side of the road primarily using their left hand to signal vehicles.
• Clustering: The t-SNE analysis revealed that while "No Gesture" is easily distinguishable, "Stop" and "Thank & Greet" exhibit overlap in static features due to similar hand positions. The introduction of dynamic features helped moderately separate these classes, though some ambiguity remained between "Go" and "Thank & Greet," likely due to similar gesture durations.

Significance and Contributions
The paper claims to bridge the gap between pedestrian gesture research and computational analysis by providing a framework that utilizes real-world data rather than simulated environments. Its primary contributions are:

1. Standardization: The proposal of a four-class taxonomy derived from cross-cultural literature and existing datasets to standardize gesture categorization for AV applications.
2. Framework Validation: Demonstrating that a combination of static pose and motion dynamics is critical for accurate gesture recognition, achieving an 87% accuracy score.
3. Behavioral Insight: Identifying that hand position and movement velocity are the most discriminative factors, offering specific insights into how AVs can better interpret pedestrian intent.
4. Dataset Utilization: Successfully adapting the WIVW dataset for pedestrian gesture recognition, providing a baseline for future research in real-world traffic contexts.

The authors conclude that while the current framework improves perceptual capabilities, future work should explore deep learning models, temporal sequence modeling (e.g., LSTM), and frequency domain features to further distinguish periodic movements and resolve overlapping clusters.