💻 computer science

Pedestrian Crossing Intention Prediction via Semantic Motion Representation and Cognitive Interaction Modeling

This paper proposes PsyMIND, an interpretable framework for early pedestrian crossing intention prediction that integrates psychology-informed semantic motion representations, contextual cues, and cognitive interaction modeling to enhance the safety and reliability of autonomous driving decisions.

Original authors: Yuemei Xi, Regan Thomas Potangaroa, Kelvin Xi ZUO, Chenchen Song, Wensheng Zhang, Kun Liu, Yan Yang

Published 2026-09-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yuemei Xi, Regan Thomas Potangaroa, Kelvin Xi ZUO, Chenchen Song, Wensheng Zhang, Kun Liu, Yan Yang

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Every day, millions of people step toward the curb, weighing a split-second decision: is it safe to cross, or should they wait? For a human, this calculation happens almost instantly, blending the sight of an approaching car, the color of a traffic light, and an internal sense of patience or urgency. For the self-driving cars of the future, replicating this human judgment is one of the most difficult challenges in engineering. These vehicles must not only see a pedestrian but understand the invisible mental process that leads to a step forward. If a car can only react after a person has already moved, it may be too late to prevent a collision. The goal is to predict the intention to cross before the foot even leaves the ground, turning a reactive machine into a proactive partner in traffic.

To solve this, researchers have traditionally taught computers to watch how people move. They track the path a person walks or the angles of their joints, looking for patterns that signal a crossing. However, a new study suggests that watching the body's geometry is not enough. The team behind this work, led by researchers from Shijiazhuang Tiedao University and others, argues that a pedestrian's decision is not just a physical movement but a psychological one. They propose that to truly understand if someone will cross, a computer must understand the quality of their movement and the context of their decision, much like a human observer does.

The researchers developed a system they call PsyMIND, which stands for Psychology-informed Semantic Motion and Cognitive Interaction Decision. Instead of simply measuring how far a person's leg moves or how fast they are walking, the system breaks down human motion into four distinct categories of experience: the shape of the body, the effort behind the movement, the space the person occupies, and the body's configuration. This approach is based on a method called Laban Movement Analysis, a framework originally developed to study human dance and expression, which the team adapted for traffic safety. By applying this framework, the system can detect subtle cues that a standard camera might miss, such as a slight forward lean of the torso that suggests a person is shifting their weight to step off the curb, or a hesitation in movement that indicates fear or uncertainty.

But the system does not stop at the body. It also looks at the world around the pedestrian. It considers the traffic signals, the speed of nearby cars, and even the pedestrian's own attributes, such as their age or gender, which might influence their risk tolerance. The researchers built a model that simulates the internal trade-off every pedestrian makes: the desire to cross quickly versus the need to stay safe. The system asks itself questions like, "Is this person waiting too long and becoming impatient?" or "Are they ignoring a red light because they see a gap in traffic?" By combining the psychological meaning of the body's movement with the pressure of the traffic environment, the model constructs a complete picture of the pedestrian's intent.

When the team tested this approach using a large collection of real-world traffic videos, the results were striking. The system correctly predicted whether a pedestrian would cross with an accuracy of 76 percent. More importantly, it achieved a recall rate of 93 percent, meaning it successfully identified almost all the people who were about to cross, even when they had not yet taken a step. In the high-stakes world of autonomous driving, missing a potential crossing is far more dangerous than falsely assuming one is happening, so this high rate of detection is a significant achievement. The system also proved effective at predicting intentions up to one and a half seconds before the actual movement began, a critical window that gives a vehicle enough time to slow down safely.

The study also revealed that previous methods, which relied heavily on simple geometric measurements or just the path a person walked, were less effective at these early stages. By moving beyond simple measurements to understand the meaning of a movement and the logic of a decision, the researchers showed that machines can be taught to see the world more like humans do. This does not mean the computer feels fear or impatience, but it can recognize the physical signs of those feelings. The findings suggest that for self-driving cars to operate safely in complex cities, they must look past the visible motion and understand the invisible decision-making process that drives it. This shift from watching what people do to understanding why they do it offers a clearer, more reliable path toward safer roads for everyone.

Technical Summary: Pedestrian Crossing Intention Prediction via Semantic Motion Representation and Cognitive Interaction Modeling

Problem Statement

Accurate prediction of pedestrian crossing intention is critical for autonomous driving (AD) decision support, as vehicles must anticipate crossing behavior before explicit stepping actions or displacement occur. While existing methods have evolved from trajectory-based approaches to skeleton geometry and multimodal fusion, they predominantly focus on optimizing observable feature combinations to improve prediction accuracy. A significant limitation identified in current research is the neglect of the underlying cognitive process driving pedestrian crossing intentions. Existing models often treat contextual cues as auxiliary inputs and rely on geometric pose representations, failing to explicitly model the psychological decision-making process (e.g., risk perception, rule compliance, and safety-efficiency trade-offs). This lack of interpretability and insufficient modeling of latent decision logic weakens the reliability of early prediction in safety-critical scenarios.

Methodology: The PsyMIND Framework

To address these gaps, the authors propose PsyMIND (Psychology-informed Semantic Motion and Cognitive Interaction Decision), a hierarchical framework designed to infer latent decision logic from observable behavior and contextual cues. The framework integrates three core components:

1. Psychological Semantic Motion (PSM) Encoder

Instead of relying solely on raw joint coordinates or geometric relations, the PSM Encoder utilizes Laban Movement Analysis (LMA) to transform pedestrian skeleton sequences into psychology-informed semantic motion representations. This maps raw motion into a four-dimensional BESS space:

Body: Characterizes body configuration and posture variation (e.g., gait size, limb openness, joint flexion).
Shape: Describes morphological adaptation and spatial directional changes (e.g., forward trunk inclination, limb-to-trunk angles).
Effort: Captures dynamic motion quality, including time (velocity/acceleration), weight (deceleration), and flow (jerk/hesitation).
Space: Reflects spatial activeness and directional tendency (e.g., cumulative displacement).

This approach elevates low-level geometric displacement into high-level semantic attributes that capture subtle pre-crossing cues like hesitation, preparatory contraction, and movement intensity.

2. Temporal Environmental Context Features (TECF)

The framework constructs contextual features to ground motion semantics in the environment. These include:

Environmental Constraints: Traffic signal states, intersection location, road type, and vehicle motion (acceleration/deceleration).
Social Attributes: Pedestrian age and gender as priors for risk tolerance.
Communicative Signals: Head orientation, gestures, and visual attention (e.g., looking at vehicles or signals).

3. Cognitive Interaction Decision (CID) Module

The CID Module models the internal decision logic of pedestrians as a trade-off between safety and crossing efficiency. It explicitly simulates four cognitive mechanisms:

Risk Perception: Combining visual attention to vehicles with vehicle threat levels.
Risk Interaction: Quantifying risk-taking tendencies by coupling forward body inclination with vehicle threat.
Patience Decay: Modeling the decay of waiting patience over time, integrated with motion jerk to predict non-compliant impulses.
Rule Interaction: Combining attention to traffic signals with signal states to determine the internalization of right-of-way rules.

4. Feature Fusion and Classification

The PSM features, TECF, and CID features are fused into a unified crossing-intention representation. A Support Vector Machine (SVM) is employed as the classifier to predict the binary crossing intention (crossing vs. non-crossing) for a future time horizon ( $T_{pred}$ ) based on an observation window ( $T_{obs}$ ).

Key Contributions

Psychological Semantic Motion Representation: The proposal of a PSM Encoder that maps raw skeleton sequences to a BESS space using LMA. This enables the capture of interpretable pre-crossing behavioral cues (posture, intensity, hesitation) that geometric methods often miss.
Cognitive Interaction Modeling: The design of the CID Module, which moves beyond simple feature fusion to explicitly model the latent cognitive decision process. It characterizes how pedestrians trade off rule compliance, risk acceptance, waiting costs, and safety-efficiency under traffic constraints.
Unified Interpretable Framework: The construction of PsyMIND, which integrates motion semantics, contextual constraints, and cognitive decision features. This provides a transparent basis for early intention prediction, shifting the paradigm from observable feature aggregation to intention-formation modeling.

Experimental Results

The framework was evaluated on the Joint Attention in Autonomous Driving (JAAD) dataset using a Leave-One-Video-Out protocol.

Performance Metrics: PsyMIND achieved an Accuracy of 0.76, an AUC of 0.83, and a Recall of 0.93.
Comparison: The model outperformed representative baselines (e.g., TrouSPI-Net, PCPA, SF-GRU, TrajFusionNet) in Accuracy and AUC. Notably, it improved AUC by 0.15 compared to the strong baseline TrajFusionNet.
Safety Orientation: While Precision (0.67) was lower than some baselines, the high Recall (0.93) indicates a safety-oriented tendency to minimize missed detections of potentially crossing pedestrians.
Early Prediction: The model maintained robust performance even at longer prediction horizons (up to 1.5s), demonstrating its ability to detect latent crossing tendencies before explicit displacement occurs.
Ablation Studies:
- Using only PSM features yielded poor performance (AUC 0.58), confirming that motion semantics require contextual grounding.
- Adding TECF significantly improved performance (AUC 0.83).
- The inclusion of CID features further boosted Recall from 0.75 to 0.93, validating the importance of modeling cognitive trade-offs.
- The PSM Encoder outperformed both Raw Coordinates and standard Geometric Features, proving the value of psychologically semantic representation.
Classifier Analysis: SVM outperformed deep learning classifiers (MLP, ResNet, 1D-CNN) in this specific feature space, achieving the best balance between class discrimination and safety-oriented detection.

Significance and Claims

The paper claims that the primary significance of this work lies in shifting pedestrian crossing intention prediction from observable feature aggregation toward interpretable intention-formation modeling.

Interpretability: By explicitly modeling the psychological decision process (risk, patience, rules), the framework offers a transparent explanation for why a pedestrian might cross, wait, or hesitate, rather than just predicting that they will cross.
Safety-Critical Utility: The high Recall and robust early prediction capabilities provide a safety-oriented basis for autonomous driving decision support, potentially reducing missed detections of hazardous crossing behaviors.
Theoretical Advancement: The work demonstrates that crossing intention is not merely a function of trajectory or pose but a dynamic cognitive outcome shaped by environmental stimuli and internal trade-offs.

The authors acknowledge limitations, including reliance on the JAAD dataset (requiring cross-dataset validation), dependence on skeleton extraction (sensitive to occlusion and distance), and the use of simplified computational approximations for complex psychological models. Future work is proposed to extend the framework to more diverse datasets and improve robustness under challenging perception conditions.