Robust Human Trajectory Prediction via Self-Supervised Skeleton Representation Learning

This paper proposes a robust human trajectory prediction method that leverages a self-supervised skeleton representation model pretrained with masked autoencoding to effectively handle missing joint data caused by occlusions, thereby improving prediction accuracy in real-world scenarios without compromising performance in clean conditions.

The Gist

Imagine you are trying to guess where a person walking down a busy street is going to end up in the next few seconds. This is the job of Human Trajectory Prediction, a technology used by self-driving cars and security cameras.

Usually, computers just look at the person's path (their "footprints" on the ground). But sometimes, that's not enough. To really understand where someone is going, you need to see how they are moving their body. Are they leaning forward to run? Are they turning their head to look at a shop?

This is where skeleton data comes in. It's like a digital stick-figure drawing of the person that tracks their joints (shoulders, elbows, knees).

The Problem: The "Blurry Glasses" Effect

In the real world, things get messy. A person might walk behind a pole, get blocked by a crowd, or the camera might glitch. When this happens, the computer's "stick-figure" breaks. Joints disappear. It's like trying to guess a dancer's next move while wearing foggy glasses where parts of their body keep vanishing.

If you feed this broken, missing data into a standard prediction model, it gets confused and makes terrible guesses.

The Old Way: "Just Get Used to It"

Previous methods tried to fix this by training the computer to "get used to" broken data. They would intentionally break the stick-figures during training so the model learned to guess even when parts were missing.

The Analogy: Imagine teaching a student to take a math test by giving them a test where half the numbers are erased. They might get better at guessing the missing numbers, but they also forget how to do the math perfectly when all the numbers are there. They become "okay" at broken data but "bad" at perfect data.

The New Solution: "The Invisible Mending Kit"

The authors of this paper propose a smarter, two-step approach. Think of it as a two-stage training camp.

Stage 1: The "Fill-in-the-Blanks" Gym (Self-Supervised Learning)

Before the computer ever tries to predict a path, it goes to a special gym.

• The Exercise: The computer is shown a perfect stick-figure, but then a "mask" is put over random parts of it (like covering the left arm and right leg with black tape).
• The Goal: The computer has to use its knowledge of how bodies work to reconstruct the missing parts in its mind. It learns that if the left shoulder is up, the left elbow is probably somewhere specific, even if it can't see it.
• The Result: The computer builds a super-strong "mental model" of human movement. It learns the essence of the skeleton, not just the raw coordinates. It becomes an expert at understanding people even when they are partially hidden.

Stage 2: The Prediction Race

Now, the computer takes this "mental model" (the pretrained encoder) and uses it for the actual job: predicting where people will walk.

• When a real-world camera sees a person with missing joints, the computer doesn't panic. It uses its "mental model" to fill in the gaps before making a prediction.
• It's like having a detective who can look at a few scattered clues and instantly visualize the whole crime scene, rather than just staring at the empty spots.

Why This is a Game Changer

The paper shows that this method solves the "trade-off" problem.

• Old Method: Good at broken data, bad at clean data.
• New Method: Good at both.

The Analogy: Imagine a musician.

• The old method is like a musician who practiced only with a broken guitar. They can play okay when strings are missing, but they sound terrible when the guitar is perfect.
• The new method is like a musician who practiced by listening to a song and mentally "hearing" the missing notes. Now, they can play beautifully on a perfect guitar, and if a string breaks during a concert, they can instantly improvise and keep the song going without missing a beat.

The Bottom Line

This research gives self-driving cars and security systems "super-vision." It allows them to understand human movement even in crowded, messy, or glitchy environments. By teaching the AI to "fill in the blanks" of human bodies first, it becomes much more robust, accurate, and reliable in the real world.

Technical Summary

1. Problem Statement

Human trajectory prediction is critical for autonomous navigation and surveillance. While recent approaches integrate human skeleton sequences to capture high-level motion cues (e.g., body orientation, intent) that pure trajectory data lacks, they face a significant challenge in real-world environments: occlusions.

• The Core Issue: In real scenarios, skeleton data is often incomplete due to joint detection errors, body orientation changes, or occlusions.
• The Trade-off: Existing methods struggle with a fundamental trade-off:
  • Models trained on clean data achieve high accuracy but degrade significantly when joints are missing.
  • Models trained directly on corrupted (masked) data become robust to missingness but often sacrifice accuracy on clean inputs because they learn to ignore informative skeleton cues.
• Limitation of Current Solutions: Simple reconstruction of missing joints before prediction often propagates reconstruction errors to the downstream predictor, failing to guarantee robust representations.

2. Methodology

The authors propose a two-stage framework that decouples robust skeleton representation learning from the downstream trajectory prediction task. The core philosophy is to learn robustness at the representation level during pretraining, rather than forcing the predictor to adapt to missing data.

Stage 1: Self-Supervised Skeleton Representation Learning

• Architecture: An asymmetric Encoder-Decoder model based on ST-GCN (Spatio-Temporal Graph Convolutional Networks).
  • Encoder: A 3-layer ST-GCN that maps masked skeleton sequences into latent features.
  • Decoder: A lightweight MLP that reconstructs the original joint coordinates from the latent features.
• Training Objective: Masked Autoencoding. A subset of joints is masked (replaced with zeros), and the model is trained to reconstruct the missing coordinates using the remaining visible joints and temporal context.
• Masking Strategies: To simulate diverse real-world occlusions, three masking patterns are used:
  1. Temporal Consistent: Same joints masked across all frames (simulating long-term occlusion).
  2. Random: Joints masked independently per frame (simulating sporadic detection failures).
  3. Body-Part: Entire body parts masked (simulating severe occlusion by obstacles).
• Loss Function: Mean Squared Error (MSE) between the reconstructed and original skeleton sequences. Crucially, the loss is applied to all joints (not just masked ones) to ensure the latent representation remains stable for both visible and invisible parts.

Stage 2: Integration into Trajectory Prediction

• Base Model: The framework is built upon Social-TransMotion, a state-of-the-art trajectory predictor using Cross-Modality and Social Transformers.
• Integration Strategy:
  • The standard linear embedding for 3D skeleton inputs is replaced by the frozen, pretrained skeleton encoder from Stage 1.
  • The encoder outputs high-level latent representations (tokens) which are fused with trajectory embeddings.
  • No Skip Connections: The model does not pass raw or reconstructed coordinates directly to the predictor; it relies solely on the robust latent features.
• Fine-tuning: Only the trajectory prediction modules are trained in the downstream stage; the skeleton encoder remains frozen to preserve the learned robust representations.

3. Key Contributions

1. Two-Stage Framework: A novel architecture that separates representation learning from prediction, explicitly addressing the accuracy-robustness trade-off.
2. Representation-Level Robustness: Demonstrates that masked autoencoding pretraining creates skeleton representations that remain informative and stable even under partial observability, rather than just learning to "guess" missing coordinates.
3. Empirical Validation: Extensive experiments on the JTA dataset (large-scale synthetic urban scenes) showing consistent performance improvements.
4. Analysis of Skeleton Reliance: Proves that the method's success is not due to reducing reliance on skeleton data (which would lower accuracy), but by preserving the utility of skeleton features under missingness.

4. Experimental Results

Experiments were conducted on the JTA dataset with varying levels of joint masking (0.0 to 0.6 missing ratio).

• Performance under Missingness:
  • The proposed method (Ours) achieved the best Average Displacement Error (ADE) and Final Displacement Error (FDE) in Clean to Moderate missingness regimes (0.0 – 0.4 mask ratio).
  • It significantly outperformed the standard baseline (Social-TransMotion) which degraded rapidly under occlusion.
  • It outperformed a "Corruption-trained" baseline (trained directly on masked data) in clean conditions, proving it did not sacrifice clean accuracy.
• Comparison with Reconstruction Baselines:
  • A baseline that reconstructs joints before prediction performed well at high missingness but was limited by the downstream predictor's architecture.
  • The proposed method remained competitive even at high missingness, suggesting that robust latent representations are superior to coordinate-level completion for severe occlusions.
• Skeleton Reliance Analysis:
  • When skeleton inputs were disabled, the proposed method showed a larger performance drop than the standard baseline. This confirms that the method actively utilizes skeleton cues effectively, rather than becoming robust by ignoring them.
• Qualitative Results:
  • In curved trajectory scenarios, the proposed method successfully predicted turning intentions (derived from body orientation) under both clean and masked conditions, whereas baselines often failed to capture curvature or predicted straight paths.

5. Significance and Conclusion

This paper addresses a critical bottleneck in deploying skeleton-aided trajectory prediction in real-world, occlusion-prone environments.

• Paradigm Shift: It moves away from the idea of "fixing" missing data before prediction or training the predictor to tolerate noise. Instead, it advocates for learning uncertainty-aware, robust representations via self-supervised pretraining.
• Practical Impact: The method offers a practical solution for autonomous robots and surveillance systems where sensor data is imperfect, ensuring high prediction accuracy without the need for perfect skeleton detection.
• Future Direction: The findings suggest that self-supervised masked modeling is a principled approach for building robustness in multimodal perception tasks, applicable beyond just trajectory prediction.

In summary, the proposed method successfully breaks the accuracy-robustness trade-off by ensuring that the skeleton features fed into the predictor are inherently resilient to missing data, leading to superior performance in both clean and occluded scenarios.