💻 computer science

Legs Over Arms: On the Predictive Value of Lower-Body Pose for Human Trajectory Prediction from Egocentric Robot Perception

This paper demonstrates that incorporating lower-body 3D skeletal keypoints and biomechanical cues into multi-agent trajectory prediction significantly improves forecasting accuracy for social robots, even when using monocular panoramic vision.

Original authors: Nhat Le, Daeun Song, Xuesu Xiao

Published 2026-02-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Nhat Le, Daeun Song, Xuesu Xiao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The "Legs Over Arms" Breakdown: Why Robots Should Watch Your Feet

Imagine you are walking through a crowded airport. You aren't just looking at where people are; you are subconsciously reading their "vibes" to avoid a collision. You notice a person’s shoulders dip, or more importantly, you see their legs swing forward, signaling they are about to step into your path. You don't need to see their facial expression to know where they are going—their legs tell the story.

This research paper, "Legs Over Arms," explores how we can teach social robots to do the exact same thing.

The Problem: The "Floating Ghost" Robot

Currently, most robots navigate crowds by treating humans like "floating ghosts" or simple dots on a map. The robot sees a person as a single point moving through space.

The problem? A "dot" doesn't have intention. A dot doesn't lean forward before it runs, and a dot doesn't pivot its hips before turning a corner. Because robots only see these "dots," they are often surprised by sudden human movements, making them clunky, scary, or socially awkward in crowded places like hospitals or malls.

The Solution: Giving the Robot "Human Eyes"

The researchers wanted to see if giving a robot "visual intelligence"—specifically the ability to see human skeletons—would make it a better navigator. They tested different "body parts" to see which ones provided the best "spoiler alerts" for where a person is headed.

Think of it like a detective looking for clues:

The Full Body: Looking at the whole person.
The Upper Body (Arms/Head): Looking at gestures and gaze.
The Lower Body (Legs/Feet): Looking at the "engine" of human movement.

The Big Discovery: "Watch the Feet, Not the Face"

After running extensive tests using 360-degree cameras (the kind that see everything around the robot at once), the researchers found something striking:

The legs are the MVP (Most Valuable Player).

When the researchers gave the robot data specifically about 3D lower-body keypoints (the joints in the hips, knees, and ankles), the robot’s ability to predict where a person would be in the next few seconds improved by a massive 13%.

The Analogy:
If you are watching a dance performance, you don't look at the dancer's hands to know if they are about to leap; you watch their feet. The researchers found that for a robot, the legs are the "steering wheel" of the human body. While arms and head movements are helpful, they are "secondary signals." The legs are the "primary signal."

Two Surprising "Bonus" Findings:

The "Biomechanical Boost": If you don't just show the robot the joints, but also tell it the math behind the movement (like "how wide is the step?" or "how fast is the knee bending?"), the robot gets even smarter. It’s like the difference between seeing a photo of a runner and seeing a high-speed video of their stride.
The "Fish-Eye" Success: 360-degree cameras (equirectangular images) are notoriously "distorted"—they make things look stretched and weird, like looking through a bubble. You might think this would confuse a robot, but the researchers found that even with this "funhouse mirror" distortion, the robot could still use 2D leg movements to predict paths effectively.

Why This Matters for the Future

This isn't just about math; it's about social harmony.

As we move toward a world where robots deliver medicine in hospitals or guide tourists in museums, we don't want them to be "clumsy ghosts" that people have to dodge. We want them to be "socially aware neighbors."

By teaching robots to "watch the legs," we are giving them the ability to read the "body language of movement," allowing them to glide through crowds smoothly, predictably, and politely.

Technical Summary: Legs Over Arms

Title: Legs Over Arms: On the Predictive Value of Lower-Body Pose for Human Trajectory Prediction from Egocentric Robot Perception
Authors: Nhat Le, Daeun Song, and Xuesu Xiao (George Mason University)

1. Problem Statement

Socially aware robot navigation in crowded environments requires accurate multi-agent trajectory prediction. Most existing research treats humans as "point masses" (simple 2D coordinates on a map), which ignores rich visual cues like body pose, gait, and orientation. This abstraction limits a robot's ability to anticipate movement, especially during short observation windows or when humans are partially occluded. The authors aim to identify which specific human skeletal features—and which body regions—provide the most predictive value when viewed from a robot's egocentric (first-person) perspective, particularly using 360° surround-view cameras.

2. Methodology

The researchers conducted a systematic comparative study using the Human Scene Transformer (HST) as a consistent backbone architecture to ensure that performance gains were due to the features themselves rather than model changes.

Feature Configurations: They evaluated several input types:
- 3D Skeletal Keypoints ( $K_{3D}$ ): Full body, lower-body only ( $K_{3D}^L$ ), and upper-body only ( $K_{3D}^U$ ).
- 2D Skeletal Keypoints ( $K_{2D}$ ): Standard COCO 17-keypoint format.
- Biomechanical Cues ( $C$ ): Derived features such as leg articulation angles, step length, arm angles, and head orientation.
Datasets:
- JRDB: A large-scale dataset providing 360° stereo camera and LiDAR data with high-quality 3D pose annotations.
- New Social Navigation Dataset: A novel dataset featuring a teleoperated robot (AgileX Scout Mini) navigating dense crowds in indoor environments using an Insta360 X4 camera, providing equirectangular panoramic images.
Task Formulation: The model predicts a distribution of future positions ( $F$ steps) conditioned on past motion histories ( $H$ steps) and the selected skeletal/biomechanical features.

3. Key Contributions

Systematic Feature Evaluation: The first study to rigorously compare different skeletal regions (upper vs. lower body) and biomechanical cues for multi-agent trajectory prediction in an egocentric context.
Identification of "Leg-Centric" Prediction: Proving that lower-body motion is the primary driver for accurate trajectory forecasting.
Validation of Panoramic 2D Cues: Demonstrating that 2D keypoints extracted from distorted equirectangular images are still highly effective for motion forecasting.
New Dataset: Introduction of a specialized dataset focused on realistic, dense social navigation scenarios.

4. Results

The study yielded several significant quantitative findings:

The Power of the Lower Body (on JRDB): Focusing on 3D lower-body keypoints ( $K_{3D}^L$ ) yielded the most dramatic improvements, reducing the Average Displacement Error (ADE) by 13% and the Negative Log-Likelihood (NLLpos) by 24% compared to the baseline.
Biomechanical Gains: Adding derived cues (like step length and joint angles) to the keypoints provided an additional 1–4% improvement, suggesting that while cues are helpful, the raw 3D keypoints already capture most of the necessary information.
3D vs. 2D: 3D keypoints significantly outperformed 2D keypoints on the JRDB dataset. This is attributed to the fact that 2D detectors struggle with the cropping of limbs caused by the limited vertical field of view in some 360° camera setups.
Panoramic Effectiveness (on New Dataset): On the equirectangular dataset, using 2D full-body keypoints ( $K_{2D}$ ) reduced ADE by 7%. This confirms that even with the inherent distortion of panoramic images, the visual cues remain highly informative.

5. Significance and Practical Implications

The paper provides actionable "engineering recipes" for developers of social robots:

Sensor Placement: To maximize the utility of 2D pose detection, cameras should be mounted at a height (e.g., ~0.85m) that minimizes extreme distortion of the human body.
Feature Selection: For real-time systems, prioritizing the detection and tracking of legs and feet is more computationally efficient and predictive than attempting to model the entire upper body or complex gestures.
Hardware Choice: While 360° cameras are cost-effective, the authors suggest that multi-fisheye rigs might be superior for maintaining a consistent vertical field of view, ensuring the legs remain visible for prediction.