FlyPose: Towards Robust Human Pose Estimation From Aerial Views

The paper introduces FlyPose, a lightweight, real-time human pose estimation pipeline optimized for aerial UAV views that achieves significant accuracy improvements across multiple datasets and is successfully deployed on-board a quadrotor, accompanied by the release of a new challenging dataset called FlyPose-104.

The Gist

Imagine a delivery drone flying over a busy city park. Its job is to drop off a package, but it needs to make sure it doesn't accidentally drop it on a person. To do this safely, the drone needs to "see" people, understand what they are doing, and know exactly where their hands and feet are, even though the drone is looking down from high above.

This is the challenge the paper "FlyPose" tackles. Here is the story of how they solved it, explained simply.

The Problem: The "Bird's Eye View" Nightmare

Most computer programs that recognize humans are trained on photos taken at eye level (like a security camera or a phone). In those photos, you see a person's face, their whole body, and their arms clearly.

But a drone is like a bird looking straight down.

• The "Squashed" Effect: From high up, a person looks like a tiny dot. Their legs and arms get squished together (foreshortening), making them hard to tell apart.
• The "Hiding" Effect: If a person is sitting down or holding an umbrella, their face and body parts are hidden (occluded).
• The "Tiny" Effect: The higher the drone flies, the smaller the person looks in the camera. It's like trying to read the text on a postage stamp from 50 feet away.

Existing AI models get very confused in this situation. They might miss a person entirely or guess the wrong pose.

The Solution: FlyPose (The "Drone's Glasses")

The researchers built FlyPose, a special set of "glasses" for drones that helps them see people clearly from the sky. They didn't just build one model; they built a two-step team:

1. The Spotter (Person Detector): Think of this as a security guard with a magnifying glass. Its only job is to scan the crowd and say, "Hey, there's a person there!" and draw a box around them.
2. The Sketcher (Pose Estimator): Once the Spotter finds a person, the Sketcher zooms in and draws a stick-figure skeleton over them, connecting the dots for shoulders, elbows, knees, etc.

How They Trained the AI (The "School of Hard Knocks")

To make FlyPose smart enough for the sky, they couldn't just use regular photos. They had to get creative:

• The "Multi-Subject" Classroom: They didn't just train the AI on one type of photo. They fed it data from many different sources: city traffic cameras, search-and-rescue footage from mountains, and even thermal (heat-sensing) cameras used at night. It's like teaching a student to recognize a friend not just in a classroom, but also in the dark, in the rain, and from a distance.
• The "Tiny Person" Challenge: They created a new, tough test set called FlyPose-104. Imagine a test where the "students" (the AI) have to identify people who are so small they are barely visible. This forced the AI to get really good at spotting tiny details.
• The "Stretch" Trick: During training, they artificially shrank the images to simulate the drone flying higher. This taught the AI to recognize people even when they were just a few pixels wide.

The Result: Fast, Light, and On-Board

The biggest hurdle for drones is weight and power. You can't strap a super-heavy, super-powerful computer to a drone; it would crash.

• The "Featherweight" Champion: FlyPose is incredibly lightweight. It's like a race car engine that fits in a bicycle frame.
• Real-Time Speed: It works fast enough to keep up with the drone's movement. The whole process (finding the person and drawing the skeleton) takes about 20 milliseconds. That's faster than a human eye blink.
• The Flight Test: They didn't just test it on a computer. They strapped the system onto a real drone and flew it. The drone successfully spotted people and guessed their poses while hovering and moving, proving it works in the real world.

Why This Matters

Why do we care if a drone can see a person's pose?

• Safety: It prevents drones from crashing into people or dropping packages on them.
• Communication: In the future, you might wave at a drone to tell it to land, or point to a specific spot to drop a package. FlyPose allows the drone to understand these hand gestures.
• Rescue: In disaster zones, drones can scan crowds to find people who are waving for help or are injured, even from high up.

The Bottom Line

FlyPose is a breakthrough because it takes a difficult problem (seeing people from the sky) and solves it with a system that is fast, light, and smart enough to fly on a real drone. It turns a drone from a simple flying camera into an intelligent observer that understands human behavior, even when looking down from the clouds.

Technical Summary

1. Problem Statement

The paper addresses the critical need for Unmanned Aerial Vehicles (UAVs) to perceive human poses and actions in human-populated environments for applications like parcel delivery, search-and-rescue, and traffic monitoring. Existing Human Pose Estimation (HPE) methods, trained primarily on ground-level imagery, fail in aerial scenarios due to:

• Steep Viewing Angles: Top-down views (up to 90°) cause severe self-occlusion of limbs and faces.
• Scale Variation: UAVs operate at altitudes (up to 50m) where humans occupy very few pixels, making them "tiny objects."
• Occlusion: Frequent occlusion by the environment or the human body itself.
• Computational Constraints: UAVs have strict payload, power, and size limits, requiring lightweight, real-time models that cannot rely on heavy cloud processing.

2. Methodology

The authors propose FlyPose, a lightweight, top-down HPE pipeline designed for edge deployment on UAVs. The system operates in two stages:

A. Person Detection (Top-Down)

• Architecture: Uses RT-DETRv2-S (Real-Time Detection Transformer) with a ResNet-18 backbone. ResNet-18 was chosen over Transformer-based backbones for better efficiency on limited hardware.
• Training Strategy:
  • Multi-Dataset Training: The model is trained on a merged dataset comprising VisDrone, Manipal-UAV, HIT-UAV (thermal), TinyPerson, and others to handle diverse backgrounds, scales, and modalities (RGB and Thermal).
  • Loss Function: Replaces standard IoU loss with Normalized Wasserstein Distance Loss (NWDL) to improve stability and localization for small objects.
  • Domain Adaptation: Includes COCO-Person data to prevent the model from losing generalization capabilities for frontal/low-altitude views.

B. Pose Estimation

• Architecture: Utilizes ViTPose-S (Vision Transformer-based Pose Estimation), selected for its balance between accuracy and latency.
• Training Strategy:
  • Fine-tuning: Pre-trained on COCO, then fine-tuned on the UAV-Human dataset.
  • Augmentation: Introduces specific downscaling augmentation (5–20%) during training to simulate the low-resolution and motion blur typical of aerial footage.
• Pipeline: The detector crops the person region, which is then resized (preserving aspect ratio) and fed into the pose estimator to predict 17 COCO keypoints via heatmap regression.

C. New Dataset: FlyPose-104

To address the lack of challenging aerial pose data, the authors released FlyPose-104, a small but difficult test set containing 104 images with 193 annotated persons. It features:

• 90° top-down views.
• Heavy self-occlusion (legs/face hidden).
• Varied altitudes (5–50m) and backgrounds (snow, water, concrete).

3. Key Contributions

1. FlyPose Pipeline: A robust, lightweight, top-down HPE pipeline optimized for aerial imagery and edge devices.
2. Multi-Dataset Training: Demonstrated that training across diverse aerial datasets (RGB + Thermal) significantly improves generalization compared to single-dataset training.
3. FlyPose-104 Dataset: The release of a challenging benchmark dataset specifically for evaluating aerial pose estimation under difficult conditions.
4. Real-World Deployment: Successful integration and testing of the system onboard a commercial quadrotor UAV during actual flight experiments.

4. Results

Detection Performance

• Multi-dataset training improved average person detection mAP by 6.8% across test sets (VisDrone, Manipal-UAV, HIT-UAV, FlyPose-104).
• The model achieved competitive results on thermal imagery (HIT-UAV), crucial for night operations.

Pose Estimation Performance

• On the UAV-Human dataset, the fine-tuned ViTPose-S achieved 65.76 mAP, a significant improvement over the baseline COCO-pretrained model (61.09 mAP) and a massive leap from the previously reported AlphaPose result (56.9 mAP).
• The full pipeline (Detector + Pose) showed a 16.3 mAP improvement on the challenging UAV-Human dataset compared to prior state-of-the-art.

Latency and Deployment

• Hardware: Tested on a Jetson Orin AGX Developer Kit (32GB RAM).
• Latency:
  • Detection: ~13 ms.
  • Pose Estimation: ~6.54 ms.
  • Total Inference: ~19.54 ms per frame (excluding preprocessing).
• Real-Flight: The system achieved ~20ms total latency (including camera stream acquisition and preprocessing), enabling 25 FPS real-time performance. This leaves sufficient time for downstream tasks like gesture recognition and tracking.

5. Significance and Limitations

Significance:

• Real-Time Edge AI: Proves that complex pose estimation is feasible on UAVs without cloud dependency, enabling immediate reaction to human gestures or actions.
• Robustness: The multi-modal (RGB/Thermal) and multi-view training approach creates a model that generalizes well across different altitudes and environments.
• Foundation for Interaction: Enables new UAV applications such as gesture-based control, automated cargo handover, and human-drone interaction.

Limitations & Future Work:

• Small Objects: While improved, tiny objects in the background are still occasionally missed by lightweight detectors.
• Facial Keypoints: Accuracy drops significantly for facial landmarks (nose, eyes) due to occlusion and low resolution; body keypoints remain more reliable.
• Thermal Ambiguity: While the model works on thermal, left-right keypoint swaps occur more frequently due to the lack of color cues.
• Multi-Person Batching: Current latency is measured for single-person inference; batching multiple detections requires further optimization to maintain real-time speeds.

In conclusion, FlyPose bridges the gap between high-accuracy pose estimation and the strict computational constraints of UAVs, providing a practical solution for next-generation autonomous aerial systems operating in human-centric environments.