🤖 AI

EKF-Based Depth Camera and Deep Learning Fusion for UAV-Person Distance Estimation and Following in SAR Operations

This paper presents a robust UAV-based system for Search and Rescue operations that fuses YOLO-pose-based monocular distance estimation with depth camera data using an Extended Kalman Filter to achieve accurate, real-time tracking and safe following of human targets.

Original authors: Luka Šiktar, Branimir Ćaran, Bojan Šekoranja, Marko Švaco

Published 2026-02-25

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Luka Šiktar, Branimir Ćaran, Bojan Šekoranja, Marko Švaco

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a rescue drone as a brave, high-tech dog on a mission to find a lost hiker in a dense forest. The dog needs to stay close enough to guide the hiker but far enough away to avoid crashing into trees or the hiker themselves. The tricky part? The dog needs to know exactly how far away the hiker is, even when the sun is blinding, the trees are blocking the view, or the hiker is running.

This paper describes a new "super-sense" system for these rescue drones that combines two different ways of seeing the world, much like how humans use both their eyes and their sense of touch to navigate.

The Problem: One Eye Isn't Enough

The researchers tried two different methods to measure distance, but both had flaws:

The "Depth Camera" (The High-Tech Ruler): This camera shoots out invisible light beams (like a bat's sonar) to measure distance.
- The Good: It's incredibly accurate when the object is close (like holding a ruler up to your nose).
- The Bad: If the object is too far away, or if there's a shiny reflection (like a lake or a window), the camera gets confused and gives wild, wrong numbers. It's like a ruler that starts stretching out of shape if you try to measure a football field.
The "Monocular Camera" (The Smart Eye): This is a standard camera that just takes pictures. It uses Deep Learning (a type of AI that has studied millions of photos) to guess the distance. It looks at a person's body, finds their shoulders and hips, and calculates, "Based on how big those shoulders look, the person must be about 5 meters away."
- The Good: It works great even when the object is far away or when there are reflections. It never gets "confused" by shiny surfaces.
- The Bad: It's not perfect. If the person is very close, the math gets a bit wobbly, and it's just an estimate, not a precise measurement.

The Solution: The "Smart Fusion" (The EKF)

The paper's big idea is to combine these two methods using a mathematical brain called an Extended Kalman Filter (EKF).

Think of the EKF as a wise coach standing between the "Ruler" and the "Smart Eye."

When the "Ruler" (Depth Camera) is working perfectly (close range, clear view), the coach listens to it.
When the "Ruler" starts screaming nonsense numbers because of a reflection or because the person is too far away, the coach ignores it.
Instead, the coach listens to the "Smart Eye" (the AI estimate) to keep the drone moving smoothly.
The coach constantly blends the two inputs, smoothing out the "Ruler's" jitters and correcting the "Smart Eye's" guesses.

How It Works in Real Life

The system uses a special AI model called YOLO-pose (which stands for "You Only Look Once - Pose"). It's like a super-fast referee that instantly spots a person in the video and draws a line between their shoulders and hips.

The Anchor: The system knows the average distance between a human's shoulders and hips.
The Calculation: By seeing how big that "shoulder-to-hip" line looks in the camera, the AI can guess the distance.
The Safety Net: If the depth camera sees a weird spike (like a reflection making the person look 20 meters away when they are actually 5), the system realizes, "That's an outlier!" and switches to trusting the AI's shoulder-hip calculation until the camera calms down.

The Results: A Smoother Ride

The researchers tested this in a gym with motion-capture cameras (the "gold standard" of truth) and then outside in the real world.

The Magic: By fusing the two methods, they reduced errors by about 15%.
The Range: They extended the drone's reliable "safe zone" from 4 meters all the way to 7 meters.
The Robustness: Even when the person was running, crouching, or when the sun was reflecting off a window, the drone didn't crash or lose the target. It kept a steady, safe distance, just like a well-trained dog following its owner.

Why This Matters for Rescue

In a Search and Rescue (SAR) mission, every second counts. If a drone is trying to follow a lost hiker to lead them to safety, it cannot afford to crash into a tree because it misread a reflection, or lose the hiker because it got confused by the distance.

This new system gives the drone a "superpower": the ability to see clearly through the noise, reflections, and darkness, ensuring it can follow a human safely and reliably, no matter how chaotic the environment gets. It turns a fragile, glitchy drone into a robust, life-saving partner.

1. Problem Statement

Search and Rescue (SAR) operations require Unmanned Aerial Vehicles (UAVs) to autonomously detect, track, and follow human targets while maintaining a safe, constant distance. A critical challenge in this domain is accurate Camera-to-Body (C-B) distance estimation under real-world conditions.

Limitations of Monocular Cameras: While deep learning can estimate distance based on object size, these approximations are often non-linear and lack absolute precision without prior knowledge of the target's exact dimensions.
Limitations of Depth Cameras: Active depth sensors (e.g., stereo or time-of-flight) provide accurate measurements within an optimal range (0.4m–4m) but suffer from exponential error growth beyond this range, sensitivity to reflections, poor lighting, and measurement outliers during rapid UAV movements or at the edges of the Field of View (FOV).
The Gap: Existing systems often rely on a single sensor modality, leading to instability in dynamic SAR environments where lighting, distance, and target orientation vary significantly.

2. Methodology

The authors propose a multi-modal sensor fusion framework that combines the strengths of a monocular camera and a depth camera using an Extended Kalman Filter (EKF). The system runs on a UAV equipped with an NVIDIA Jetson Xavier NX.

A. Hardware and Setup

UAV Platform: Hexsoon EDU450 with an Orange Cube flight controller.
Vision System: Intel RealSense D435i (providing both RGB monocular and depth data).
Processing: Ubuntu 20.04/22.04, ROS2, and PyTorch (via Docker).
Validation: OptiTrack motion capture system (8 cameras) used as ground truth for indoor testing.

B. Deep Learning Pipeline

Detection & Recognition:
- YOLOv11: Used for real-time person detection and tracking with Re-identification (Re-ID) to maintain target identity across frames.
- Dlib: Used for face recognition to identify specific individuals against a reference template.
Keypoint Estimation (YOLO-pose):
- The system extracts 17 body keypoints.
- Key Metric: It calculates the distance between the midpoint of the shoulders and the midpoint of the hips (S-H distance). This metric is chosen because it is a stable anthropometric dimension largely invariant to body orientation (unlike face-based metrics).
Monocular Distance Approximation:
- The pixel-based S-H distance is converted to a physical C-B distance using a non-linear function (Eq. 1) derived from an average human height (1.80m).
- Limitation: This provides a stable trend but lacks absolute accuracy, especially at close range.

C. Depth Camera Processing & Outlier Filtering

The system aligns the monocular image with the depth map.
It extracts depth values specifically along the S-H line defined by the keypoints.
Outlier Detection: A derivative-based filter checks the rate of change in the measured distance. If the change between consecutive samples exceeds 25% of the average deviation of the last 10 samples, the measurement is flagged as an outlier (caused by reflections, edge noise, or sudden motion) and discarded.

D. Sensor Fusion (Extended Kalman Filter)

The EKF fuses the two data streams:

Prediction Step: Uses the monocular keypoint approximation (Eq. 1) as the state prediction. This provides stability and handles rapid body orientation changes.
Correction Step: Uses the depth camera measurement to correct the prediction.
- If the depth measurement is valid (not an outlier), the EKF updates the state.
- If the depth measurement is an outlier, the EKF skips the correction step and relies solely on the prediction, preventing the filter from diverging due to noisy data.
State Vector: Defined as $x_k = [p_k, \dot{p}_k]^T$ , representing the estimated distance and its derivative (velocity).

3. Key Contributions

Novel Fusion Framework: A robust EKF-based architecture that fuses deep learning-based monocular depth approximation with direct depth sensor measurements, specifically tailored for UAV-human interaction.
Keypoint-Centric Filtering: Instead of filtering the entire depth image, the system filters depth data strictly along the Shoulder-Hip (S-H) line. This ensures the depth data corresponds exactly to the tracked target, reducing noise from background objects.
Extended Operational Range: By fusing the stable (but less accurate) monocular estimate with the accurate (but range-limited) depth sensor, the system extends the reliable depth detection range from the sensor's native 4m up to 7 meters.
Real-Time Outlier Suppression: A derivative-based logic that dynamically switches between prediction and correction modes, ensuring the UAV does not react to transient sensor noise (e.g., reflections or edge artifacts).

4. Results and Performance

The system was validated in three indoor scenarios against OptiTrack ground truth and tested in unstructured outdoor environments.

Indoor Performance (Static Camera):
- Scenario 1 (Discrete movement): The fused method reduced the average error to 0.83 cm (compared to 10.45 cm for monocular and 13.01 cm for depth alone).
- Scenario 2 (Continuous movement): Fused error was 6.02 cm, outperforming both individual sensors.
- Scenario 3 (Lateral movement): The fused method achieved an error of 0.61 cm, significantly better than the depth camera (7.22 cm), which suffered from edge noise.
- Overall Improvement: The fusion reduced Mean Error, RMSE, and Standard Deviation by up to 15.3% across scenarios.
Outdoor Performance (UAV in Flight):
- Tested on the Hexsoon EDU450 in uncontrolled conditions (wind, varying light, GPS inaccuracies).
- The UAV successfully tracked a target moving (walking, running, crouching) while maintaining a safe distance of 5 meters (within a 3m–10m operational range).
- The system operated at 15 FPS on the Jetson Xavier NX, demonstrating suitability for real-time SAR missions.

5. Significance

This research addresses a critical bottleneck in autonomous SAR operations: reliable distance estimation in dynamic, unstructured environments.

Safety: By accurately maintaining a safe following distance, the risk of collision between the UAV and the rescuee is minimized.
Robustness: The system overcomes the inherent weaknesses of individual sensors (depth camera noise/outliers and monocular scale ambiguity) through intelligent fusion.
Operational Viability: The ability to extend the effective sensing range to 7 meters and operate in real-time on embedded hardware makes this framework a practical solution for deploying autonomous drones in life-saving missions where human operators may be unavailable or unable to react quickly enough.

In conclusion, the paper presents a highly effective, real-time solution for UAV-based human tracking that leverages the complementary nature of deep learning pose estimation and active depth sensing, significantly enhancing the reliability of autonomous SAR systems.