🤖 machine learning

End-to-End Real-Time Drone-Based Person Detection Framework Using Deep Learning

This paper presents an end-to-end real-time person detection framework for drone-based surveillance that leverages a YOLOv8-nano model trained on the VisDrone2019 dataset to effectively handle scale variations caused by altitude changes, achieving reliable detection with over 41 FPS at altitudes between 16 and 25 meters.

Original authors: Payel Sarmah, Ayush Ranjan, Piyush Kaushik Bhattacharyya, Anil Kr. Shaw, Pradip Kr. Das

Published 2026-07-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Payel Sarmah, Ayush Ranjan, Piyush Kaushik Bhattacharyya, Anil Kr. Shaw, Pradip Kr. Das

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you're flying a drone like a high-tech kite, but instead of just snapping photos, it's trying to play a game of "I Spy" with people on the ground. That's exactly what this team of researchers set out to do: build a system that lets a drone spot people in real-time while it's zooming around, sending the video back to a computer on the ground to do the heavy lifting.

The Big Idea: A Drone with a Super-Brain
Think of the drone as the eyes and the computer on the ground as the brain. The drone's own processor is too small to be a super-genius, so it streams the video wirelessly to a powerful workstation equipped with a graphics card (GPU) that acts like a lightning-fast detective. This detective uses a special AI model called YOLOv8-nano. You can think of YOLOv8-nano as a very fast, very hungry student who has studied thousands of pictures of people from above. It's designed to be quick enough to keep up with the drone's movement without getting tired or lagging behind.

The Challenge: The "Zoom" Problem
The researchers found that spotting people from the sky is tricky. If the drone flies too high, people look like tiny ants; if it flies too low or at a weird angle, the background gets messy. Older methods, like trying to subtract the background (imagine trying to find a person by only looking at what isn't there), fail miserably because the drone is always moving, making the whole world shift around. The paper argues that these old tricks don't work for flying cameras and that deep learning (the "super-genius" approach) is the only way to handle the chaos of a moving sky.

The Experiment: Flying High and Low
To test their system, the team took their drone for a spin at the Indian Institute of Technology Guwahati. They didn't just guess; they flew the drone at different heights and angles to see what worked best.

Here is what they discovered, measured with real numbers:

The Sweet Spot: The system works best when the drone is flying between 16 and 25 meters above the ground. In this zone, the detection is highly reliable, and the system stays super fast, running at 41 FPS (frames per second) or even hitting a peak of 50 FPS. That's like watching a movie at a smooth, high speed without any stuttering.
The Training: Before the real flight, they taught the AI using a massive dataset called VisDrone2019, which has over 10,000 images and 263 video sequences of people in cities and villages. After training for 200 epochs (rounds of learning), the model achieved specific scores:
- Precision: 57.4% (When it says "I see a person," it's usually right).
- Recall: 41% (It finds about 41% of all the people actually there).
- mAP (mean Average Precision): 44.8% (A general score of how good it is at finding and boxing people).
- mAP50:95: 20.3% (A stricter score that checks if the box around the person is drawn perfectly).

The "Gotchas": Where It Struggles
The paper is very honest about where the system trips up. It's not a magic wand that sees everything perfectly.

The Green Camouflage: If a person is wearing clothes that blend in with the grass (like green or low-contrast colors) and standing on a lawn, the drone might miss them. The colors look too similar, confusing the AI.
The Busy Background: If there are banners, signs, or tables that look like people from above, the drone gets confused and might miss the real people or draw boxes on the wrong things. The researchers specifically noted that in busy areas with these distractions, the system misses some targets in those specific spots.
The Extreme Angles: If the drone looks straight down (a top-down view) or if the person is at the very edge of the video, the system loses reliability because there aren't enough features to recognize the person.

The Verdict
The researchers suggest that this setup is a solid step forward for real-time surveillance, proving that you can detect people live from a drone if you fly at the right height and use the right computer. However, they admit it's not a solved problem yet. The system still misses some people, especially in tricky lighting, crowded spots, or busy backgrounds. They plan to fix this in the future by adding "tracking" (so the drone remembers a person even if it blinks for a second) and by trying to put the brain on the drone itself so it doesn't need a wire to the ground.

In short, they built a drone that can spot people while flying, but it still needs a little help to see through the grass and the busy streets. It's a working prototype that shows great promise, but it's still learning the ropes of the real world.

Technical Summary: End-to-End Real-Time Drone-Based Person Detection Framework

Problem Statement
Unmanned Aerial Vehicles (UAVs) are increasingly utilized for security, search and rescue, and border surveillance. However, existing monitoring frameworks often fail to maintain detection consistency when targets undergo significant scale variations caused by changes in flight altitude. Traditional computer vision techniques, such as background subtraction and frame differencing, degrade rapidly in aerial environments due to global scene changes, steep viewing angles, and dynamic illumination. Furthermore, while deep learning models like YOLO have advanced the field, many studies rely on static datasets or offline video analysis, leaving a gap in real-time, sparse object detection with complex backgrounds during live aerial operations.

Methodology
The authors propose an integrated, end-to-end real-time detection pipeline designed to identify persons via wireless live drone video feeds. The system architecture comprises three primary stages:

Live Feed Acquisition: The system utilizes a commercial drone (DJI Mini 4 Pro) to capture high-definition video. Due to the limited computational power of onboard drone processors, the video feed is transmitted wirelessly to a remote ground station. A mobile interface acts as a local network relay, broadcasting the stream to a high-performance workstation for analysis. This setup ensures the drone remains focused on flight stability while the ground station handles heavy inference.
GPU-Accelerated Detection: The core detection engine employs the YOLOv8-nano architecture. This model was chosen for its optimal balance between inference speed and accuracy, critical for low-latency aerial surveillance. The system leverages CUDA-enabled hardware to bypass CPU bottlenecks. The model performs single-stage detection, outputting bounding boxes and confidence scores. A post-processing step filters predictions based on a confidence threshold ( $\theta$ ) and restricts the output to the "person" class (class ID 0).
Real-Time Performance Monitoring: The pipeline calculates processing latency ( $\tau_{proc}$ ) for every frame to determine the instantaneous Frame Rate (FPS). This metric is overlaid directly onto the video feed to ensure the inference rate remains synchronized with the input stream, preventing buffer overflow.

Dataset and Training
The model was fine-tuned on the VisDrone2019 dataset, which includes over 10,000 static images and video sequences captured in diverse urban and rural environments. To align with the specific goal of person detection, the distinct 'Pedestrian' and 'People' categories were merged into a unified 'Person' class. The dataset was partitioned into training (6,471 images), validation (548 images), and testing (1,610 images) sets. All images were resized to 640×640 pixels for input. Training was conducted for 200 epochs using an SGD optimizer with a learning rate of 0.01 on an NVIDIA GeForce RTX 4060 GPU.

Key Contributions
The paper outlines three primary contributions:

System Integration: The design and implementation of a UAV-based detection pipeline that integrates YOLOv8-nano with wireless video streaming, enabling real-time inference on live aerial footage rather than offline recordings.
Operational Analysis: An empirical analysis of the relationship between flight altitude, camera gimbal pitch, and detection performance (FPS and stability) to establish guidelines for optimal UAV deployment.
Visualization: The development of a real-time visualization system that displays detection results and performance metrics simultaneously.

Results

Quantitative Performance: On the VisDrone2019 test set, the YOLOv8-nano model achieved a precision of 57.4%, a recall of 41%, an mAP@50 of 44.8%, and an mAP@50:95 of 20.3%. The authors note that the higher precision indicates the system is reliable when flagging a detection, effectively filtering out background clutter.
Field Validation: Extensive flight tests were conducted at the Indian Institute of Technology (IIT) Guwahati at altitudes ranging from 15 to 30 meters.
- Optimal Performance: The system demonstrated near-total detection reliability and peak performance at an altitude of 25 meters with a camera gimbal pitch of -50°, achieving a frame rate of 48.53 FPS.
- Altitude Impact: At lower altitudes (16 meters), detection stability improved due to higher spatial resolution and better contrast between targets and the background, though the frame rate dropped to 35.31 FPS.
- Challenges: The study identified specific limitations, including reduced detection reliability when targets are at the extreme edges of the frame, in strictly vertical top-down views (due to lack of visible features), or in areas with chromatic homogeneity (e.g., green grass blending with subjects) or complex backgrounds (e.g., banners resembling human shapes).

Significance and Claims
The authors claim that this work effectively addresses the dual challenges of identifying targets at varying scales and ensuring near-to-accurate localization during aerial observation. By demonstrating that a lightweight model (YOLOv8-nano) can achieve high frame rates (>41 FPS) and reliable detection in real-world, live-streamed scenarios, the paper validates the feasibility of using commercial drones coupled with ground-based GPU workstations for immediate situational awareness. The study concludes that while the system performs robustly within specific altitude and angle boundaries (16–25 meters), future work is required to address edge-case failures and to migrate processing to onboard edge units for fully autonomous operations in communication-denied areas.

Technical Summary: End-to-End Real-Time Drone-Based Person Detection Framework

More like this