🔬 applied physics

EchoHawk: A Reproducible Acoustic Pipeline for Drone Detection, Classification, and Direction-Finding, with a Cautionary Study of Session-Level Data Leakage

This paper introduces EchoHawk, a fully reproducible open-source acoustic pipeline for drone detection, classification, and direction-finding, while critically exposing how session-level data leakage in existing public datasets artificially inflates performance metrics and demonstrating the necessity of recording-session-grouped cross-validation to obtain honest evaluation results.

Original authors: David Shulman

Published 2026-06-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: David Shulman

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to find a specific type of hummingbird in a busy park. You can't see it, but you can hear its unique wingbeat. This paper, EchoHawk, is like a new, open-source "ear" designed to listen for those hummingbirds (drones), figure out exactly where they are, and track their flight path, all while ignoring the noise of the park.

Here is the story of what the researchers built and discovered, explained simply:

1. The Problem: Why Listen?

Drones are getting smaller and quieter. Traditional "radar" (like a lighthouse beam) often misses them because they are too small to reflect the beam. Cameras need a clear line of sight and good light. Radio detectors only work if the drone is talking to a remote control.

Acoustic sensing (listening) is different. It's like having a super-sensitive ear that works in the dark, through fog, and doesn't need to be seen. The catch? It doesn't work as far away as radar, and it can get confused by other noisy things like cars or lawnmowers.

2. The Solution: The EchoHawk Pipeline

The authors built a complete "recipe" (a pipeline) to turn raw sound into a drone alert. Think of it as a three-step detective process:

Step 1: The "Who" (Detection): Drones make a very specific sound. Their spinning blades create a musical note (the "blade-passing frequency") and a stack of higher notes (harmonics), kind of like a guitar string vibrating. The system listens for this specific musical signature to say, "That's a drone!"
Step 2: The "Where" (Direction): The system uses a circle of microphones (an array). By listening to the tiny split-second difference in when the sound hits each microphone, it can triangulate the direction, just like how your two ears help you tell if a sound is coming from the left or right.
Step 3: The "Path" (Tracking): A single guess at where a drone is can be shaky. The system uses a mathematical "filter" (like a smoothing tool) to predict the drone's next move based on its current speed and direction, making the tracking smooth and steady.

3. The "Hard Mode" Test

To prove their system is tough, they didn't just test it against silence. They built a synthetic simulator (a video game for sound) where they pitted drones against "confusers"—specifically, ground vehicles like cars.

The Analogy: Imagine trying to hear a specific flute in an orchestra. Most tests just ask you to find the flute in a quiet room. This test asked them to find the flute while a trumpet (the car) is playing a very similar, low note right next to it. This makes the job much harder and more realistic.

4. The Big Discovery: The "Leaky Bucket"

This is the most important part of the paper. The researchers found a hidden flaw in how many people test these AI systems.

The Flaw: Imagine you have a 10-minute recording of a drone flying. You chop it into 100 tiny 1-second clips. If you randomly shuffle these 100 clips and put some in the "training" pile (to teach the AI) and some in the "test" pile (to grade the AI), you have a problem.
The Analogy: It's like giving a student a math test where the "practice problems" and the "final exam" are actually just different pages from the same textbook chapter. The student isn't learning the math; they are just memorizing the specific font and paper texture of that one chapter.
The Result: Because the "training" and "test" clips came from the same recording session, the AI was cheating. It was recognizing the background noise of that specific recording session rather than learning what a drone actually sounds like.
The Fix: The authors forced the system to group the data by recording session. They made sure that if a clip from a specific recording was in the training set, no other clips from that same recording could be in the test set.
The Shock: When they fixed this "leak," the AI's performance dropped significantly. A system that looked like it was 79% accurate dropped to 74%. This proves that previous studies might have been overly optimistic because they didn't check for this leak.

5. The Results

The AI vs. The Old Way: They compared a simple, hand-crafted rule-based system (like a human writing down rules) against a deep learning AI (a neural network). The AI won, especially when they needed to be very careful not to raise false alarms.
The Direction Finding: They tested different mathematical methods to find the direction. Some methods were better at finding the drone in quiet conditions, while others were more robust in noisy environments.
Reproducibility: The authors released all their code, their sound simulator, and their tests. They want anyone to be able to run the exact same experiment and get the exact same numbers, without needing to download any secret data.

Summary

EchoHawk is a transparent, open-source toolkit for listening to drones. Its biggest contribution isn't just a new algorithm, but a warning label: "If you test your drone detector by randomly slicing up recordings, you are cheating yourself. You must keep whole recordings together to get a honest score."

They built a system that works, but more importantly, they showed the community how to measure it correctly so we don't get fooled by inflated numbers.

Technical Summary: EchoHawk

Problem Statement

The paper addresses the urgent need for affordable, reliable counter-unmanned-aerial-system (counter-UAS) sensing. While radar, electro-optical/infrared cameras, and radio-frequency (RF) interception have specific strengths, they face limitations regarding small radar cross-sections, line-of-sight requirements, and autonomous drone silence. Passive acoustic sensing is proposed as a complementary modality due to its covert nature, low cost, and effectiveness against the structured rotor noise of multirotor drones. However, acoustic detection faces challenges in range, environmental noise sensitivity, and the difficulty of distinguishing drones from other low-frequency harmonic sources like ground vehicles. Furthermore, the field suffers from methodological issues where machine learning models are evaluated on datasets with hidden data leakage, leading to overly optimistic performance estimates.

Methodology

The authors present EchoHawk, an open, end-to-end, and fully reproducible acoustic pipeline. The methodology is built on physically transparent signal models and rigorous experimental protocols.

1. Signal and Array Models

Acoustic Model: Drone noise is modeled as a harmonic stack at the blade-passing frequency ( $f_0$ ) and its integer multiples, superimposed on broadband turbulence. The model includes geometric amplitude decay, frequency modulation (RPM jitter), and additive noise. Crucially, the "hard negative" class (ground vehicles) is modeled with a similar harmonic structure but lower fundamental frequency and faster amplitude decay to create a realistic, non-trivial discrimination problem.
Propagation: The model accounts for spherical spreading ($6$ dB per distance doubling) and frequency-dependent atmospheric absorption, noting that higher harmonics attenuate faster, making low-frequency detection critical for long range.
Array Model: A planar microphone array is modeled using steering vectors for far-field sources. The design adheres to the Nyquist criterion for spatial sampling ( $d \le \lambda/2$ ) to avoid grating lobes.

2. Feature Extraction and Estimation

Blade-Passing Frequency (BPF): Estimated using the Harmonic Product Spectrum (HPS) to reinforce energy aligned with the fundamental and its harmonics.
Feature Representations:
- Classical Baseline: Uses Mel-Frequency Cepstral Coefficients (MFCCs), spectral shape descriptors (centroid, spread, flatness), and the HPS estimate.
- Deep Learning: Uses log-mel spectrograms as input for a compact Convolutional Neural Network (DroneCNN).
Direction of Arrival (DOA):
- Beamforming: Implements Delay-and-Sum (Bartlett), Minimum-Variance Distortionless Response (MVDR/Capon), and MUltiple SIgnal Classification (MUSIC) estimators based on the spatial covariance matrix.
- Time-Delay: Utilizes Generalized Cross-Correlation with Phase Transform (GCC-PHAT) and Steered-Response Power (SRP-PHAT) for robust localization in noisy conditions.
- Fusion: Narrowband spectra are fused incoherently over the harmonic band to produce a wideband azimuth estimate.
Tracking: A constant-velocity Kalman filter (or $\alpha-\beta$ filter) smooths successive noisy bearing estimates by exploiting the temporal continuity of drone motion.

3. Experimental Protocol and Data Leakage Correction

A central methodological contribution is the identification and correction of session-level data leakage.

The Issue: Public datasets (specifically the DroneAudioDataset) are pre-segmented into short clips. Naive random splitting at the clip level places adjacent slices of the same continuous recording into both training and test sets. This allows classifiers to learn session-specific nuisance factors (background, channel characteristics) rather than generalizable acoustic signatures.
The Correction: The authors enforce recording-session–grouped cross-validation, ensuring that no continuous recording session appears in both training and test partitions. This restores an unbiased estimate of generalization to unseen recordings.

Key Contributions

Open Reference Pipeline: An end-to-end implementation of detection, BPF estimation, DOA, and tracking, accompanied by a synthetic data generator that allows full reproducibility without external data downloads.
Hard Synthetic Benchmark: A simulator that pits drones against ground-vehicle confusers with overlapping low-frequency harmonic content, creating a realistic detection challenge across a wide Signal-to-Noise Ratio (SNR) range.
Quantified Data Leakage Study: A formal demonstration of how session-level leakage inflates performance metrics in acoustic drone detection, providing corrected, trustworthy baselines.
Comparative Analysis: A systematic evaluation of hand-crafted feature baselines (Random Forest) versus deep learning (CNN), and classical DOA estimators (Bartlett, MVDR, MUSIC) under varying SNR and array geometries.

Results

Synthetic Performance: On the hard synthetic benchmark, the Random Forest baseline achieved an AUC of $\approx 0.93$ and accuracy of $\approx 0.86$ , demonstrating the difficulty of the drone-vs-vehicle task.
DOA Accuracy: In synthetic tests, MVDR and MUSIC outperformed Bartlett beamforming at low SNR due to narrower main lobes. At high SNR, all methods converged to sub-degree accuracy.
Real-Data Detection (Corrected): Under session-grouped evaluation, the DroneCNN significantly outperformed the Random Forest baseline. At a 1% false-alarm rate, the CNN detected 93.8% of drones compared to 74.5% for the baseline.
Impact of Leakage Correction: Sealing the data leakage revealed a significant performance drop. For the Random Forest baseline, the Probability of Detection ( $P_d$ ) at a 1% false-alarm rate dropped from 0.796 (leaky/file-grouped) to 0.745 (honest/session-grouped). Similar drops were observed for the CNN, confirming that previous evaluations likely overestimated generalization capabilities.
Tracking: Temporal filtering reduced bearing root-mean-square error from $\sim 5.7^\circ$ (raw per-frame) to $\sim 1.9^\circ$ in moving-source simulations.

Significance and Claims

The paper claims that its primary contribution is methodological rather than purely algorithmic. It argues that in acoustic sensing tasks where continuous recordings are segmented into clips, only recording-grouped cross-validation yields honest performance estimates. The authors emphasize that naive clip-level splits measurably inflate results, potentially misleading the community about the true difficulty of the problem.

EchoHawk serves as a reproducible reference to ensure that future evaluations in counter-UAS acoustic sensing are conducted with rigorous data hygiene. The authors explicitly state that while the real-audio benchmark uses "easy negatives" (everyday sounds), the synthetic benchmark addresses the harder operational confusers, and the leakage analysis applies universally to any study segmenting continuous audio. The work prioritizes "methodological correctness and honest measurement" over headline figures.