⚡ electrical engineering

Reverberation-based Features for Sound Event Localization and Detection with Distance Estimation

This paper introduces novel reverberation-based features utilizing the direct-to-reverberant ratio and signal autocorrelation to address the lack of distance-specific inputs in 3D sound event localization and detection, achieving state-of-the-art performance on the STARSS23 dataset.

Original authors: Davide Berghi, Philip J. B. Jackson

Published 2026-04-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Davide Berghi, Philip J. B. Jackson

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are walking through a busy, echoey train station. You hear a siren, a announcement, and footsteps. Your brain doesn't just tell you what those sounds are; it also tells you where they are coming from. It knows the siren is 20 meters away to your left, and the footsteps are right next to you.

For a long time, computers trying to do the same thing (a field called Sound Event Localization and Detection, or SELD) were great at identifying the sound and its direction (left/right, up/down), but they were terrible at guessing how far away the sound was. It was like having a map that showed you the compass direction of a friend, but no idea if they were standing next to you or a mile away.

This paper introduces a new "superpower" for these computers: The ability to judge distance by listening to the echoes.

Here is a simple breakdown of how they did it:

1. The Problem: The "Flat" Map

Most current systems treat sound like a flat map. They know a sound is "North," but they don't know if it's 1 meter North or 100 meters North. To fix this, the researchers needed to teach the computer to understand reverberation (the way sound bounces off walls and floors).

2. The Solution: Two New "Ears"

The researchers invented two new ways for the computer to "listen" to the room's echoes. Think of these as two new sensory organs added to the computer's brain.

Feature A: The "Direct vs. Echo" Balance (DRR)

Imagine you are in a large cathedral. If someone whispers right next to your ear, you hear their voice clearly, and the echo is faint. If they whisper from the back of the cathedral, their voice is quiet, but the echo is loud and long.

The Analogy: The researchers created a feature that acts like a volume knob comparing the "Direct Sound" (the straight line from the source to your ear) against the "Reverberant Sound" (the messy pile of echoes bouncing off walls).
How it works: They use a special algorithm to separate the "clean" voice from the "messy" echo. By calculating the ratio between the two, the computer can guess: "If the echo is louder than the voice, the source must be far away."

Feature B: The "Floor Bounce" Timer (Autocorrelation)

This is the cleverer of the two. Imagine you drop a pebble in a pond. The ripples hit the edge of the pond and bounce back.

The Analogy: The researchers focused on the very first echo that hits the microphone. Usually, this is the sound bouncing off the floor.
- If the sound source is close to the floor, the bounce happens almost instantly.
- If the sound source is far away, the sound has to travel a longer path to hit the floor and bounce back, so the "bounce" arrives a tiny bit later.
How it works: They use a mathematical trick called autocorrelation (which is like comparing a sound to a slightly delayed copy of itself) to find that tiny gap between the original sound and the first floor-bounce. The size of that gap tells the computer exactly how far away the sound is.

3. The Experiment: The "STARSS23" Test

To see if this worked, they tested their new "ears" on a massive dataset called STARSS23. This dataset is like a giant library of recorded sounds in different rooms, where the computer knows the actual distance of every sound.

They taught a computer (using a neural network architecture called CNN-Conformer) to look at:

Standard sound charts (Log-mel spectrograms).
Direction clues (Intensity Vectors).
PLUS their new Distance Features (The "Direct vs. Echo" balance or the "Floor Bounce" timer).

4. The Results: A New World Record

The results were impressive:

Better Distance Guessing: The computer became much better at guessing how far away a sound was.
Better Overall Performance: Interestingly, adding these distance features didn't just help with distance; it actually helped the computer identify what the sound was and where it was coming from more accurately too.
The Winner: The "Floor Bounce" timer (Feature B) turned out to be the champion, achieving the best results in the world for this specific task.

Why This Matters

Think of a robot trying to help an elderly person. If the robot hears a fall, it needs to know: "Is the person in the next room, or are they right here on the floor?"

Old robots might hear the fall and run to the wrong room.
New robots with these "reverberation ears" can instantly tell, "That sound is 2 meters away, right in front of me," and rush to help immediately.

In a Nutshell

The researchers realized that echoes hold the secret to distance. By teaching computers to measure the balance between a direct voice and its echo, and to time the arrival of the first floor-bounce, they gave machines the ability to truly understand 3D space, making them smarter, safer, and more helpful in our real world.

1. Problem Statement

Sound Event Localization and Detection (SELD) involves identifying active sound events over time and estimating their spatial positions. While traditional SELD focuses on Direction of Arrival (DOA) estimation (azimuth and elevation), it often ignores source distance, which is critical for applications like human-robot interaction and safety.

Recent efforts to extend SELD to 3D (3D SELD) by incorporating distance estimation have primarily focused on architectural modifications (e.g., adding output branches or changing network backbones). However, a significant gap exists: existing methods lack input features specifically designed for distance estimation. Standard features like log-mel spectrograms and intensity vectors are effective for detection and DOA but do not explicitly encode the acoustic cues necessary for accurate distance prediction.

2. Methodology

The authors propose two novel reverberation-based feature formats designed to capture distance cues inherent in acoustic reflections. These features are compatible with both First-Order Ambisonics (FOA) and Microphone Array (MIC) formats.

A. Direct-to-Reverberant Ratio (DRR) Features

Concept: Distance is inferred from the energy balance between the direct sound and the reverberant sound.
Implementation:
1. Dereverberation: The Weighted Prediction Error (WPE) algorithm is applied to the omnidirectional channel ( $W$ ) of FOA audio to estimate the direct sound signal, $d(t)$ .
2. Reverberant Extraction: The reverberant component, $r(t)$ , is obtained by subtracting $d(t)$ from the original signal.
3. Feature Calculation: The Power Spectral Density (PSD) of both components is calculated. The DRR is computed as the ratio of direct power to reverberant power across time and frequency.
4. Mapping: This ratio is mapped to the log-mel scale to create the DRRmel feature, allowing concatenation with standard SELD inputs.
5. Variant (D+R): Instead of a ratio, the direct and reverberant components are converted into separate log-mel spectrograms and fed into the network to allow the model to learn the relationship independently.

B. Short-term Power of Autocorrelation (stpACC)

Concept: This feature targets the Initial Time Delay Gap (ITDG), the time interval between the direct sound and the first major reflection (typically from the floor). As distance increases, the ITDG generally decreases (assuming a standard source height relative to the microphone).
Implementation:
1. Analysis: The authors analyzed Room Impulse Responses (RIRs) at varying distances, observing that the first reflection peak shifts closer to the direct sound peak as distance increases.
2. Computation:
  - Compute the Short-Time Fourier Transform (STFT) of the signal.
  - Calculate the Autocorrelation Coefficient (ACC) in the frequency domain.
  - Normalize the ACC and compute its short-term power using a Hann-windowed moving average.
3. Feature Representation: The resulting stpACC is a 2D signal representing the energy of early reflections over time-lag, capturing both the timing and intensity of the first reflection.

C. Experimental Setup

Dataset: STARSS23 (providing 3D SELD labels at 100ms resolution).
Architecture: A CNN-Conformer architecture (CNN encoder + Conformer module + Feed-forward layers).
Training Strategy: Models were pre-trained on synthetic data (DCASE2024 Task 3) and fine-tuned on real-world STARSS23 data to mitigate distribution mismatches.
Augmentation: Audio Channel Swap (ACS) was used to increase data diversity.

3. Key Contributions

Novel Feature Extraction: Introduction of two reverberation-based features (DRRmel/D+R and stpACC) specifically designed to address the distance estimation gap in 3D SELD.
Preliminary Analysis: A study demonstrating that stpACC effectively captures distance-related information by analyzing how audio clips interact with RIRs at different distances, specifically focusing on floor reflections.
Comprehensive Benchmarking: Extensive evaluation on real-world data (STARSS23) across different network architectures (CNN-Conformer and ResNet-Conformer) and input formats (FOA and MIC), benchmarking against the state-of-the-art CDPD features.
State-of-the-Art Performance: Demonstration that these features achieve superior distance estimation without degrading other SELD metrics.

4. Results

The experiments were evaluated using DCASE 2024 metrics: F1 score ( $F_{\le 20^\circ/1}$ ), DOA Error (DOAE), Relative Distance Error (RDE), and the overall SELD score.

Distance Estimation (RDE):
- The stpACC feature alone achieved the best RDE (0.262) among single-feature additions, outperforming the baseline and CDPD features.
- Combining stpACC with DRR further reduced RDE to 0.251, though this came at a slight cost to localization accuracy.
- The D+R variant (separate direct/reverberant inputs) also showed strong performance (RDE 0.273), suggesting the model benefits from learning the components separately.
Overall SELD Score:
- The configuration using Log-Mels + IVs + stpACC achieved the highest overall SELD score (0.341), balancing distance accuracy with detection and localization.
- The proposed features consistently improved distance estimation while maintaining or slightly improving detection (F1) and localization (DOAE) metrics compared to baselines without distance features.
Comparison with CDPD:
- The proposed features outperformed the Coherence and Direct-Path Dominance (CDPD) features, particularly when using FOA inputs. CDPD features showed weaker performance on FOA data, whereas the proposed features are format-agnostic (working well with both FOA and MIC).
Architecture Impact:
- The CNN-Conformer architecture generally outperformed the ResNet-Conformer for these specific features, likely due to the latter requiring more extensive pre-training.

5. Significance

This paper addresses a critical limitation in 3D SELD systems: the lack of dedicated input features for distance estimation. By leveraging acoustic reverberation cues (specifically DRR and ITDG via autocorrelation), the authors provide a robust, format-agnostic solution that significantly enhances distance prediction.

The work demonstrates that feature engineering is as crucial as architectural design in SELD. The proposed stpACC feature achieves state-of-the-art distance estimation, enabling more precise 3D spatial understanding of sound events. This advancement is vital for real-world applications where knowing how far a sound source is (e.g., a person falling, a car approaching) is just as important as knowing where it is directionally. The code and features are made open-source to facilitate further research in 3D audio processing.