⚡ electrical engineering

ToS: A Team of Specialists ensemble framework for Stereo Sound Event Localization and Detection with distance estimation in Video

This paper introduces the Team of Specialists (ToS) ensemble framework, which integrates three complementary sub-networks to effectively address the multimodal challenges of 3D Sound Event Localization and Detection with distance estimation in video, achieving state-of-the-art performance on the DCASE2025 Task 3 benchmark.

Original authors: Davide Berghi, Philip J. B. Jackson

Published 2026-01-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Davide Berghi, Philip J. B. Jackson

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a complex mystery in a busy room: you need to figure out what sound is happening (a dog barking?), where it is coming from (left, right, far, near?), and when it is happening (is it a quick bark or a long howl?).

Doing all three of these things at once is incredibly hard for a single computer brain. It's like asking one person to be a detective, a cartographer, and a time-travel expert all at the same time. They might get overwhelmed and miss details.

This paper introduces a new solution called ToS (Team of Specialists). Instead of one "super-brain," the authors built a team of three different computer models, each with a specific superpower, working together to solve the mystery.

Here is how the team works, using simple analogies:

The Three Specialists

Think of the team as a group of experts hired to solve a crime scene. Each expert looks at the evidence through a different lens:

The "What & Where" Expert (Spatio-Linguistic Specialist):
- Superpower: This expert is great at understanding the meaning of the sound (using a language-trained AI) and figuring out the location.
- Analogy: Imagine a detective who is also a mapmaker. They can tell you, "That's a dog barking," and immediately point to the corner of the room. They are very good at identifying the sound and its position, but they might not be as focused on the exact timing of the event.
The "Where & When" Expert (Spatio-Temporal Specialist):
- Superpower: This expert focuses on location and time.
- Analogy: Think of a security guard with a stopwatch and a laser pointer. They are excellent at tracking where a sound is moving and how long it lasts, but they might not care as much about the specific "vocabulary" of the sound (e.g., distinguishing a specific type of bird call from another).
The "What & When" Expert (Tempo-Linguistic Specialist):
- Superpower: This expert focuses on the meaning of the sound and the timing.
- Analogy: This is like a music critic with a stopwatch. They can tell you exactly what instrument is playing and when the note hits, but they might not be the best at pinpointing exactly how far away the instrument is in the room.

How They Work Together (The Ensemble)

Individually, each expert is good, but they all have blind spots. The magic happens when they vote on the answer.

The Rule: If at least two of the three experts agree that a sound is happening, and they agree on roughly where it is, the team accepts it as a fact.
The Result: If the "What & Where" expert and the "What & When" expert both say, "Yes, a dog is barking on the left," the team is very confident. If only one expert says it, the team ignores it to avoid false alarms.

By combining their votes, the team creates a final answer that is smarter and more accurate than any single expert could produce alone. It's like a jury where the final verdict is only reached if the majority agrees, ensuring a high-quality decision.

The Test Drive

The authors tested this "Team of Specialists" on a specific challenge called DCASE 2025 Task 3. This challenge involves watching a video with stereo sound (two channels, like left and right speakers) and trying to find sounds, locate them, and guess their distance.

The Competition: They compared their team against the current "champions" (the best existing computer models) that try to do everything in one go.
The Outcome: The Team of Specialists won. They were better at identifying sounds, pinpointing locations, and estimating distances than any of the single-model competitors.
- They improved the accuracy of finding sounds by about 12% compared to the best single model.
- They were significantly better at guessing the direction of the sound.

The Trade-off

The paper notes one catch: Because the team uses three different models instead of one, it requires more computer power to run. It's like hiring three detectives instead of one; it costs more and takes more effort, but the result is a much more reliable solution.

Summary

The paper doesn't claim this technology will cure diseases or predict the weather. It simply claims that for the specific task of finding and locating sounds in videos, hiring a team of specialized experts who vote on the answer is better than relying on one generalist. This approach consistently beat the current best methods in their tests.

Technical Summary: Team of Specialists (ToS) Ensemble Framework for 3D SELD

Problem Definition
The paper addresses Sound Event Localization and Detection with distance estimation (3D SELD) in video. This multimodal task requires the simultaneous identification of active sound events from a predefined set, tracking their temporal activity, and estimating their spatial coordinates (Direction of Arrival - DOA, and distance). The specific challenge addressed is the DCASE2025 Task 3 Stereo SELD, which utilizes frontal perspective video and stereo audio. This setup replaces previous formats (FOA/MIC and 360° video) with mainstream media formats but introduces constraints, such as azimuth-only DOA estimation within [-90°, 90°] and the need to predict on-screen versus off-screen activity. The core difficulty lies in the requirement for joint reasoning across three distinct dimensions: spatial cues for localization, temporal patterns for activity tracking, and semantic understanding for sound classification. Single models often struggle to effectively integrate these dimensions simultaneously.

Methodology: The Team of Specialists (ToS)
The authors propose the Team of Specialists (ToS), an ensemble framework designed to navigate multi-dimensional complexity by decomposing the task into three complementary sub-networks. Each sub-network is a specialized audio-visual model trained on the same 3D SELD task but optimized to focus on a distinct pair of dimensions.

Architectural Foundation:
- Visual Branch: Unlike prior works using ResNet50 with global average pooling (which degrades spatial resolution), ToS employs OWL-ViT, a language-aligned visual encoder. OWL-ViT produces a fine-grained grid of visual embeddings.
- Audio Branch: The models utilize log-mel spectrograms and spatial features. For stereo audio, the primary spatial feature is the Inter-Channel Level Difference (ILD), calculated from the short-time Fourier transforms (STFTs) of the left and right channels.
- Fusion Mechanism: The models use a Cross-Modal Conformer (CMC) to fuse embeddings from different modalities, adapting the standard Conformer architecture for cross-attention between generic modalities.
The Three Specialists:
- Spatio-Linguistic Specialist: Focuses on spatial and semantic dimensions. It integrates a ResNet-Conformer SELD encoder with CLAP (a language-aligned audio encoder) via a CMC. The visual branch uses OWL-ViT with temporal average pooling to preserve spatial and semantic richness. This model prioritizes semantic understanding and localization.
- Spatio-Temporal Specialist: Focuses on spatial and temporal dimensions. It omits the CLAP encoder and the initial CMC to de-emphasize semantic reasoning. The visual branch extracts OWL-ViT embeddings per frame and pools across the channel dimension to preserve spatial structure over time. The audio branch uses SELD-specific features (spectrograms and ILD) to associate sound events with DOA.
- Tempo-Linguistic Specialist: Focuses on temporal and semantic dimensions. It mirrors the Spatio-Linguistic architecture but omits explicit spatial features (ILD) from the SELD encoder input, relying instead on implicit directional cues in spectrograms. The visual branch processes frames independently with OWL-ViT, averaging embeddings across the spatial dimension to yield a temporally ordered sequence of semantic representations.
Ensemble Strategy:
The final predictions are generated via a majority voting mechanism. A sound event is considered active at a specific time frame only if detected by at least two specialists, and their DOA predictions fall within a 20° angular threshold. The final DOA and distance are computed as the mean of the contributing specialists' estimates. For on/off-screen activity, a single positive prediction from any specialist is sufficient to classify the event as on-screen.

Key Contributions

Framework Proposal: Introduction of the Team-of-Specialists (ToS) framework, which integrates specialized audio-visual subnetworks to combine spatial, temporal, and linguistic knowledge into a unified understanding system.
Benchmarking: Evaluation of both individual specialists and the full ToS ensemble on the DCASE2025 Task 3 Stereo SELD development set using stereo audio and frontal video.
Performance Demonstration: Empirical evidence showing significant performance gains over state-of-the-art (SOTA) methods and reference baselines.

Experimental Results
The framework was evaluated on the DCASE2025 Task 3 development set (30,000 clips) using metrics including F1 score ( $F_{\le20^\circ/1}$ ), on/off-screen F1 ( $F_{\le20^\circ/1/on}$ ), Direction of Arrival Error (DOAE), Relative Distance Error (RDE), and On/Off Accuracy.

Individual Performance: The Spatio-Linguistic Specialist emerged as the top-performing individual model (F1: 36.9%), suggesting that semantic understanding and spatial localization are the most critical dimensions for this task. The Spatio-Temporal and Tempo-Linguistic specialists showed slightly lower performance in specific metrics, reflecting their reduced focus on the third dimension.
Ensemble Performance: The full ToS ensemble achieved an F1 score of 40.2% and an on/off-screen F1 of 30.7%. This represents a 3.3-point gain (12% relative increase) over the best-performing individual specialist.
Comparison with SOTA: ToS outperformed existing audio-visual models, including AV-Conf, MVANet, and the authors' previous work (Berghi [10]). Specifically, ToS achieved an F1 score nearly 5 percentage points higher than the previous best (Berghi [10]) and the lowest DOAE (16.7°).
Trade-offs: The authors note that this performance improvement comes with increased computational overhead due to the aggregation of three independent models.

Significance and Claims
The paper claims that the ToS framework effectively addresses the challenge of joint reasoning across semantic, spatial, and temporal dimensions by leveraging a modular, ensemble-based approach rather than overburdening a single architecture. The results demonstrate that specialized networks, when combined, can yield more robust and accurate SELD predictions than monolithic models.

The authors maintain a modest stance regarding future work, noting that the current results serve as a proof of concept. They suggest that future improvements could be achieved by strengthening the specialists with appropriate tasks, training, and pre-training curricula, and by investigating specialist pre-training to fully exploit their complementarity. The work is funded by the EPSRC-BBC Prosperity Partnership 'AI4ME'.