Technical Summary: SelectTSL – Prompt-Guided Selective Target Sound Localization

1. Problem Formulation

Current Sound Source Localization (SSL) systems face a fundamental limitation: they are "semantic-blind." While deep learning-based SSL methods have achieved success in estimating the Direction of Arrival (DoA) for acoustic events, they typically localize all active sources indiscriminately. This contrasts with human auditory perception, where listeners can selectively attend to a specific target (e.g., a specific speaker) amidst competing sounds and noise (the "cocktail party problem").

Conversely, Target Sound Extraction (TSE) methods can isolate specific sources using multimodal prompts (text or audio) but prioritize waveform reconstruction fidelity over spatial consistency. Consequently, TSE often degrades the inter-channel phase information required for precise localization.

SelectTSL addresses the gap between these two fields by formulating a new task: Prompt-Guided Selective Target Sound Localization. The goal is to build an end-to-end framework that, given a multimodal prompt (text description or audio cue), localizes only the user-specified target source(s) in complex, multi-source, and noisy acoustic scenes, while suppressing interference. The system must also handle time-varying numbers of active target sources (cardinality) and moving targets.

2. Methodology: The SelectTSL Architecture

SelectTSL is an end-to-end architecture designed to decouple semantic selection from spatial estimation. It consists of two primary stages: a Prompt-Guided Selective Attention (PGSA) module and a DoA Estimator.

A. Prompt-Guided Selective Attention (PGSA) Module

This module acts as a semantic filter to extract target-consistent features from the audio mixture.

• Input Processing: It accepts a dual-channel mixture and a multimodal prompt (text or audio).
• Prompt Encoders: It utilizes a frozen CLAP (Contrastive Language-Audio Pretraining) model to encode text and audio cues into semantic embeddings. These are projected and fused into a unified guidance vector.
• Extraction Network: The module employs a dual-path RNN (DPRNN) backbone. It uses a Feature-wise Linear Modulation (FiLM) cascade to condition the magnitude features of the input mixture on the fused prompt vector.
• Cross-Attention: A cross-attention mechanism injects the semantic guidance into the audio features, allowing the network to focus on time-frequency regions relevant to the target.
• Outputs:
  1. Target Magnitude Spectrogram: A refined magnitude representation of the target source.
  2. Extraction-Informed Embeddings (EIEs): A stack of feature maps from intermediate DPRNN blocks that encode target-specific information. These EIEs serve as high-level semantic guidance for the subsequent localization stage.

B. DoA Estimator

This stage refines spatial cues using the semantic guidance from the PGSA module to estimate the DoA and source count.

• IPD Enhancer: This is a critical component that refines the raw Inter-Channel Phase Difference (IPD). It operates via two parallel branches:
  • Acoustic Branch: Processes the extracted target magnitude and raw spatial cues (IPD, ILD).
  • Semantic Branch: Processes the EIEs from the PGSA module.
  • Fusion: The branches are fused using a gated FiLM mechanism. The semantic branch modulates the acoustic features to enable spatially selective IPD enhancement. The network predicts residuals (deltas) to the raw IPD, recovering an enhanced IPD ($IPD_{enh}$) that is biased toward the prompt-specified target.
• Progressive Refinement Temporal Modeling (PRTM): The enhanced spatial cues, target magnitude, and ILD are concatenated and processed by a stack of Feature Refinement Blocks (FRBs), a Dilated Temporal Convolutional Network (TCN), and a Bidirectional GRU (BiGRU) to model temporal dependencies.
• Prediction Heads:
  • DoA Head: Outputs a frame-level posteriorgram over 180 azimuth bins (0°–179°).
  • Cardinality Head: A lightweight head that predicts the frame-level number of active target sources (0, 1, or 2).
• Inference: The system couples these predictions by selecting the top-$\hat{n}$ peaks from the DoA posteriorgram, where $\hat{n}$ is the predicted cardinality.

C. Training Objective

The model is trained jointly with three loss terms:

1. Separation Loss: Negative SI-SNR to ensure the PGSA module effectively extracts the target waveform.
2. DoA Loss: Binary Cross Entropy (BCE) on the DoA posteriorgram.
3. Cardinality Loss: Cross Entropy (CE) on the source count.
   Note: The weights are tuned to balance the scale differences between the separation loss (typically $O(10^1)$) and the sparse DoA loss (typically $O(10^{-1})$).

3. Key Contributions

1. Task Formulation: The paper defines the task of prompt-guided selective target sound localization in dynamic, noisy, and multi-source environments with unknown and time-varying target counts.
2. SelectTSL Framework: An end-to-end architecture supporting both text and audio prompts. It introduces the PGSA module to generate extraction-informed embeddings (EIEs) and an IPD Enhancer that uses these embeddings to refine spatial phase cues.
3. Cardinality Handling: A lightweight cardinality head jointly predicts DoA heatmaps and frame-level source counts, enabling stable tracking of moving speakers and intermittent activity without fixed track slots.
4. Robust Performance: Extensive experiments demonstrate consistent improvements over strong baselines in both synthetic and real-world recordings, showing robust generalization to realistic room acoustics.

4. Experimental Results

The authors evaluated SelectTSL on a synthesized dataset (covering diverse acoustic conditions, moving sources, and varying SNRs) and a real-world dataset (TAU-SRIR).

• Baselines: Comparisons included track-wise methods (IPDNet, EINV2, DCASE25), SELD+DoA methods (SELDnet, SELDT), pure DoA methods (SRP-DNN, FN-SSL), and prompt-based methods (Text-Queried SEL).
• Performance:
  • Static Metrics: SelectTSL achieved a Mean Angular Error (MAE) of 0.98° and an F1 score of 95.67% on the "Mix" condition (complex multi-source/noisy input), significantly outperforming the next best prompt-based baseline (MAE 5.37°).
  • Dynamic Metrics: In terms of trajectory tracking (MOTA*), SelectTSL achieved 91.57%, vastly superior to other methods which often dropped below 20% or failed to track in complex mixtures.
  • Ablation Studies:
    • Removing the IPD Enhancer increased MAE to 2.10°.
    • Removing the Cardinality Head reduced MOTA* from 91.57% to 58.19%, highlighting its importance for tracking.
    • End-to-end training outperformed a two-stage (separate selection and localization) approach.
    • The system showed robustness to varying room sizes, reverberation times, and motion speeds, though performance degraded with extremely fast motion or highly asymmetric rooms.
  • Real-World Generalization: On the TAU-SRIR real-room dataset, the model maintained a mean MAE of 2.62° and MOTA* of 77%, demonstrating strong transfer capabilities.

5. Significance and Claims

The paper claims that SelectTSL successfully bridges the gap between semantic-aware extraction and spatial-aware localization. By unifying these objectives, the system overcomes the limitations of:

• Conventional SSL: Which localizes all sources without selectivity.
• TSE: Which extracts targets but distorts spatial cues necessary for localization.

The authors emphasize that their coupled design, which uses semantic prompts to guide the refinement of spatial phase cues (IPD), allows the system to focus exclusively on user-specified targets. This enables robust localization and tracking even in scenarios with moving speakers, multiple interfering sources, and varying numbers of active targets. The work suggests that future systems for smart speakers, hearing aids, and robotic audition can move beyond "semantic-blind" processing to truly interactive, intent-driven spatial sensing.