Systematic Evaluation of Time-Frequency Features for Binaural Sound Source Localization

Technical Summary: Systematic Evaluation of Time-Frequency Features for Binaural Sound Source Localization

Problem Statement
Binaural Sound Source Localization (SSL) aims to determine the spatial position of sounds in three dimensions using two-channel audio, replicating human auditory cues. While deep learning models, particularly Convolutional Neural Networks (CNNs) and Transformers, have achieved state-of-the-art accuracy, the specific impact of input feature design on model performance remains under-explored. Previous research has utilized various feature representations (amplitude-based, phase-based, or covariance matrices), but often without systematic comparison across diverse conditions. Furthermore, existing studies frequently rely on specific microphone arrays or restrict feature combinations to pairs, leaving a gap in understanding how feature selection influences generalization across different content types (e.g., speech vs. natural sounds) and listening conditions (e.g., matched vs. mismatched Head-Related Transfer Functions).

Methodology
The authors conducted a systematic evaluation using a controlled experimental framework to determine the most effective time-frequency features for binaural SSL.

• Datasets: The study utilized the Binamix library to synthesize datasets using the SADIE II database (20 subjects).
  • Training/Validation: Generated using TSP speech data with specific subject HRIRs.
  • Test Sets: Included an in-domain speech set (TSP–SSL) and two out-of-domain sets (SynBAD–Var and SynBAD–Fix) containing diverse natural and synthetic sounds (e.g., castanets, pink noise, reverberant speech) to test generalization.
• Feature Representations: Four core features were investigated individually and in combinations of up to four:
  • Amplitude-based: Magnitude Spectrogram (Mag) and Interaural Level Difference (ILD).
  • Phase-based: Phase Spectrogram (Phase) and Interaural Phase Difference (IPD).
  • Combinations included single features, pairs (e.g., ILD + IPD), triplets, and a full set of four.
• Model Architecture: A lightweight CNN was employed to ensure that performance differences were attributable to feature quality rather than model complexity. The architecture consists of three convolutional layers (32, 64, and 128 channels), ReLU activations, max-pooling, global average pooling, and a fully connected layer.
• Training and Evaluation: Models were trained to predict azimuth using a circular-aware Mean Squared Error (MSE) loss. Performance was measured using Mean Angular Error (MAE) on both in-domain and out-of-domain test sets, including scenarios with mismatched HRTFs.

Key Contributions

1. Systematic Feature Comparison: The paper provides a comprehensive analysis of amplitude and phase features, evaluating them individually and in various combinations (up to four features) for binaural SSL.
2. Generalization Analysis: It explicitly investigates how feature choices affect model robustness across in-domain (speech) and out-of-domain (diverse natural/synthetic sounds) content, as well as across different HRTF subjects.
3. Benchmarking: The authors released source code and datasets to the community to facilitate reproducibility and fair comparison of future methods.
4. Architecture Agnostic Insight: By using a simple CNN, the study isolates the impact of feature engineering, demonstrating that feature selection can be more critical than increasing model complexity.

Results

• In-Domain Performance (Speech): For speech-only data (TSP–SSL), simple two-feature combinations, specifically ILD + IPD, achieved performance comparable to richer four-feature sets (MAE of 4.5° at 0° elevation). This suggests that for matched domains, minimal complementary cues are sufficient, and adding more features yields diminishing returns.
• Out-of-Domain Performance (Diverse Content): On the challenging SynBAD–Var test set (containing natural and synthetic sounds), single features and simple pairs performed poorly. The best generalization was achieved by the three-feature combination of Phase L/R + ILD + IPD, which consistently outperformed other configurations across all source types and elevations.
• Feature Robustness:
  • Monaural features (Mag/Phase spectrograms alone) failed significantly on unstructured signals like white noise.
  • Explicit binaural cues (ILD/IPD) provided robustness, keeping errors below 10° for broadband noise.
  • Reverberation surprisingly improved localization for some signals (e.g., bursty pink noise) by adding temporal/spectral structure.
  • Pure tones remained difficult for all feature sets due to psychoacoustic ambiguity.
• Comparison with SOTA: The proposed low-complexity CNN (0.1M parameters) using optimal feature sets outperformed larger Transformer-based baselines (FAViT, AMViT, BAST-MAMBA) on out-of-domain data. Notably, models relying solely on magnitude spectrograms (like BAST-MAMBA) struggled, highlighting the difficulty of learning interaural relationships from raw channel spectrograms without explicit cues.

Significance and Claims
The paper concludes that feature design is often more impactful than model complexity for binaural SSL.

• For domain-specific tasks (e.g., speech localization), compact feature sets like ILD + IPD are sufficient and efficient.
• For general-purpose localization requiring robustness to diverse content and HRTF mismatches, richer inputs combining channel phase spectrograms with explicit interaural cues (ILD and IPD) are essential.
• The study underscores that simply scaling up model architecture (e.g., using large Transformers) is insufficient if the input features do not adequately represent the necessary spatial cues. The authors position their work as providing practical guidance for selecting feature sets based on the target application's domain constraints.