Weakly Supervised Detection and Temporal Localization of Whale Calls in Long-Duration Bioacoustic Data

Technical Summary: Weakly Supervised Detection and Temporal Localization of Whale Calls

Problem Statement
Passive Acoustic Monitoring (PAM) systems generate continuous, multi-month recordings of marine soundscapes, often totaling terabytes of data. While detecting whale vocalizations is critical for population monitoring and conservation (supporting SDG 14), automated analysis faces a significant bottleneck: the annotation cost. Current deep learning approaches, primarily Convolutional Neural Networks (CNNs), require two distinct and costly annotation efforts: binary presence/absence labels for classification and precise temporal boundaries (start/end times) for localization.

• The Discrepancy: Assigning a binary label to a 30-minute recording takes seconds, whereas timestamping every individual call within that recording requires hours of expert effort.
• The Limitation of Existing Methods: Standard CNNs and foundation models typically operate on fixed-size, short-duration inputs (e.g., ≤15 seconds). To process long-duration streams (minutes to hours), these methods rely on temporal compression or window-selection heuristics, which degrade performance and fail to provide temporal localization without frame-level supervision. Consequently, fully supervised models suffer significant performance drops on long inputs (e.g., F1 scores dropping from ~0.8 to ~0.2 on 1800s recordings) and cannot localize calls without expensive instance-level annotations.

Methodology: DSMIL-LocNet
The authors propose DSMIL-LocNet (Dual-Stream Multiple Instance Learning-Localization Network), a weakly supervised framework that performs both classification and temporal localization using only recording-level (bag-level) presence/absence labels.

1. Multiple Instance Learning (MIL) Formulation:

   • Each long-duration recording is treated as a "bag" ($B_i$) containing multiple non-overlapping short segments called "instances" ($x^j_i$).
   • The model receives only a single binary label $y_i \in \{0, 1\}$ for the entire bag (indicating if any call is present), while the specific instance labels remain unknown (latent) during training.
   • The goal is to learn attention weights ($\alpha^j_i$) that assign high probability to instances containing calls and low probability to background noise.

2. Dual-Stream Architecture:

   • Spectral Stream: Processes Mel-spectrograms (derived from STFT) using a CNN encoder with residual blocks to capture frequency modulation.
   • Temporal Stream: Processes hand-crafted temporal features extracted directly from the audio waveform, including Root Mean Square (RMS) energy, Zero-Crossing Rate (ZCR), Peak Amplitude, Envelope Statistics, and Temporal Centroid.
   • Fusion: A fusion module dynamically weights and combines features from both streams to adapt to varying noise conditions and call types.

3. Attention and Localization:

   • The model employs an attention mechanism where the weight $\alpha^j_i$ for each instance is calculated via a softmax function over the bag.
   • Classification: The bag-level prediction is a weighted sum of instance embeddings.
   • Localization: Temporal localization is achieved by identifying instances where the attention weight $\alpha^j_i$ exceeds a specific threshold. This allows the model to output timestamps without ever seeing frame-level labels during training.

4. Loss Function:
   The training objective combines four components to address specific challenges in long-duration bioacoustics:

   • Focal Binary Cross-Entropy ($L_{BCE}$): Addresses class imbalance (rare positive calls).
   • Temporal Smoothness ($L_{temp}$): Encourages coherent attention across adjacent instances to reflect call continuity.
   • Sparsity ($L_{sparsity}$): Ensures the model focuses on a small fraction of relevant segments.
   • Instance Consistency ($L_{inst}$): Promotes consistent feature representations.

Key Contributions

1. Weakly Supervised Formulation: The authors formulate long-duration whale call analysis as an MIL problem requiring only a single binary label per multi-minute recording, removing the dependency on expensive frame-level annotations.
2. Dual-Stream Architecture: They introduce a network combining spectral CNN features with hand-crafted temporal features, utilizing a multi-component loss function to handle attention dilution, class imbalance, and temporal coherence in long bags.
3. Simultaneous Classification and Localization: The framework demonstrates that a single set of learned attention weights can serve both bag-level classification and instance-level temporal localization.

Experimental Results
The method was evaluated on the AcousticTrends BlueFinLibrary, comprising 1,880.25 hours of Antarctic whale recordings with high background noise (73–99.9%).

• Performance on Long Durations: DSMIL-LocNet achieved F1 scores of 0.88–0.91 on recordings ranging from 300s to 1800s. In contrast, fully supervised CNN baselines (ANIMAL-SPOT, DeepWhaleNet, etc.) degraded significantly to F1 scores of 0.19–0.64 at these durations due to information loss from temporal compression.
• Localization Capability: Unlike the supervised baselines, which provided no temporal localization without additional annotation, DSMIL-LocNet successfully localized calls.
• Duration Trade-offs:
  • Classification: Performance improved with longer bag durations (up to 1800s) as the likelihood of containing a call increased.
  • Localization: Precision peaked at shorter bag durations (60–120s) and degraded slightly at longer durations (0.5–0.6 at 600s) due to attention dilution across more instances. However, the authors note that even reduced-precision localization is more valuable than the "no localization" provided by strong supervision baselines on long inputs.
• Ablation Study: Removing the dual-stream design or the multi-component loss resulted in significant performance drops, confirming the necessity of both spectral/temporal integration and specific loss constraints.

Significance and Claims
The paper claims that DSMIL-LocNet offers a practical solution for operational-scale PAM deployments where expert time is limited. By enabling both detection and localization from inexpensive, recording-level labels, the method bridges the gap between short-segment model optimization and the reality of continuous, long-duration marine monitoring.

The authors emphasize that this approach is not limited to marine bioacoustics. They suggest the underlying principle—converting inexpensive bag-level labels into structured temporal predictions—applies to any domain characterized by continuous data streams, sparse target events, and high annotation costs, such as seismology, medical acoustics, and terrestrial biodiversity surveys. The work demonstrates that weak supervision can provide capabilities (joint classification and localization on long inputs) that strong supervision with fixed-input architectures fails to support.