Technical Summary: CASTELLA – Long Audio Dataset with Captions and Temporal Boundaries

1. Problem Statement

Audio Moment Retrieval (AMR) is a task designed to locate specific segments ("moments") within long audio recordings that correspond to a given text query. While Sound Event Detection (SED) focuses on short clips (typically ~10 seconds) and coarse-grained classes, AMR addresses long-form audio (minutes long) and complex, free-format text queries.

Despite the potential utility of AMR for applications like browsing broadcast content or analyzing surveillance footage, the field lacks a robust, large-scale benchmark using real-world data. Previous research, such as the initial AMR study by [6], relied heavily on synthetic datasets (constructed by overlaying sounds on background noise) and evaluated models on fewer than 100 manually annotated samples. This reliance on synthetic data and small-scale evaluation has resulted in unreliable performance metrics and an unclear understanding of how models perform in real-world environments. Existing audio datasets either lack temporal boundaries, lack free-format captions, or consist of audio clips that are too short to be considered "long audio."

2. Methodology and Dataset Construction (CASTELLA)

To address these gaps, the authors introduce CASTELLA (CAptionS and TEmporaL boundaries for Long Audio), a large-scale, human-annotated dataset.

Dataset Characteristics

• Scale: CASTELLA contains 1,862 audio recordings (1,009 for training, 213 for validation, 640 for testing), totaling over 120 hours of audio. This is approximately 24 times larger than the previous manually annotated AMR dataset (UnAV-100 subset).
• Content: Audio is sourced from YouTube (specifically a subset of the AudioCaps split) and ranges from 1 to 5 minutes in length.
• Annotations: Each recording includes:
  • Global Captions: A summary of the entire audio recording.
  • Local Captions: Free-format descriptions of salient audio moments.
  • Temporal Boundaries: Start and end timestamps for each local moment.
• Statistics: The dataset contains 3,881 local captions and 11,308 timestamps. The average audio contains 2.1 local moments, and the average moment duration is short, with a distribution heavily skewed toward shorter segments (long-tail distribution).

Annotation Procedure

The dataset was built using a crowdsourcing service with a rigorous multi-step process:

1. Selection: Annotators identified up to five salient local moments, avoiding non-local events that span the entire recording.
2. Captioning: Annotators described moments in detail without relying on visual information (though they watched videos to identify moments initially). Global captions were generated as summaries of local captions.
3. Translation: Initial annotations were in Japanese and translated to English by experienced translators, utilizing machine translation (LLMs) as a base where appropriate.
4. Boundary Assignment: Temporal boundaries were marked with 1-second resolution. Annotators were instructed to include ranges that capture the described moment, even if they slightly overlapped with non-relevant content.
5. Quality Control: A second annotator reviewed results. Crucially, the authors performed a final manual review listening only to the audio (without video) to remove captions that could not be verified by sound alone and to correct boundary errors, ensuring the dataset reflects audio-only retrieval capabilities.

3. Experimental Setup and Baselines

The authors established a baseline for AMR using the AM-DETR architecture, which combines an audio-text model (MS-CLAP) with a Detection Transformer (DETR).

• Training Strategies:
  • Training solely on the synthetic Clotho-Moment dataset.
  • Training solely on the real-world CASTELLA dataset.
  • Pre-training on Clotho-Moment followed by fine-tuning on CASTELLA.
• Models Tested: The study evaluated QD-DETR, Moment-DETR, and UVCOM (an advanced architecture) on the CASTELLA benchmark.
• Metrics: Performance was measured using Recall@1 (R1) at IoU thresholds of 0.5 and 0.7, and Mean Average Precision (mAP).

4. Key Results

The experiments yielded three primary insights:

1. Effectiveness of Hybrid Training: The most significant finding is that a model pre-trained on synthetic data (Clotho-Moment) and then fine-tuned on CASTELLA significantly outperformed models trained on either dataset alone.
   • In terms of Recall@1@0.7, the hybrid approach achieved 16.2, compared to 5.8 for the synthetic-only model and 9.7 for the real-world-only model. This represents a 10.4-point improvement over the synthetic-only baseline.
2. Architecture Trends: The performance trends across different DETR-based architectures in AMR mirrored those observed in Video Moment Retrieval (VMR). Specifically, UVCOM achieved the best performance (20.3 R1@0.7), while Moment-DETR performed the worst.
3. Challenge of Short Moments: Performance drops significantly for ground-truth moments shorter than 10 seconds. This aligns with challenges seen in VMR but is more acute in AMR, as the CASTELLA dataset shows a high frequency of short-duration events.

5. Significance and Claims

The paper positions CASTELLA as a necessary step toward reliable AMR evaluation and development.

• Benchmark Reliability: By providing a large-scale, real-world dataset, CASTELLA addresses the unreliability of previous evaluations based on <100 samples and synthetic data.
• Training Strategy Validation: The results demonstrate that while synthetic data is useful for pre-training, fine-tuning on real-world data is critical for achieving high performance in practical AMR applications.
• Foundation for Future Work: The authors state that CASTELLA enables the establishment of baselines and supports future research into simultaneous localization and captioning of events, as well as retrieval across collections of long audio recordings.

The authors conclude that CASTELLA is publicly available to facilitate further research in audio moment retrieval, moving the field from synthetic limitations to real-world applicability.