Technical Summary: ReasonAudio

Problem Statement

While Text–Audio Retrieval is a critical component for media search, content organization, and intelligent assistants, existing benchmarks and methods predominantly focus on semantic matching between audio and text captions. Current approaches assume relevance can be determined by surface-level similarity, overlooking the fact that real-world queries often require advanced logical reasoning. Specifically, existing systems struggle with tasks involving negation understanding, temporal ordering, concurrent event recognition, and duration discrimination. There is a lack of benchmarks designed to evaluate these reasoning capabilities beyond simple semantic alignment.

Methodology

The ReasonAudio Benchmark

To address this gap, the authors introduce ReasonAudio, the first reasoning-intensive benchmark for Text–Audio Retrieval. The dataset is constructed through a three-stage pipeline:

1. Atomic Sound Collection: 200 high-quality, clearly identifiable atomic sound samples were curated from FSD50K and Freesound, avoiding hierarchical or semantically similar concepts to prevent ambiguity.
2. Corpus and Query Synthesis:
   • Audio Corpus: 10,000 composite audio clips were synthesized by combining 2–8 atomic sounds. These were arranged in two patterns: sequential ordering (non-overlapping, 6,000 clips) and temporal overlap (simultaneous events, 4,000 clips).
   • Queries: 1,000 text queries were generated using predefined templates based on the metadata of the synthesized clips.
3. Relevance Annotation: Ground-truth relevance was determined via a deterministic program that verified if an audio clip's composition metadata satisfied all logical constraints of a query, followed by manual verification.

The benchmark evaluates five specific reasoning tasks:

• Negation: Retrieving clips containing target events while excluding undesired ones.
• Order: Retrieving clips where events occur in a specified temporal sequence.
• Overlap: Retrieving clips where events temporally overlap.
• Duration: Retrieving clips where an event persists for a specified duration.
• Mix: Retrieving clips satisfying a combination of the above constraints.

Evaluation Framework

The authors evaluated 10 State-of-the-Art (SOTA) models across three retrieval paradigms:

1. Two-Stage Pipelines: Converting audio to text (using Qwen2-Audio or Step-Audio) followed by text retrieval (using BGE-M3 or Qwen3-Embedding).
2. CLIP-style Embedding Models: Contrastive learning models (CLAP, AudioCLIP, Wav2CLIP) encoding audio and text into a shared space.
3. MLLM-based Embedding Models: Unified multimodal models fine-tuned from Multimodal Large Language Models (LCO-Embedding, e5-omni, OmniEmbed).

Performance was measured using Accuracy@1 (Acc@1) and nDCG@10. Additionally, a multiple-choice retrieval task (two options) was designed to isolate reasoning ability from acoustic matching, and t-SNE visualizations were used to analyze embedding alignment.

Key Results

• Overall Performance: All evaluated models struggle significantly with reasoning-intensive retrieval. The best-performing model, OmniEmbed-7B, achieved only 20.1% average accuracy.
• Task-Specific Challenges:
  • Negation and Duration: Models performed particularly poorly on these tasks. For Negation, even the best models scored between 2.0% and 10.0%. In the multiple-choice setting, OmniEmbed-7B scored only 27.5% on Negation, indicating a failure to capture negation semantics and a bias toward matching mentioned sounds.
  • Order and Overlap: Models showed relatively better performance on these tasks (e.g., OmniEmbed-7B reached 24.5% on Order and 18.0% on Overlap), though still far from satisfactory.
• Paradigm Comparison:
  • Two-Stage Pipelines: Performed the worst (average Acc@1 < 4.2%), primarily bottlenecked by the inability of audio captioning models to accurately describe composite sound events.
  • CLIP-style Models: Largely ineffective, with Acc@1 generally below 10%.
  • MLLM-based Models: Outperformed other paradigms but failed to inherit the strong reasoning capabilities of their pretrained backbones. The authors attribute this to contrastive fine-tuning emphasizing similarity matching over logical constraint satisfaction.
• Embedding Alignment: t-SNE analysis revealed substantial misalignment in the shared embedding space. Positive samples for the same query did not form compact clusters, and negative samples were often closer to the query embedding than positive ones, explaining the poor performance on logical constraints.

Significance and Claims

The paper claims that ReasonAudio is the first benchmark explicitly designed to evaluate logical reasoning beyond semantic matching in Text–Audio Retrieval. Its primary significance lies in exposing critical limitations in current multimodal retrieval systems:

1. Reasoning Gap: Existing models, including those based on large language models, fail to handle sound matching under logical constraints, particularly regarding negation and duration.
2. Training Paradigm Limitations: The results suggest that current training paradigms (specifically contrastive fine-tuning of MLLMs) are insufficient for preserving and transferring reasoning capabilities into retrieval settings.
3. Future Direction: The authors posit that future work must address the misalignment between text and audio embeddings regarding logical constraints to enable robust reasoning in multimodal retrieval.

The authors maintain a modest stance, noting that while the dataset size (1,000 queries) is small, it is consistent with accepted standards in information retrieval, and the pipeline is scalable. They conclude that current models are fundamentally limited in reasoning-intensive audio retrieval tasks.