Technical Summary: Afrispeech Semantics

Problem Statement

Audio Language Models (ALMs) have demonstrated strong performance in surface-level tasks such as transcription, captioning, and retrieval. However, their capacity for semantic reasoning—the ability to deduce logical relationships, verify consistency, and avoid over-inference based strictly on acoustic evidence—remains insufficiently benchmarked. Existing evaluations often reward plausible responses that rely on linguistic priors or world knowledge rather than evidence present in the audio signal. Furthermore, current benchmarks lack diversity in domains and accents, failing to explicitly test failure modes such as semantic over-inference, sensitivity to accent variation, and robustness to unfamiliar named entities. This gap is particularly critical for interactive and reasoning-oriented applications where models must accurately infer what can and cannot be concluded from spoken audio.

Methodology

The authors introduce Afrispeech Semantics, a unified framework designed to evaluate audio reasoning across diverse domains and accents. The methodology encompasses dataset construction, task formulation, and a controlled inference protocol.

Dataset Construction

The evaluation utilizes four distinct speech corpora covering clinical, conversational, and read-speech domains, with a specific focus on African accents and dialects:

• AfriSpeech-200: 200 hours of clinical and general domain speech from 13 African countries.
• AfriSpeech-Dialog: Simulated spontaneous conversations to test robustness against natural disfluencies.
• Afri-Names: Read-speech focusing on named entities and commands across 12 accents.
• Afrispeech-Medical: Real-world doctor-patient tele-consultations.

Hypothesis Generation and Verification: To ensure semantic validity, hypotheses were generated using Large Language Models (LLMs) under strict prompts forbidding unsupported negation and extraneous knowledge. Crucially, these candidates underwent human verification and correction by trained annotators. Annotators listened to the full audio to verify that hypotheses were entailed, contradicted, or unsupported, editing out hallucinations or ambiguous phrasing. This process ensures that every hypothesis is grounded in acoustic evidence, including accented and unfamiliar speech patterns.

Task Taxonomy

The framework evaluates five specific reasoning tasks, categorized by complexity and reasoning type (Table 1):

1. Audio Entailment: A three-way classification (Entailment, Neutral, Contradiction) to test deductive reasoning.
2. Consistency: A binary task (Consistent/Inconsistent) to test logical compatibility without a neutral option.
3. Plausibility: Tests whether models incorrectly accept hypotheses that are plausible under commonsense but unsupported by the audio.
4. Accent Drift: Evaluates whether semantic predictions change systematically when the same content is spoken with different accents.
5. Accent Restraint: Tests whether models suppress accent-based cues when the accent is semantically irrelevant, measuring the tendency to hallucinate content in low-semantic contexts.

Experimental Setup

The study benchmarks 10 state-of-the-art ALMs, including:

• Contrastive Models: (e.g., LAION-CLAP, MSCLAP) which rely on embedding similarity.
• Next-Token Prediction Models: (e.g., Qwen2.5-Omni, AudioFlamingo3, Kimi-Audio, SALMONN) which generate free-form responses.
• Cascaded Systems: Combinations of ASR (Whisper, IBM Granite) and LLMs (Llama-3.1, Mistral, Qwen2.5).

Evaluation is conducted in a zero-shot setting using a unified inference protocol. For generative models, a lightweight language model maps free-form outputs to discrete labels. Metrics include accuracy, precision, recall, macro-averaged F1, and class-specific accuracies to detect asymmetric errors like over-entailment.

Key Results

Model Performance

• Generative vs. Contrastive: Next-token prediction models substantially outperform contrastive baselines on reasoning tasks. Contrastive models often perform near chance on fine-grained semantic entailment, highlighting the limitations of embedding-only approaches for complex reasoning.
• Top Performers: Qwen2-based variants (e.g., Qwen2.5-Omni) and AudioFlamingo models achieved the highest macro F1 scores (often >0.82) on plausibility and consistency tasks.
• Cascaded Systems: The Whisper + Qwen2.5-7B cascade achieved the highest overall performance (Macro F1 0.737) on the medical entailment task, outperforming native end-to-end ALMs. This suggests that for complex reasoning, the precision of a discrete transcription step remains superior to direct audio-to-text latent mapping.

Systematic Failure Modes

• Over-Entailment: Even the best models exhibit a systematic tendency to mark plausible but unsupported hypotheses as true (Entailment), indicating a reliance on linguistic priors rather than acoustic evidence.
• Accent Sensitivity: Significant "accent-conditioned semantic drift" was observed. For instance, on the AfriNames dataset, AudioFlamingo2 showed a 96.75% bias rate (inferring different meanings based on accent), whereas Qwen2AudioInstruct showed a lower bias rate of 33.25%.
• Semantic Restraint: Models frequently hallucinate unsupported content when audio input is minimal. While Qwen2.5Omni achieved a Semantic Restraint Accuracy (SRA) of 66.0%, many other models hallucinated in over 60% of cases.
• Domain Shift: Performance varied significantly across domains. Models excelling on read-speech (AfriSpeech-Gen) often lagged on spontaneous clinical dialogues (Medical), suggesting a need for domain-specific adaptation.

Significance and Contributions

The paper makes three primary contributions to the field of audio language modeling:

1. Unified Reasoning Framework: It proposes multiple task formulations designed to test whether models preserve meaning under pronunciation variation rather than relying on contextual priors, moving beyond simple transcription accuracy.
2. Rigorous Data Curation: By combining LLM generation with systematic human verification, the authors created a dataset where hypotheses are strictly grounded in audio evidence, including challenging accented and unfamiliar speech patterns.
3. Comprehensive Benchmarking: The study reveals consistent patterns of over-entailment, domain-specific reasoning failures, and accent-conditioned semantic drift that are not captured by existing benchmarks.

The authors conclude that current benchmarks substantially underestimate reasoning errors in ALMs. They argue that the field requires more comprehensive, domain-aware evaluation to guide the development of robust and equitable audio language models that can reason carefully based on acoustic evidence. The resources, including datasets and evaluation scripts, are released to support reproducibility and further research into mitigating these reasoning failures.