Technical Summary: MedMosaic

Problem Statement

The evaluation of audio and multimodal reasoning models has increasingly shifted from recognizing lower-level modalities toward higher-order inferences, such as temporal abstraction, causal reasoning, and cross-modal grounding. However, existing benchmarks for medical audio reasoning suffer from significant limitations. They rely predominantly on short, isolated audio segments, single-turn questions, or ambient noise, thereby failing to capture the complexity of real-world clinical interactions. Crucially, they often lack the ability to assess long-range temporal reasoning, multi-turn conversational dynamics, and the integration of subtle, domain-specific acoustic cues (e.g., breathing sounds, vocal strain, hesitation) with spoken language. Furthermore, the collection of genuine medical audio recordings is hindered by strict data privacy regulations and the high costs of expert annotation, leading to a scarcity of large-scale, diverse medical audio datasets.

Methodology

To bridge these gaps, the authors introduce MedMosaic, a large-scale benchmark for medical audio question-answer (QA) pairs, encompassing 46,701 QA pairs. The dataset is constructed using a scalable, synthetic generation pipeline that ensures strict control over reasoning complexity while simulating realistic clinical constraints.

Data Generation Pipeline

The core of MedMosaic is a controlled generation framework utilizing Gemini-3-flash to generate QA pairs from diverse audio sources. The pipeline integrates:

1. Diverse Audio Sources: The benchmark combines open-source datasets (e.g., CoughVID, HLS-CMDS, Primock, Ekacare) with a synthetic dataset generated via text-to-speech models from ElevenLabs v3. This enables the creation of high-fidelity medical dialogues containing physiological artifacts (coughing, wheezing, sighing) and emotional markers.
2. Construction of Synthetic Audio Recordings: Raw transcripts are enriched with acoustic placeholder tags and processed to insert category-specific sounds (breathing, pain-related, emotional) as well as indicators of natural conversation flow. A curated pool of 151 voices ensures demographic and role-specific diversity.
3. Categories of QA Pairs: The dataset covers seven distinct reasoning regimes:
   • Sound-Only: Non-verbal physiological sounds (heart, lungs, cough) requiring reasoning based solely on acoustic structure and temporal patterns.
   • Speech-Only: Clinical dialogues testing the understanding of verbal content and conversational cues without external medical knowledge.
   • Speech + Sound: Integrated dialogues with embedded physiological sounds requiring cross-modal reasoning.
   • Multi-Turn QA: Sequences of three turns with explicit, answer-dependent dependencies, where incorrect answers in early turns introduce ambiguities for subsequent questions.
   • Long-Form Audio: Recordings exceeding 3 minutes, requiring models to extract and integrate evidence distributed across widely separated dialogue turns (multi-step temporal reasoning).
   • Voice-Based QA: Questions and answers are vocalized and appended to the audio stream, testing the model's ability to process continuous streams containing both clinical conversations and spoken instructions.
   • Open QA: Unrestricted generation tasks requiring synthesis of evidence and interpretation of subtle cues (hesitation, qualifiers) rather than selecting from fixed options.

Evaluation Protocol

The benchmark evaluates 13 audio and multimodal reasoning models, including LALMs (e.g., Audio Flamingo, SALMONN), unified foundation models (e.g., Qwen-Omni, Gemini-2.5), and reasoning-centric models.

• Metrics: Structured QA pairs are evaluated against the ground truth (generated by Gemini-3-flash) using string-matching accuracy. Open-ended responses are assessed following the "LLM-as-a-Judge" paradigm (GPT-5.1), based on correctness, relevance, completeness, and clarity.
• Validation: A validation process by subject matter experts (SMEs), involving two healthcare professionals, confirmed the clinical validity of the synthetic data. 72.4% of sampled QA pairs were accepted without modification, while only 4.8% were rejected.

Key Contributions

1. A Large-Scale Benchmark for Medical Audio Recordings: MedMosaic offers the first comprehensive benchmark covering multiple reasoning regimes (short/long context, multi-turn, open) and modalities (speech, sound, combined) specifically for medical audio recordings.
2. Scalable Synthetic Generation Pipeline: The authors demonstrate that synthetic generation can produce clinically grounded, challenging evaluation data at scale with minimal human intervention, thereby overcoming data privacy and annotation bottlenecks.
3. Reasoning-Focused Evaluation: The benchmark goes beyond simple recognition to test complex capabilities such as multi-step temporal reasoning, cross-modal integration, and the interpretation of paralinguistic cues in clinical settings.
4. Strict Design Constraints: The dataset enforces an "anti-hallucination" principle, where answers must be derivable exclusively from the audio, and distractors are designed to be acoustically confusing and lexically similar to prevent superficial reasoning.

Results

Benchmark results indicate that medical audio reasoning remains a significant challenge for current state-of-the-art systems:

• Performance Ceiling: Even the best-performing model, Gemini-2.5-Pro, achieved only a weighted average accuracy of 68.1%. Other models, such as Gemini-2.5-Flash (60.5%) and Qwen-2.5-Omni-7B (42.8%), showed markedly lower performance.
• Task Variability: Performance varied drastically by question type. Models faced considerable difficulties with categories such as "Sound-Only" (e.g., heart murmurs) and "Long-Form" reasoning, with models like Audio Flamingo 3 and GPT-4o-Audio exhibiting pronounced biases toward speech or pure sound inputs rather than demonstrating balanced cross-modal competence.
• Reasoning Limitations: Analysis of Chain-of-Thought (CoT) traces revealed that models often fail to integrate evidence across multiple turns or reject valid physiological audio inputs (e.g., GPT-4o-Audio rejected a significant proportion of heart and cough sound waveforms), leading to low accuracy in specific categories.

Significance

This work positions MedMosaic as a critical step toward evaluating the readiness of AI for real-world clinical deployment. The results underscore that despite advances in large language and audio models, medical reasoning under realistic constraints remains a persistent limitation. The benchmark highlights that current models struggle with the subtle, sparse, and temporally distributed nature of clinical evidence. By establishing a scalable paradigm for creating domain-specific benchmarks, MedMosaic provides a necessary tool to advance the development of more robust, domain-specific multimodal reasoning models capable of handling the complexity of medical audio recordings. The authors emphasize that while the benchmark advances evaluation, current performance levels do not indicate readiness for clinical use, necessitating further validation and model improvements.