Technical Summary: Robustness Assessment of Large Audio Language Models in Multiple-choice Evaluation

Problem Statement

The evaluation of Large Audio Language Models (LALMs) relies heavily on Multiple-Choice Question Answering (MCQA) benchmarks. However, current evaluation protocols are fragile. Subtle linguistic variations—such as reordering answer choices, paraphrasing questions, or rewording options—can yield substantially different performance metrics. Furthermore, existing benchmarks often contain "linguistic hints" that allow models to infer correct answers based on textual priors alone, without processing the audio signal. Current frameworks typically report a single accuracy score, masking these instabilities and potentially conflating genuine audio understanding with language bias. This paper argues that if accuracy does not capture genuine model capability, reported improvements may be artifacts of evaluation fragility rather than true advancement.

Methodology

The authors propose a systematic evaluation framework to assess the robustness of LALMs against linguistic perturbations while keeping the audio signal constant. The study spans three benchmarks (MMAU, MMAR, MMSU) and four models: Audio Flamingo 2, Audio Flamingo 3, Qwen2.5-Omni-7B-Instruct, and Kimi-Audio-7B-Instruct.

Perturbation Strategy

The methodology involves modifying the text input (questions and options) while preserving the original audio. The perturbations are applied individually and in combination:

1. Choice Ordering: All 24 possible permutations of the four answer choices are evaluated.
2. Rephrasing: Questions, ground-truth answers, and distractors are paraphrased using gemini-2.5-flash and gemma-3-12b-it. This generates six variations per element plus the original, resulting in seven distinct versions. Manual validation confirmed that semantic meaning was preserved.
3. Mixed Perturbations: A probabilistic approach (0.5 probability per modification) is used to combine all perturbation types, creating a practical setting to capture cumulative sensitivity.

Evaluation Metrics

Beyond standard mean accuracy, the study introduces two specific metrics to quantify robustness:

• Consistency Rate (CR): Measures the agreement among multiple responses to the same question under different perturbations (applicable when ground truth remains unchanged).
• Correctness Rate (CoR): A stricter metric that counts a question as correct only if the model answers it correctly under all evaluated perturbations. This captures the model's reliability across varying conditions.

Controls

To isolate language bias, the authors evaluate text-only Large Language Models (LLMs) on the same benchmarks (providing questions and options but no audio). This establishes a baseline for performance attributable to textual priors rather than audio understanding.

Key Results

Language Bias and Text-Only Performance

Text-only LLMs significantly outperform random chance on all benchmarks. Notably, gemma-3-27B-it achieves 48.3% accuracy on MMAU (22.6 percentage points above random chance), surpassing several dedicated audio models (e.g., LTU-AS, SALMONN) in text-only settings. On MMAU and MMAR, text-only models even outperform the audio model Qwen-Audio-Chat, indicating that current benchmarks contain exploitable textual shortcuts.

Sensitivity to Perturbations

• Choice Ordering: Models show moderate sensitivity. For instance, Kimi-Audio-7B-Instruct exhibits a standard deviation of 2.7% in accuracy on MMAU due to ordering changes.
• Question Rephrasing: Results in moderate variability, suggesting models have some capacity to handle semantic shifts in the prompt.
• Ground-Truth Answer Rephrasing: Causes higher variance in accuracy, particularly for Audio Flamingo 2 and Audio Flamingo 3.
• Distractor Rephrasing: This yields the most significant performance drops. Audio Flamingo 2 shows a standard deviation of up to 13.7% in accuracy on MMAU when distractors are rephrased. The Correctness Rate (CoR) drops drastically across all models under this condition (e.g., Audio Flamingo 2 CoR on MMAU drops to 0.21).

Model Comparisons

• Audio Flamingo 3: Achieves the highest overall accuracy and remains relatively robust to choice ordering and question rephrasing.
• Qwen2.5-Omni-7B: Demonstrates superior stability against distractor rephrasing, achieving the highest CoR across benchmarks in this category.
• Kimi-Audio-7B-Instruct: Performs competitively on ground-truth rephrasing but is heavily affected by distractor rephrasing and mixed perturbations.
• Audio Flamingo 2: Consistently ranks lowest, showing particularly weak resilience to distractor rephrasing.

Length Bias

The study identifies a bias toward selecting longer answer candidates. Since paraphrased choices tend to be longer, models often select the longest option. For example, Audio Flamingo 2 selected the longest choice 52.38% of the time, rising to 70.93% when the longest choice was actually correct. This suggests LALMs may rely on choice length as a selection shortcut.

Significance and Contributions

The paper claims to address a critical gap in LALM evaluation by demonstrating that standard MCQA metrics are insufficient for assessing true audio understanding. The specific contributions are:

1. Quantification of Language Bias: Establishing that text-only models can solve a significant portion of these benchmarks, necessitating text-only controls to isolate genuine audio capabilities.
2. Demonstration of Fragility: Proving that minor changes in choice ordering or wording (especially distractors) cause huge performance shifts, with accuracy standard deviations reaching up to 13.7%.
3. Identification of Length Bias: Uncovering a tendency for models to prefer longer answer candidates, which confounds evaluation when paraphrasing alters option length.
4. Proposed Evaluation Protocol: Introducing a "mixed-perturbation" setting scored via Correctness Rate (CoR). This approach captures model sensitivity to linguistic variations at a practical computational cost, offering a more reliable measure of robustness than mean accuracy alone.

The authors conclude that the field must move beyond single-number accuracy reports. By adopting a framework that accounts for linguistic variations and utilizes text-only baselines, researchers can better distinguish between true audio reasoning and artifacts of evaluation fragility.