💬 NLP

Hearing the Order: Investigating Position Bias in Large Audio-Language Models

This paper presents the first systematic investigation into position bias in Large Audio-Language Models, demonstrating through extensive experiments that answer choice ordering significantly impacts performance and reliability, while showing that permutation-based strategies can effectively mitigate this issue.

Original authors: Yu-Xiang Lin, Chen-An Li, Sheng-Lun Wei, Po-Chun Chen, Hsin-Hsi Chen, Hung-yi Lee

Published 2026-02-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yu-Xiang Lin, Chen-An Li, Sheng-Lun Wei, Po-Chun Chen, Hsin-Hsi Chen, Hung-yi Lee

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are taking a multiple-choice test. You know the answer is "C," but the test maker decides to move the options around so that "C" is now in the first spot, or the last spot, or the middle.

For a human, this doesn't matter. You read the question, find the right fact, and pick the right letter. But for Large Audio-Language Models (LALMs)—the super-smart AI computers that listen to sound and answer questions—this paper reveals a shocking secret: They are terrible at ignoring where the answer is placed.

Here is the story of the paper, explained simply with some analogies.

1. The Problem: The "Seat Preference" Syndrome

The researchers discovered that these AI models have a weird "seat preference." It's like a student who always guesses "A" because they are nervous, or a student who refuses to pick "D" because it feels too far away.

The Experiment: The team took six different AI models and asked them the same audio questions (like "What sound is this?").
The Twist: They shuffled the answer choices. Sometimes the correct answer was in spot A, sometimes B, C, or D.
The Result: The AI's score changed wildly just because the answer moved seats!
- Some models got 24% better or 24% worse just by moving the right answer to a different letter.
- Imagine if a basketball player made 80% of their shots when the hoop was on the left, but only 56% when the hoop was on the right. That's how unstable these AIs are.

2. The "Spoken" Twist

The researchers didn't just test text; they tested audio. They turned written questions into spoken words using a robot voice.

The Finding: The bias got even worse or changed patterns when the AI had to listen instead of read.
The Metaphor: It's like a person who is great at reading a menu but gets confused and picks the wrong dish when the waiter just says the options out loud in a specific order. The AI isn't just listening to the sound; it's getting distracted by the order the sounds come in.

3. The Identity Crisis: Is it the Letter or the Order?

The team wondered: "Is the AI confused because the answer is in position 'A', or is it confused because the letter 'A' is just a bad label?"

The Test: They tried removing the letters A, B, C, D entirely and just listing the options.
The Result: It didn't really help. The AI still had a bias toward certain positions (like the first or last spot), regardless of what letter was attached to them. It's not about the name of the seat; it's about where the seat is located in the row.

4. The "Magic Shuffle" Solution

So, how do we fix a student who can't handle seat changes?

The Old Way: Just give the test once and hope for the best. (This is unreliable).
The New Way (Permutation): The researchers tried a "Magic Shuffle." They asked the AI the same question 24 times, but every time, they shuffled the order of the answers differently.
The Voting: Then, they took a vote. If the AI picked the right answer 20 out of 24 times (regardless of where the answer was), they counted that as a win.
The Outcome: This worked! By averaging out all the different orders, the "seat preference" canceled itself out, and the AI's true intelligence shone through.
The Catch: It's slow. It's like asking a student to take the same test 24 times to get one fair grade. It's accurate, but it takes a lot of time and computer power.

5. Why This Matters: The Ranking Chaos

The most dangerous part of this discovery is how it changes who we think is the "best" AI.

The Metaphor: Imagine a race where the runners' starting positions are random. If Runner A always starts at the front and Runner B always starts at the back, Runner A might win every time, not because they are faster, but because they had a better start.
The Reality: The paper showed that when they shuffled the answers, the ranking of the models changed completely.
- The "winner" in one test became the "loser" in another, just because the answer choices were arranged differently.
- This means our current leaderboards (the lists that tell us which AI is the smartest) might be lying to us.

The Bottom Line

This paper is a wake-up call. It tells us that Large Audio-Language Models are currently very fragile. They are easily tricked by the order of things, much like a person who gets confused if you rearrange the furniture in a room.

To truly know how smart these AIs are, we can't just ask them a question once. We have to ask them the same question in every possible order and average the results. Until we do that, we don't really know who the "smartest" AI is; we just know who is best at guessing the right seat.

1. Problem Statement

The paper addresses a critical, yet previously unexplored, reliability issue in Large Audio-Language Models (LALMs): Position Bias.

Context: LALMs are increasingly evaluated using Multiple-Choice Questions (MCQs) on benchmarks like MMAU, MMAR, and MMLU.
The Issue: Similar to text-only Large Language Models (LLMs) and Vision-Language Models (VLMs), LALMs may not base their decisions solely on semantic content. Instead, their predictions can be systematically influenced by the order of answer options (e.g., preferring option A over D regardless of correctness).
Consequence: This bias undermines the validity of current evaluation practices. A model's reported performance may reflect structural weaknesses in its architecture or training rather than its true audio-reasoning capabilities, leading to misleading model rankings.

2. Methodology

The authors conducted a systematic investigation involving six state-of-the-art LALMs across three major benchmarks and their spoken counterparts.

A. Experimental Setup

Models Evaluated: Six LALMs covering diverse architectures and sizes:
- Gemini-2.0-Flash
- Phi-4-Multimodal
- Qwen2.5-Omni (3B and 7B)
- Voxtral-Mini (3B) and Voxtral-Small (24B)
Benchmarks:
- Text-based: MMAU, MMAR, MMLU.
- Speech-based (SPEECH-XX): Created by converting text questions and options into speech using GPT-4o mini TTS. Samples exceeding 180 seconds were filtered, retaining only those with exactly four options.
Bias Induction Strategy: The authors systematically reassigned the correct answer to each of the four positions (A, B, C, D) while randomly shuffling the distractors. This ensured the correct answer appeared equally often in every position, isolating the effect of position on model accuracy.

B. Metrics

To quantify bias and robustness, the study utilized:

$\Delta$ Accuracy: The difference in performance between the original dataset and the reassigned-answer settings.
Relative Standard Deviation (RSD): Measures the variability of accuracy across different option positions.
Choice Kullback-Leibler Divergence (CKLD): Measures the divergence between the model's prediction distribution and the ground-truth label distribution across positions.

C. Mitigation Strategies

The paper evaluates permutation-based strategies to mitigate bias:

Cyclic Permutation: Rotating options through positions.
Full Permutation: Testing all possible permutations of the answer options and determining the final answer via majority voting (similar to self-consistency but driven by option shuffling rather than sampling).

3. Key Results

A. Ubiquity of Position Bias

Universal Vulnerability: None of the six tested models were immune to position bias. All exhibited significant performance fluctuations based on option order.
Magnitude of Fluctuation: Shuffling options caused accuracy variations of up to 24% (observed in Phi-4-Multimodal).
Specific Biases:
- Phi-4-Multimodal: Strongly favored position A and avoided D.
- Voxtral-Mini-3B: Avoided D in text but preferred it in speech.
- Gemini-2.0-Flash: Generally preferred position D.
- Qwen2.5-Omni: Consistently avoided D and preferred A.

B. Impact of Identifiers

The study tested whether standard identifiers (A, B, C, D) mitigate bias.

Finding: While identifiers generally improved raw accuracy, they did not significantly reduce position bias or stabilize performance across shuffled orders.

C. LALMs vs. Text-Only LLMs

Inheritance vs. Divergence: Comparing LALMs to their text-only counterparts (e.g., Voxtral vs. Mistral, Qwen-Omni vs. Qwen-7B) revealed mixed results.
- Some models (Voxtral) inherited bias patterns from their text-only base.
- Others (Qwen series) showed distinct behaviors after audio-language instruction tuning, suggesting bias is not always a direct carry-over but can be altered by modality-specific training.

D. Model Ranking Instability

Ranking Swaps: Position bias significantly altered model rankings. For instance, in the MMAU benchmark, Qwen2.5-Omni-7B overtook Gemini-2.0-Flash when bias was mitigated. Similarly, rankings in MMAR shifted dramatically between Phi-4 and Voxtral models.
Conclusion: Conventional single-order evaluation yields unreliable leaderboards.

E. Effectiveness of Permutation

Mitigation: Both cyclic and full permutation strategies effectively reduced bias (lowering RSD and CKLD).
Performance Gain: Full permutation (evaluating all orders and voting) provided the most reliable estimates of true capability, often improving accuracy by several percentage points compared to the original setting.
Trade-off: These methods are computationally intensive but necessary for fair evaluation.

4. Key Contributions

First Systematic Investigation: This is the first paper to comprehensively analyze position bias specifically within the domain of Large Audio-Language Models.
Empirical Evidence of Flaw: Demonstrated that current LALM evaluations are fundamentally flawed due to sensitivity to option ordering, with fluctuations reaching up to 24%.
Benchmarking Framework: Established a rigorous protocol for testing position bias (reassigning correct answers to fixed positions) applicable to both text and speech modalities.
Validation of Mitigation: Proved that permutation-based evaluation (specifically full permutation with majority voting) is an effective, albeit costly, method to recover true model performance and stabilize rankings.

5. Significance

Reliability of Evaluation: The paper challenges the validity of current MCQ-based benchmarks for LALMs. It suggests that reported scores may be artifacts of option ordering rather than genuine reasoning ability.
Call to Action: It urges the research community to adopt permutation-based evaluation as a standard practice or to develop new, bias-resistant evaluation frameworks.
Future Directions: Highlights the need for architectural or training-level solutions to eliminate position bias entirely, moving beyond the computationally expensive permutation workaround to ensure fair and efficient deployment of LALMs.