💬 NLP

ORCA: Open-ended Response Correctness Assessment for Audio Question Answering

This paper introduces ORCA, a lightweight model-based framework for assessing open-ended responses in audio question answering that leverages a three-stage human-AI annotation pipeline to achieve high correlation with human judgments and effectively identify problematic benchmark items.

Original authors: Šimon Sedláček, Sara Barahona, Bolaji Yusuf, Laura Herrera-Alarcón, Santosh Kesiraju, Cecilia Bolaños, Alicia Lozano-Diez, Sathvik Udupa, Fernando López, Allison Ferner, Ramani Duraiswami, Jan Černock

Published 2026-06-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Šimon Sedláček, Sara Barahona, Bolaji Yusuf, Laura Herrera-Alarcón, Santosh Kesiraju, Cecilia Bolaños, Alicia Lozano-Diez, Sathvik Udupa, Fernando López, Allison Ferner, Ramani Duraiswami, Jan Černocký

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to understand the world through sound—speech, music, and environmental noises. You have a new generation of "Audio Language Models" (LALMs) that can listen to a recording and answer questions about it. But how do you know if the robot is actually right?

For a long time, researchers used a multiple-choice quiz format (like "A, B, C, or D?") because it's easy to grade automatically. But in the real world, people don't just pick options; they give open-ended answers. Grading these free-form answers is like trying to grade a student's essay: it's subjective, and different teachers might give different scores for the same answer.

This paper introduces ORCA (Open-ended Response Correctness Assessment), a new tool designed to be the "super-teacher" for these audio robots. Here is how it works, broken down into simple concepts:

1. The Problem: The "Teacher Disagreement"

Imagine you ask a class of human teachers to grade a student's answer to a tricky question.

Scenario A: The question is clear, and all teachers agree the answer is "Good." (Low disagreement).
Scenario B: The question is vague. Some teachers think the answer is brilliant; others think it's wrong. (High disagreement).

Old grading tools would just take the average score and ignore the fact that the teachers were fighting about it. ORCA is different. It doesn't just predict the average score; it also predicts how much the teachers would argue about that answer. If ORCA predicts a high "disagreement score," it's telling you, "Hey, this question is tricky, and humans can't agree on the answer."

2. The Solution: A Three-Stage Training Camp

To build ORCA, the researchers didn't just throw data at a computer. They used a "curriculum learning" approach, which is like training an athlete in three phases:

Phase 1: The Synthetic Drill. They used other AI models to generate millions of fake practice questions and answers. This gave ORCA a massive amount of basic training data without needing human time yet.
Phase 2: The Simulation. They took real benchmark questions and had AI models generate answers and "judge" them. This bridged the gap between fake data and real data.
Phase 3: The Human Touch. Finally, they brought in real human experts. They asked humans to grade answers from 15 different audio robots. Crucially, they didn't just ask for a score (1 to 5); they asked for feedback. If a human said, "I can't answer this because the question is unclear," ORCA learned to flag that.

This process resulted in a dataset of nearly 10,000 human annotations, which they used to teach ORCA how to think like a human grader.

3. The Magic Trick: Text-Only Evaluation

You might think, "To grade an audio answer, you need to listen to the audio." Surprisingly, ORCA doesn't need to listen to the sound file itself.

Instead, ORCA looks at a text summary (called a "rationale") that explains why an answer is correct based on the audio. It's like a teacher reading a student's essay about the audio, rather than listening to the audio themselves. This makes ORCA incredibly fast and efficient because it doesn't have to process heavy audio files every time it grades.

4. The Results: Better than the Giants

The researchers tested ORCA against some of the biggest, most powerful AI models available (like Google's Gemini).

Accuracy: ORCA was better at predicting what humans would think than even the most expensive, massive AI models.
Efficiency: ORCA is "lightweight." It's a small model that runs fast, whereas the competitors are huge and slow.
The "Disagreement" Superpower: ORCA successfully identified which questions were confusing or ambiguous. When humans disagreed on an answer, ORCA's "uncertainty meter" went up, signaling that the question itself might be flawed.

5. Why This Matters

The paper claims that ORCA is a reliable, fast, and smart way to evaluate how well audio robots are learning. By modeling human disagreement, it helps researchers find the "bad apples" in their test questions—those confusing questions that trip up even the smartest humans.

In short, ORCA is a lightweight, smart grader that not only tells you if an audio robot is right or wrong but also warns you when the question itself is too confusing to have a single right answer.

Technical Summary: ORCA – Open-ended Response Correctness Assessment for Audio

1. Problem Statement

The evaluation of Large Audio Language Models (LALMs) is currently dominated by Multiple-Choice Question (MCQ) frameworks due to their ease of automation. However, MCQs fail to capture the nuances of real-world usage, where open-ended responses are more natural. Evaluating these open-ended responses presents significant challenges:

Semantic Equivalence: Traditional lexical metrics (e.g., BLEU, F1) cannot handle semantically equivalent answers that differ in phrasing.
Subjectivity and Disagreement: Audio question answering (QA) often involves subjective judgments regarding acoustic events, musical characteristics, or emotional tone. Human annotators frequently disagree on the correctness of answers, a phenomenon often treated as noise in classical approaches.
Evaluation Costs: Existing model-based evaluators (LLM-judges) often suffer from high computational costs, prompt sensitivity, and a lack of calibration. Furthermore, using audio models to judge audio models creates circular dependencies.

The paper argues that annotator disagreement often reflects genuine ambiguity in the data rather than mere noise, and that current evaluation metrics fail to model this distributional information, leaving them unable to identify problematic benchmark items.

2. Methodology

2.1 ORCA Framework

The authors propose ORCA (Open-ended Response Correctness Assessment), a lightweight, model-based framework designed to score free-form answers from LALMs while explicitly modeling the variability in human judgments.

Text-Only Evaluation: To ensure efficiency and avoid circular dependencies, ORCA evaluates answers augmented by textual grounding (rationales) that summarize pertinent audio context. This allows text-based models to assess audio QA without processing raw audio.
Distributional Modeling: Unlike standard regressors that predict a single mean score, ORCA predicts the full distribution of human correctness judgments. It employs three likelihood functions:
1. Beta Distribution: Models ratings as a continuous distribution on $[0, 1]$ , capturing variance and consensus.
2. Multinomial Distribution: Models ratings as discrete categories (1–5), learning cross-category correlations.
3. Bernoulli Distribution: A baseline predicting only the mean correctness (no variance).
Architecture: ORCA is initialized from pre-trained text LLMs (e.g., OLMo-2, Gemma 3, Llama 3.2) and fine-tuned using LoRA. The model takes concatenated inputs of the question, reference answer, rationale, and candidate answer, outputting parameters for the chosen likelihood function.

2.2 Three-Stage Annotation Pipeline

To generate high-quality training data and improve benchmark quality, the authors developed a three-stage pipeline:

Stage 1 (Generation):
- Rationale Generation: Uses Gemini-2.5-Flash to generate textual rationales (justifications) for reference answers based on audio.
- Candidate Generation: Prompts 15 state-of-the-art LALMs to generate open-ended responses for benchmark questions.
Stage 2 (Annotation & Feedback): Human annotators rate candidate answers (1–5 scale) and provide structured feedback on issues (e.g., incomplete questions, insufficient rationales, incorrect references). LLM-judges perform parallel evaluations.
Stage 3 (Correction): Human experts review flagged instances and implement corrections (rephrasing questions, enhancing rationales, fixing references) with AI assistance, ensuring high-quality training data.

2.3 Curriculum Learning

ORCA is trained using a three-stage curriculum:

Stage 1 (s1): Synthetic data from LLM-judges on general QA pairs (no rationales).
Stage 2 (s2): Synthetic data from LLM-judges on benchmark questions with rationales.
Stage 3 (s3): Human-annotated data from the refined benchmarks.
This approach progressively increases data quality and specificity, reducing dependency on extensive human-annotated datasets.

3. Key Contributions

ORCA Model: A lightweight framework that predicts the distribution of human correctness judgments (Beta or Multinomial), enabling robust uncertainty quantification for open-ended audio QA.
Annotation Framework: A three-stage pipeline combining human judgment, structured feedback, and human-AI correction, yielding 9,663 annotations across 3,699 question-answer pairs from 15 LALMs on three benchmarks (MMAU, MMAR, MMAU-Pro).
High-Quality Dataset: The process achieved a Krippendorff's $\alpha$ of 0.82 post-filtering, significantly improving upon the initial agreement of 0.76 by filtering out unreliable instances.
Generalization: Extensive experiments demonstrate ORCA's ability to generalize to unseen LALMs and unseen benchmarks.
Open Release: The authors release trained models, source code, and the curated annotation dataset.

4. Results

4.1 Performance vs. LLM-Judges

ORCA models (specifically the Multinomial and Bernoulli variants based on Llama 3.2-3B and OLMo2-1B) outperform several open-weight LLM-judge baselines and the proprietary Gemini 2.5 Flash in predicting average human correctness ratings.

Correlation: On the unseen MMAU-Pro benchmark, the Llama 3.2-3B (Multinomial) ORCA achieved a Spearman correlation ( $\rho$ ) of 0.8451 and a Mean Absolute Error (MAE) of 7.79%, outperforming Gemini 2.5 Flash ( $\rho=0.8389$ , MAE=8.90%).
Efficiency: ORCA requires only a single forward pass, offering significantly lower inference costs compared to API-based judges or larger models like Gemma 3-12B.

4.2 Generalization

Unseen LALMs: In scenarios where ORCA is tested on LALMs not seen during training, it consistently outperforms Gemini 2.5 Flash, particularly on models with high response variance (e.g., Audio Reasoner).
Unseen Benchmarks: ORCA generalizes effectively to the MMAU-Pro benchmark, which was never used in training.

4.3 Uncertainty and Disagreement Modeling

Variance Correlation: ORCA's predicted variance strongly correlates with human inter-annotator disagreement. This allows the model to identify items where human raters genuinely disagree.
Flagged Item Detection: ORCA assigns significantly higher predictive variance to benchmark items flagged by human reviewers as ambiguous or problematic (e.g., unclear questions, insufficient rationales) compared to clean items. This demonstrates its utility in diagnosing benchmark quality issues.

4.4 Curriculum Learning Impact

Ablation studies confirm that the three-stage curriculum ( $s1 \to s2 \to s3$ ) yields the best performance. Pre-training on synthetic LLM-judge data ( $s1, s2$ ) significantly improves performance on human-annotated data ( $s3$ ) compared to training on $s3$ alone.

5. Significance and Claims

The paper positions ORCA as a foundational step toward shifting audio QA evaluation from restrictive multiple-choice paradigms to natural, real-world open-ended assessments.

Reliability: ORCA provides a reliable, lightweight alternative to expensive LLM-judges, achieving higher accuracy with lower computational cost.
Uncertainty Quantification: By modeling the distribution of human judgments, ORCA captures genuine ambiguity in subjective tasks, offering end-users confidence scores and helping benchmark creators identify and correct flawed items.
Data Quality: The annotation pipeline itself serves as a quality control mechanism, exposing and correcting data quality issues in existing benchmarks (MMAU, MMAR), thereby releasing improved versions of these datasets.
Scalability: The text-only evaluation approach, augmented by rationales, allows for efficient scaling independent of audio duration and avoids the circularity of audio-model-judging-audio-models.

The authors conclude that while the Bernoulli objective achieves the highest absolute correlation for mean scores, the Multinomial distribution offers the most robust framework by balancing correctness prediction with the effective modeling of human disagreement.