ParaPairAudioBench: Paralinguistic Pairwise Audio Benchmark for LALM-as-a-Judge

Technical Summary: PARAPAIRAUDIOBENCH

Problem Statement
While Large Audio-Language Models (LALMs) have become standard tools for the automatic evaluation of generated speech, current methodologies predominantly focus on holistic naturalness. This approach leaves fine-grained paralinguistic distinctions—such as speaking rate, emphasis, style, age, and gender—underexplored. Existing evaluation frameworks struggle to determine whether synthesized speech accurately adheres to specific, intended paralinguistic cues. Furthermore, the reliability of LALMs acting as "judges" in controlled pairwise settings, particularly regarding their ability to abstain from making a decision when samples are indistinguishable (calibration) and their dependence on lexical versus acoustic cues, remains largely uninvestigated.

Methodology
To address these gaps, the authors introduce PARAPAIRAUDIOBENCH, a diagnostic pairwise benchmark comprising 5,175 audio pairs. The benchmark is designed to evaluate LALM-as-a-Judge systems across five distinct paralinguistic dimensions:

1. Style: Expressive attributes (e.g., whispering, happiness).
2. Rate: Global temporal pacing.
3. Emphasis: Localized prosodic prominence at the word level.
4. Age: Speaker age characteristics.
5. Gender: Speaker gender characteristics.

The dataset is constructed from validated public corpora (Expresso, Sonos Voice Control, LibriTTS, EARS) and adheres to strict design principles:

• Pairwise Format: Each instance presents two audio samples (A and B) with an instruction. The judge must select A, B, or [[Tie]] (indicating a draw or indistinguishability).
• Tie Conditions: The benchmark explicitly includes "Tie" cases (where both samples are equally good or equally bad) to test calibration. For Emphasis, ties are restricted to "Both Bad" instances due to the difficulty of verifying perceptually identical stress profiles without frame-level annotation. Rate excludes ties entirely due to natural rhythmic variations.
• Transcript Control: To isolate acoustic reliance from textual reliance, pairs are balanced between Same-Transcript (identical text) and Cross-Transcript (different text) conditions.
• Robustness Checks: To measure position bias, every pair is evaluated twice with swapped input orders (Audio A first vs. Audio B first).

The study evaluates five representative LALMs (Gemini 2.5 Flash, GPT-4o Audio, SpeechJudge-7B, Kimi-Audio-7B, Qwen2.5-Omni-7B) against human raters.

Key Results
The experiments reveal significant limitations in current LALM judges compared to human performance:

• Performance Gap: Even the strongest model, Gemini 2.5 Flash, lags behind human judgment by an average of 17.7 percentage points. Across all models, the average gap to human performance is 32 percentage points.
• Calibration Failure: Models exhibit severe failure in "Tie" scenarios. Instead of abstaining when samples are indistinguishable, models frequently force a preference. For instance, while humans achieve over 75% accuracy in identifying ties for Emphasis, Style, and Gender, models often drop to below 20% accuracy in these same categories. SpeechJudge-7B shows the most extreme collapse, with tie accuracy as low as 1.6–1.7%.
• Criterion-Specific Weaknesses:
  • Models perform relatively well on Rate (a global cue) but struggle with Emphasis (a localized cue).
  • Age estimation is inherently difficult even for humans (low inter-rater agreement), though some models slightly exceed human performance in non-tie cases, they fail to calibrate correctly in ambiguous cases.
• Lexical vs. Acoustic Dependency: A transcript-controlled analysis reveals an asymmetric reliance on modality:
  • Style: Models heavily rely on lexical content. Performance drops significantly (e.g., Gemini 2.5 Flash drops from 83.8% to 36.6%) when transcripts differ, indicating they use text cues rather than acoustic prosody for style judgment.
  • Emphasis: The trend reverses; models perform better with cross-transcript pairs. This suggests that global prosodic scaffolding in different sentences helps compensate for weak local sensitivity to word-level emphasis within identical utterances.
• Position Bias: All evaluated models exhibit systematic position bias. SpeechJudge shows the largest bias (29.4% gap between Acc@A and Acc@B), consistently favoring the second position, while Qwen2.5-Omni favors the first.

Significance and Contributions
The paper claims three primary contributions:

1. Diagnostic Multi-dimensional Evaluation: By decomposing paralinguistic judgment into five criteria, the benchmark reveals that models excel at globally distributed cues (Rate) but fail at localized prosodic prominence (Emphasis), a distinction invisible to aggregate naturalness scores.
2. Calibration and Robustness Assessment: The inclusion of explicit Tie conditions and order swaps exposes systematic calibration failures and position biases (up to 29.4%p) that are critical for deploying LALMs as reliable judges.
3. Modality-Aware Dependency Analysis: The benchmark demonstrates that current LALMs often over-rely on textual cues for certain attributes (Style) while under-utilizing acoustic context for others (Emphasis), highlighting the need for evaluation protocols that control for transcript matching.

Conclusion
PARAPAIRAUDIOBENCH serves as a controlled diagnostic tool to assess the reliability of LALM-as-a-Judge systems. The authors conclude that while LALMs show promise, they currently lack the fine-grained perceptual sensitivity and calibration required for robust paralinguistic evaluation. The benchmark prioritizes reliable diagnostic contrasts over exhaustive coverage, providing a foundation for future work on debiasing and improving calibration in audio evaluation pipelines.