SpeakerSleuth: Can Large Audio-Language Models Judge Speaker Consistency across Multi-turn Dialogues?

1. Problem Statement

While Large Audio-Language Models (LALMs) have emerged as powerful tools for evaluating speech generation quality, their ability to assess speaker consistency across multi-turn dialogues remains unexplored.

• The Challenge: In multi-speaker dialogue generation (e.g., for movies or voice assistants), maintaining a consistent speaker identity (timbre, pitch, voice quality) across all utterances is critical. Current generation models often suffer from "speaker confusion," "timbre drift," or voice quality variations that are only detectable when listening to the full dialogue context, not in isolation.
• The Gap: Existing evaluation methods rely on pairwise acoustic similarity (embedding models) with manually set thresholds, which fail to assess consistency holistically. It is unclear if LALMs, which process both text and audio, can reliably act as "judges" for speaker consistency or if they prioritize textual coherence over acoustic fidelity.

2. Methodology: The SpeakerSleuth Benchmark

The authors introduce SpeakerSleuth, the first benchmark designed to systematically evaluate speaker consistency in multi-turn dialogues.

A. Data Construction

• Scale: 1,818 human-verified evaluation instances derived from 197 unique speakers across four diverse datasets: Bazinga (TV/Movies), AMI (Business meetings), Behavior-SD (Synthesized speech), and DailyTalk (Casual conversation).
• Scenario Generation: For each dialogue, three scenarios are created to test different levels of acoustic difficulty:
  1. S1 (Fully Consistent): Original dialogue where the target speaker's turns are consistent.
  2. S2 (Gender Switch): One turn is voice-converted to an opposite-gender speaker (clear acoustic deviation).
  3. S3 (Similar Speaker): One turn is voice-converted to an acoustically similar speaker (subtle timbre deviation), selected via ECAPA-TDNN embeddings.
• Verification: A rigorous pipeline involving automated text filtering and human audio verification ensures naturalness, clarity, and correct scenario labeling.

B. Task Formulation

The benchmark evaluates LALMs on three distinct tasks reflecting real-world requirements:

1. Detection: Binary classification. Given a set of turns from a target speaker, does the model determine if they are consistent (YES/NO)?
2. Localization: Fine-grained identification. If inconsistent, which specific turn(s) are problematic?
3. Discrimination: Relative ranking. Given a masked turn and three candidates (Original, Similar Speaker, Different Speaker), can the model rank them by acoustic similarity to the target?

C. Experimental Setup

• Models Evaluated: 12 state-of-the-art LALMs (including GPT-4o-audio, Gemini-2.5-Pro/Flash, Qwen2.5/3-Omni, MiniCPM-o, etc.) and 6 speaker embedding methods (using WavLM and ECAPA-TDNN backbones).
• Input Conditions: Primary evaluation uses audio-only input with a reference sample. Ablation studies test the impact of adding textual context (transcripts of other speakers) and removing the reference audio.

3. Key Results

A. Detection and Localization Failures

• Unstable Thresholds: LALMs struggle with absolute binary judgments. Models exhibit extreme biases: some (e.g., MiniCPM-o, Audio-Flamingo-3) are overly lenient (predicting "consistent" even with gender switches), while others (e.g., Gemini-2.5-Flash-Lite) are overly strict.
• Poor Balanced Accuracy: Most models score below 60% on balanced accuracy. The best model, Gemini-2.5-Pro, achieves ~64.7% but struggles significantly with subtle timbre differences (S3).
• Localization Failure: Models that detect inconsistency generally fail to pinpoint the specific turn. Many default to flagging all turns or no turns, showing an inability to perform fine-grained turn-level analysis.

B. The Modality Imbalance (Text vs. Audio)

• Textual Context Degrades Performance: Contrary to the hypothesis that text helps anchor the conversation, providing textual transcripts of other speakers dramatically degrades acoustic judgment.
• Mechanism: Models prioritize textual coherence over acoustic cues. If the dialogue text flows logically, models tend to declare the speaker consistent, even when obvious acoustic switches (like gender changes) occur. This reveals a fundamental modality imbalance where LALMs attend disproportionately to text tokens.

C. Discrimination Success

• Relative Judgment Works: While models fail at absolute detection, they perform significantly better in Discrimination (ranking candidates).
• Dissociation: Strong models like Gemini-2.5-Pro achieve high classification accuracy (~81.5%) and ranking metrics (NDCG@2 ~92.6%) in discrimination tasks. This suggests LALMs possess inherent acoustic discrimination capabilities but fail to integrate them into absolute consistency judgments without explicit reference anchors.

D. Impact of Reference Audio

• Removing the reference audio sample causes many models to default to "Consistent" judgments, highlighting their reliance on explicit anchors rather than building robust internal speaker representations from dialogue context alone.

4. Key Contributions

1. SpeakerSleuth Benchmark: A comprehensive, human-verified benchmark with 1,818 instances across four datasets, covering detection, localization, and discrimination tasks.
2. Systematic Evaluation: The first large-scale evaluation of 12 LALMs and 6 embedding methods for multi-turn speaker consistency, revealing critical performance gaps.
3. Discovery of Modality Imbalance: Identification of a critical flaw where LALMs prioritize textual coherence over acoustic evidence, leading to failures in detecting obvious inconsistencies (e.g., gender switches) when context is provided.
4. Detection-Discrimination Dissociation: Evidence that LALMs can discriminate acoustic variants effectively (relative judgment) but fail at absolute consistency verification (binary judgment) due to unstable internal thresholds.

5. Significance and Implications

• Reliability of AI Judges: The findings suggest that current LALMs are not yet reliable as standalone judges for speaker consistency in multi-turn dialogues, particularly when textual context is involved.
• Future Directions: Building reliable audio-language judges requires:
  • Better calibration of internal decision thresholds.
  • Mechanisms to balance attention between text and audio modalities to prevent text from overriding acoustic cues.
  • The necessity of providing reference audio samples in real-world applications (e.g., TTS validation) to anchor judgments.
• Broader Impact: As synthetic speech becomes indistinguishable from human speech, robust evaluation tools are essential for preventing fraud, misinformation, and ensuring the quality of voice assistants and media production. SpeakerSleuth provides the framework to probe and improve these capabilities.

Code and Data Availability: The authors have open-sourced the benchmark, code, and data at https://github.com/holi-lab/SpeakerSleuth.