The Impact of Automatic Speech Transcription on Speaker Attribution

Technical Summary: The Impact of Automatic Speech Transcription on Speaker Attribution

Problem Statement
Speaker attribution from speech transcripts involves identifying a speaker based on linguistic patterns in their text, a task distinct from acoustic speaker recognition which relies on voice properties. While prior research has established the feasibility of this task using human-transcribed ("gold standard") data, real-world applications often rely on Automatic Speech Recognition (ASR) systems. These systems introduce transcription errors (substitutions, deletions, insertions) that could theoretically degrade attribution performance. However, the extent to which ASR errors impact speaker attribution, and whether the objective of recovering the "true" transcript correlates with attribution success, remains unexplored.

Methodology
The authors conducted a comprehensive study using the Fisher English Training Speech Transcripts corpus, specifically focusing on the "hard" difficulty setting where conversational topics are controlled to prevent topic-based leakage.

1. ASR Systems: Five distinct ASR systems were employed to generate transcripts from the same audio data, creating a spectrum of error rates (measured via concatenated minimum-permutation Word Error Rate, or cpWER):

   • GigaSpeech (14% cpWER)
   • AssemblyAI (19% cpWER)
   • Switchboard (21% cpWER)
   • Whisper (26% cpWER)
   • TED-LIUM3 (32% cpWER)
   • A German ASR system was also used to generate ~90% error transcripts for floor-performance analysis.

2. Attribution Models: Five authorship attribution models, previously trained on written text, were tested "out-of-the-box" on these transcripts:

   • SBERT: Semantic similarity (content-focused).
   • StyleD (StyleDistance): Content-independent style representation.
   • LUAR: Universal authorship representations (content + style).
   • CISR: Content-independent style representations.
   • PANgrams: Character 4-grams with TF-IDF weighting.

3. Experimental Design:

   • Error Correlation: Models were evaluated on ASR transcripts with varying cpWERs to determine if error rates correlate with attribution performance (AUC).
   • Masking Experiments: Content words and style words (fillers, function words) were masked separately to isolate which word types drive errors and attribution signals.
   • Floor Analysis: Transcripts were artificially corrupted by replacing all tokens with a single random word, and utterance lengths were manipulated to determine the minimum information required for attribution.
   • Probing and Generalization: A probing classifier (MLP) was trained on top of model embeddings to assess if fine-tuning helps. Additionally, models trained on gold-standard data were tested on ASR transcripts to evaluate train-test mismatch.

Key Results

• Resilience to Errors: Contrary to intuition, speaker attribution performance does not significantly degrade as transcription error rates increase. In several cases, models performed better on ASR transcripts (up to 32% cpWER) than on gold-standard human transcripts. For instance, LUAR achieved 0.74 AUC on ASR data versus 0.71 AUC on gold data.
• Decoupling of WER and Attribution: The Word Error Rate (WER) is a poor predictor of attribution performance. The study found that ASR errors often occur on "style words" (fillers, restarts, non-standard pronunciations) rather than content words. These errors can inadvertently capture speaker-specific idiosyncrasies, providing useful signals for attribution.
• Content vs. Style:
  • Models relying heavily on content (SBERT) performed below chance on the "hard" setting (where topics are controlled) but improved significantly when content words were masked, suggesting they were previously misled by topic overlap.
  • Style-focused models (StyleD, LUAR) maintained or improved performance, indicating that stylistic features remain robust despite transcription noise.
• The Role of Utterance Length: Even when content was completely removed (replaced by random words), attribution performance remained above chance. The study identified utterance length (the number of tokens a speaker produces) as a surprisingly strong feature. When utterance lengths were equalized across speakers, performance dropped to chance levels. This suggests that in the absence of linguistic content, models rely on how much a speaker speaks.
• Generalization Gap: Models trained on gold-standard transcripts do not generalize well to ASR transcripts, indicating a domain mismatch that current models cannot easily bridge without specific adaptation.

Significance and Claims
The paper claims to be the first comprehensive study of the impact of automatic transcription on speaker attribution. Its primary contribution is the finding that speaker attribution is surprisingly resilient to word-level transcription errors, and that the goal of recovering a perfect transcript is minimally correlated with the goal of identifying a speaker.

The authors suggest that ASR errors are not merely noise; they can capture speaker-specific features (such as idiosyncratic pronunciation or filler usage) that are revealing of speaker identity. Consequently, speaker attribution on errorful ASR transcripts can be as effective, or even more effective, than on human-transcribed data.

Limitations and Ethical Considerations
The authors caution that the reliance on utterance length as a feature may be an artifact of the Fisher dataset's specific structure (e.g., how turns are segmented). They also warn against applying these methods in high-stakes forensic settings due to potential spurious correlations (e.g., models exploiting topic controls rather than true speaker identity). Furthermore, they note that while ASR errors can be helpful, they could also be detrimental in scenarios involving voice cloning, where the content might indicate one speaker while transcription errors indicate another, creating conflicting signals.