Technical Summary: PAREDA – A Multi-Accent Speech Dataset of Natural Language Processing Research Discussions

1. Problem Statement

While modern Automatic Speech Recognition (ASR) systems achieve high accuracy on standard benchmark corpora, their performance frequently degrades when exposed to real-world variability, specifically accented speech, spontaneous conversation, and domain-specific terminology. Existing state-of-the-art (SOTA) models, such as Whisper, exhibit significant performance gaps when processing non-mainstream English accents (e.g., Nigerian, Indian) compared to mainstream American English. Furthermore, there is a lack of publicly available resources designed to evaluate ASR performance on domain-specific academic speech that simultaneously involves non-mainstream English accents and technical jargon. Current datasets either focus on read speech (lacking conversational nuance) or general-domain dialogues (lacking technical depth), failing to address the intersection of dialectal variation and specialized vocabulary found in academic settings.

2. Methodology

Dataset Creation (PAREDA)

The authors introduce PAREDA (PAper REading DAtaset), a small-scale multi-accent speech dataset comprising 3.9 hours of recorded audio. The dataset is constructed through the following methodology:

• Speakers: Three participants representing non-mainstream English accents: Australian (en-AU), Indian (en-IN), and Northern Chinese (en-ZH). All possess native or superior English proficiency and experience in NLP research (ranging from students to a lecturer).
• Content: Discussions centered on 21 NLP research papers sourced from the ACL Anthology, categorized into applied linguistics, linguistic research, bias mitigation, and historical/cultural linguistics.
• Interaction Types:
  • Monologue: A 2-minute summary of a paper's abstract (or full body if the abstract is non-informative).
  • Non-Monologue (Dialogue): A structured 5-minute question-and-answer session where a host (en-ZH speaker) questions respondents (en-AU and en-IN speakers).
• Preprocessing: Audio was manually segmented into clips under 30 seconds to accommodate model input constraints. Transcriptions were generated using CrisperWhisper and manually post-edited, standardized to American English spelling conventions. Inter-annotator reliability was verified with a mean symmetric Word Error Rate (WER) of 2.77%.

Experimental Setup

The study evaluates the performance of SOTA ASR models under various conditions:

• Models: Whisper API (Large), Phi-4, and CrisperWhisper.
• Training Strategy: A two-stage fine-tuning procedure for Whisper models:
  1. Stage 1: Fine-tuning on GLOBE, a large-scale multi-accent English corpus, to improve accent robustness.
  2. Stage 2: Further fine-tuning on the PAREDA dataset to adapt to domain-specific academic speech.
• Evaluation Conditions: The models were tested under normal conditions, accelerated speech (1.5x speed), and with added background noise (-10dB). Cross-variety evaluations were also conducted to test generalization between accents.

3. Key Results

Zero-Shot Performance and Baseline

In a zero-shot setting, SOTA models demonstrated poor performance on PAREDA, confirming the dataset's challenging nature. Whisper API exhibited the highest WER (up to 18.21% on Australian samples), while CrisperWhisper performed best among the zero-shot models (lowest WER of 4.08% on Indian samples).

Impact of Fine-Tuning

Fine-tuning on PAREDA yielded significant improvements:

• Domain Adaptation: Models fine-tuned on PAREDA (Stage 2) reduced WER by at least 10% compared to previous stages. For instance, the Whisper Medium model's WER dropped from 13.46% (baseline) to 4.53% after PAREDA fine-tuning.
• GLOBE Limitations: Fine-tuning solely on the GLOBE corpus (Stage 1) resulted in a slight increase in WER across all model sizes, suggesting that broad multi-accent data does not necessarily improve recognition in specialized professional fields without domain-specific adaptation.

Environmental and Linguistic Factors

• Speech Rate and Noise: Increasing speech speed to 1.5x or adding -10dB background noise caused sharp drops in accuracy for all models. The gap between systems narrowed, indicating that none of the architectures have learned truly tempo-invariant acoustic representations.
• Accent Mixing: The introduction of the en-ZH accent as a condition did not result in a consistent, clear shift in WER across all models; performance varied depending on the specific model and accent combination.
• Error Analysis: Qualitative analysis of Whisper-tiny errors revealed two primary failure pathways:
  1. Prosodic Deletion: High error rates in function words (articles, prepositions) and fillers due to vowel reduction in accented speech.
  2. Acoustic Mis-modeling: Technical vocabulary (NLP terms) showed an average error rate of 45.8%, which is six times higher than the 7.6% error rate for grammatical words. This suggests that uncommon phoneme sequences combined with accent shifts overwhelm the recognizer.

Per-Accent Tuning

Targeted fine-tuning on specific accent subsets yielded mixed results. For smaller models (Tiny/Small), per-accent tuning generally worsened performance compared to the all-accent baseline. Conversely, larger models (Medium) benefited from per-accent tuning, showing WER reductions across all accents.

4. Significance and Contributions

The paper positions PAREDA as a first-of-its-kind resource addressing a critical gap in ASR research: the intersection of dialectal variation and domain-specific technical vocabulary in spontaneous speech.

• Benchmarking: PAREDA serves as a challenging testbed that exposes the limitations of current SOTA models in real-world academic scenarios, particularly regarding technical jargon and accent diversity.
• Validation of Fine-Tuning: The study demonstrates that while broad multi-accent datasets (like GLOBE) provide breadth, they are insufficient for specialized domains. Fine-tuning on small, domain-specific datasets is a highly effective strategy for improving ASR robustness in low-resource, specialized contexts.
• Future Directions: The authors suggest that PAREDA can be used to investigate the propagation of ASR errors into downstream NLP tasks and to develop novel bias mitigation techniques. They advocate for future work to expand the dataset to include more speakers and a wider variety of global Englishes, emphasizing that addressing the dual challenges of prosody and technical vocabulary is essential for creating inclusive speech recognition technologies.