Technical Summary: Toward Conversational Hungarian Speech Recognition

Problem Statement

The field of Automatic Speech Recognition (ASR) has been significantly advanced by large-scale datasets in high-resource languages, yet low-resource languages like Hungarian remain underrepresented. While recent self-supervised learning and multilingual models offer promise, their effectiveness is hindered by a scarcity of high-quality, diverse speech corpora that capture the full spectrum of natural human communication. Hungarian, characterized by its morphologically rich and agglutinative nature, presents specific challenges even for large multilingual models. Furthermore, existing Hungarian resources often suffer from inconsistent annotation practices, restricted accessibility, or a lack of spontaneous and conversational data, which are critical for training robust conversational ASR and speaker diarization systems.

Methodology

To address these gaps, the authors constructed two new datasets derived from the previously unprocessed portions of the Hungarian speech corpus known as BEA (Neuberger et al., 2014).

Dataset Construction

1. BEA-Large: This dataset extends the previously released BEA-Base benchmark. It incorporates an additional training subset (train-293) comprising 293 speakers not present in BEA-Base, bringing the total to 433 speakers and 255 hours of spontaneous speech.

   • Metadata Enrichment: Unlike BEA-Base, BEA-Large includes detailed segment-level metadata, such as speaker age, gender, occupation, and specific roles (Target Speaker/SPK, Experiment Leader/EXP, Discourse Partner/DP).
   • Structure: The dataset maintains the original development and evaluation partitions of BEA-Base to ensure experimental comparability while expanding the training data threefold.

2. BEA-Dialogue: Derived from 242 speakers not included in BEA-Base, this dataset focuses specifically on natural dialogues.

   • Segmentation: Utterances were extracted and grouped into coherent dialogue units based on silent intervals, then merged into segments with a target duration of 30 seconds.
   • Speaker Independence: To ensure rigorous evaluation for conversational ASR and diarization, the dataset was partitioned into training, development, and evaluation sets based on three distinct experiment leaders (fem1, fem3, fem4). This ensures complete speaker independence across subsets for all three roles (SPK, EXP, DP).
   • Content: The dataset includes 85 hours of spontaneous conversations, featuring overlapping speech and natural turn-taking.

Experimental Setup

The authors established reproducible baselines using publicly available models:

• ASR Models:
  • Fine-tuned Models: The Fast Conformer-CTC Large model (120M parameters) from the NeMo toolkit was fine-tuned on various configurations: train-114 (BEA-Base), train-293 (BEA-Extension), and the combined train-114 + train-293 (BEA-Large). For BEA-Dialogue, Serialized Output Training (SOT) was employed, using a <sc> token to explicitly mark speaker transitions.
  • Zero-shot Models: Three Whisper variants (large-v3, large-v2, medium) were evaluated without fine-tuning.
• Diarization Models: Two state-of-the-art systems were evaluated on BEA-Dialogue without additional fine-tuning: pyannote.audio (hybrid neural approach) and Sortformer (transformer-based end-to-end architecture).
• Metrics: Performance was measured using Word Error Rate (WER), Character Error Rate (CER), and for dialogue, concatenated minimum-permutation WER (cpWER) and speaker change accuracy (scAcc). Diarization performance was measured via Diarization Error Rate (DER).

Key Contributions

1. BEA-Large Dataset: An extended corpus of 255 hours of spontaneous Hungarian speech from 433 speakers, enriched with granular metadata (age, gender, occupation, role) to support research in ASR and related areas.
2. BEA-Dialogue Dataset: A specialized corpus of 85 hours of natural dialogues, explicitly designed with speaker-independent splits for conversational ASR and speaker diarization research, addressing a critical shortage in Hungarian dialogue data.
3. Systematic Benchmarking: The provision of reproducible baselines using public models, establishing performance reference points for future research in Hungarian speech technology.

Results

• ASR Performance on BEA-Large:
  • The fine-tuned Fast Conformer model significantly outperformed zero-shot Whisper models across both repetitive and spontaneous subsets.
  • The best results were achieved using the combined training sets (train-114 + train-293), yielding a WER of 14.18% and a CER of 4.56% on the spontaneous evaluation subset.
  • Interestingly, including data from experiment leaders and discourse partners in training did not negatively impact performance on speaker-independent evaluation sets; in fact, it improved results compared to using only target speaker data.
• ASR Performance on BEA-Dialogue:
  • Fine-tuned models demonstrated the benefits of domain-specific adaptation, though the Whisper-large-v2 model surprisingly outperformed the fine-tuned model on the development set.
  • The ratio of CER to WER was notably higher than in non-conversational datasets, indicating frequent fine-grained, character-level deviations in spontaneous dialogue.
  • Speaker change accuracy (scAcc) for the fine-tuned model reached 82.16% on the evaluation set.
• Diarization Performance on BEA-Dialogue:
  • Baseline Diarization Error Rates (DER) ranged from 12.46% (Sortformer on dev) to 17.40% (Pyannote on eval), providing a reference point for future improvements.

Significance and Claims

The paper claims that these resources advance Hungarian speech technology by providing the necessary data and methodological frameworks to tackle the specific challenges of conversational ASR, such as disfluencies, overlaps, and informal speech patterns. By releasing these datasets alongside reproducible baselines, the authors aim to catalyze research in spontaneous Hungarian speech recognition.

Furthermore, the work offers a methodological blueprint for developing spontaneous and conversational benchmarks in other low-resource languages. The authors emphasize that while the results highlight the persistent difficulty of conversational ASR, the establishment of these baselines is a crucial step toward democratizing speech technology across linguistic boundaries. Future work is identified as focusing on advanced methods for overlap handling, speaker change detection, and context-aware modeling.