Scaling Conversational Hungarian ASR: The BEA-Dialogue+ Corpus

Technical Summary: Scaling Conversational Hungarian ASR with the BEA-Dialogue+ Corpus

Problem Statement
Conversational Automatic Speech Recognition (ASR) for Hungarian is currently constrained by a scarcity of publicly available, dialogue-style training data. While the existing BEA-Dialogue corpus addresses the need for conversational data, its strict design criterion—requiring complete speaker disjointness across training, validation, and evaluation splits—severely limited the usable dataset size to only 85 hours. This reduction is particularly disadvantageous for modern neural-network-based ASR systems, which rely heavily on both the volume and diversity of training data to achieve high performance. The authors posit that a trade-off exists between maintaining strict data separation to prevent leakage and expanding the dataset to improve model training, a balance that had not been systematically explored for Hungarian dialogue ASR.

Methodology
To address this limitation, the authors introduced BEA-Dialogue+, an expanded version of the corpus derived from the original BEA database recordings. The core methodological shift involves relaxing the speaker-disjointness constraint:

• Split Criteria: While the original BEA-Dialogue ensured no speakers appeared in multiple splits, BEA-Dialogue+ maintains strict separation only for primary speakers (the main informants). It allows experimenters and dialogue partners to appear across different splits (training, dev, eval), provided the primary speaker of a recording does not overlap.
• Data Processing: The corpus consists of natural conversations segmented into approximately 30-second units. The authors utilized Serialized Output Training (SOT) for model training, where speaker changes are explicitly marked with a <sc> token. This allows the model to recognize speaker boundaries while preserving the linguistic integrity of utterances, even during overlaps.
• Experimental Setup: The study benchmarked several models, including Whisper (medium, large-v2, large-v3) and FastConformer architectures. Models were evaluated in two configurations: zero-shot (pre-trained only) and fine-tuned (ft) on the Hungarian data. Evaluation metrics included Word Error Rate (WER), Character Error Rate (CER), and their dialogue-specific variants (cpWER and cpCER), which minimize errors over permutations of speaker-delimited units. Speaker-change accuracy (scAcc) was also measured for fine-tuned models.

Key Contributions

1. BEA-Dialogue+ Corpus: The release of a 200-hour conversational Hungarian speech corpus (a ~2.5x increase over the original 85 hours). This dataset preserves the separation of primary speakers while significantly increasing the volume of usable training material.
2. Quantitative Analysis of Split Relaxation: A detailed comparison between BEA-Dialogue and BEA-Dialogue+, demonstrating that relaxing the split constraints increases the complexity of the task. The new corpus contains fewer segments with zero or one speaker change and a higher frequency of overlapping speech, making it a more challenging benchmark.
3. Unified Benchmarking: A comprehensive evaluation of Whisper and FastConformer models on both corpus versions under a unified protocol, specifically analyzing the impact of SOT-based fine-tuning for dialogue transcription.
4. Trade-off Investigation: An analysis of the trade-off between increased data volume and the risk of speaker overlap (data leakage), providing empirical evidence on how these factors influence model performance.

Results

• Zero-Shot Performance: Models without fine-tuning performed significantly worse on BEA-Dialogue+ compared to the original BEA-Dialogue, showing a relative degradation of at least 10% in error rates. The authors attribute this to the increased complexity of the new corpus, specifically the higher frequency of speaker changes and overlapping speech, which challenges models lacking explicit boundary information.
• Fine-Tuned Performance: Fine-tuning yielded consistent improvements across all metrics (WER, CER, cpWER, cpCER). Notably, FastConformer models pre-trained on Hungarian data and then fine-tuned achieved the best results, outperforming even Whisper models.
• Impact of Overlap: While the expanded corpus introduced speaker overlap (potential data leakage), the results suggest that the performance gains in fine-tuned models are not solely due to this leakage. The models showed particular improvement on complex segments with multiple speaker changes, indicating that the increased data volume allowed for better adaptation to dialogue structures.
• Metric Correlation: The study observed that cpWER and cpCER were consistently lower than standard WER/CER, with the gap widening for fine-tuned models. Furthermore, scAcc (speaker-change accuracy) correlated with task difficulty; as the number of speaker changes increased, scAcc decreased, reflecting the inherent difficulty of the task rather than just model failure.

Significance
The paper claims that BEA-Dialogue+ provides a substantially larger yet still demanding benchmark for Hungarian dialogue ASR. It serves as a practical resource for training and evaluating dialogue transcription systems, demonstrating that relaxing strict speaker-disjointness constraints can yield significant data volume benefits without rendering the benchmark trivial. The authors conclude that while the expanded corpus presents a more difficult task for pre-trained models, it enables more effective training for fine-tuned systems, highlighting the value of dialogue-specific continued training. The corpus is intended to support future research in Conversational AI and Hungarian ASR, with the caveat that the precise extent of data leakage effects requires further investigation through independent evaluation sets.