BanglaFake: Constructing and Evaluating a Specialized Bengali Deepfake Audio Dataset

Technical Summary: BanglaFake – Constructing and Evaluating a Specialized Bengali Deepfake Audio Dataset

Problem Statement
Deepfake audio generation poses a significant threat to speech processing and biometric verification systems, particularly Automatic Speaker Verification (ASV). While detection mechanisms have advanced for high-resource languages like English and Chinese, low-resource languages such as Bengali suffer from a critical lack of large-scale, high-quality datasets. This scarcity hinders the development of robust detection models capable of identifying subtle acoustic artifacts in synthetic speech. Existing benchmarks primarily focus on high-resource languages, leaving a gap in research for underrepresented linguistic communities.

Methodology
To address this gap, the authors constructed BanglaFake, a specialized dataset comprising both real and synthetic Bengali utterances. The methodology involved three primary stages:

1. Data Sourcing and Composition:

   • Real Audio: The dataset includes 12,260 real utterances sourced from the SUST TTS Corpus (a phonetically balanced Bengali speech dataset) and Mozilla Common Voice. These samples feature seven distinct speakers (male and female).
   • Deepfake Audio: 13,260 synthetic utterances were generated using a state-of-the-art, end-to-end Text-to-Speech (TTS) model based on the VITS (Variational Inference with adversarial learning for end-to-end Text-to-Speech) architecture. This model was trained from scratch specifically on the SUST TTS Corpus.
   • Voice Conversion Scenarios: To simulate voice conversion attacks, the authors utilized 2,260 samples from Mozilla Common Voice (five speakers) where the original recordings served as "bonafide" audio and the VITS-generated outputs served as the "deepfake" counterparts.
   • Format: All data was standardized into the LJ Speech format (WAV, 22,050 Hz, 6–7 seconds duration) to ensure compatibility with existing TTS and detection frameworks.

2. Model Training and Generation:
   The VITS model employs a combination of variational inference and adversarial learning. The process involves:

   • Converting text to phonemes and aligning them with latent variables using Monotonic Alignment Search (MAS).
   • Utilizing a posterior encoder to extract latent variables from target waveforms and a prior encoder to refine their distribution.
   • Employing a HiFi-GAN-based decoder to up-sample variables into raw waveforms.
   • Using a stochastic duration predictor with flow-based generative approaches to model phoneme durations.
   • The trained model was then used to synthesize new speech from text, creating the deepfake corpus.

3. Evaluation:
   The dataset was evaluated through both qualitative and quantitative analyses:

   • Qualitative (MOS): A Mean Opinion Score (MOS) test was conducted with 30 native Bengali speakers (aged 20–25). Participants rated 10 randomly selected synthesized sentences (outside the training corpus) on naturalness and intelligibility. Scores were calculated using a "Robust-MOS" approach, excluding the highest and lowest ratings.
   • Quantitative (Feature Analysis): The separability of real and deepfake audio was analyzed using t-SNE visualization of Mel-Frequency Cepstral Coefficients (MFCCs).

Key Results

• Dataset Scale: The final BanglaFake dataset contains 12,260 real and 13,260 deepfake utterances, totaling over 25,000 samples.
• Perceptual Quality: The Robust-MOS results indicated a naturalness score of 3.40 and an intelligibility score of 4.01. These scores suggest the synthetic audio achieves acceptable quality and human-like characteristics, making it a challenging testbed for detection models.
• Detection Difficulty: The t-SNE visualization of MFCC features revealed significant overlap between the clusters of real and deepfake audio samples. This overlap confirms that the synthetic speech generated by the VITS model possesses high naturalness, thereby presenting a genuine challenge for distinguishing between authentic and manipulated audio.

Significance and Contributions
The paper claims the following contributions to the field of audio forensics and low-resource language research:

1. First Public Bengali Deepfake Dataset: BanglaFake is introduced as the first publicly available dataset dedicated to Bengali deepfake audio detection, addressing the scarcity of resources for this low-resource language.
2. High-Quality Synthetic Benchmark: By utilizing a VITS model trained from scratch on a phonetically balanced corpus, the authors created a dataset where the deepfake samples are of high naturalness, moving beyond simple voice conversion to full speech synthesis.
3. Foundation for Detection Research: The dataset serves as a benchmark for training and evaluating deepfake detection models, specifically aiming to enhance safeguards against AI-generated media misuse in Bengali.
4. Open Availability: The dataset and all relevant artifacts are publicly released on Hugging Face and GitHub under an open license to facilitate further research.

The authors conclude that while the current dataset focuses on a single male speaker (for the synthetic generation) and specific sources, it establishes a necessary foundation for future work. They note that expanding the dataset to include multiple speakers (male and female) and complex phonemes remains a direction for future development to further improve diversity and robustness.