HARNESS: Lightweight Distilled Arabic Speech Foundation Models

1. Problem Statement

Large Self-Supervised Learning (SSL) models (e.g., HuBERT, XLS-R) have revolutionized speech processing but face significant barriers in resource-constrained environments due to their massive size and computational costs.

• The Arabic Challenge: Arabic is a low-resource language in terms of high-quality, labeled data and is characterized by extreme dialectal diversity (MSA, Gulf, Levantine, Egyptian, Maghrebi, etc.) and code-switching. Existing multilingual models often fail to capture these specific linguistic and acoustic nuances, while monolingual models trained on high-resource languages (like English) underperform on Arabic tasks.
• The Deployment Gap: Training large, language-specific SSL models from scratch is computationally expensive. Furthermore, deploying these large models on edge devices or in low-latency scenarios is often infeasible. There is a lack of research on systematically compressing Arabic-centric SSL models into lightweight variants without sacrificing performance on critical downstream tasks like Automatic Speech Recognition (ASR), Dialect Identification (DID), and Speech Emotion Recognition (SER).

2. Methodology

The authors propose HArnESS (HuBERT-based Arabic and English Self-Supervised Speech), a model family trained from scratch using an iterative self-distillation framework.

A. Training Pipeline

The process follows a HuBERT-style masked prediction paradigm with three distinct phases:

1. Phase 1 & 2 (Teacher Training):

   • Data: A bilingual mixture of ~23,000 hours of Arabic and English speech (including QASR, MGB3, LibriSpeech, Common Voice, and YouTube crawls).
   • Architecture: A large teacher model (HArnESS-L) with 24 Transformer layers.
   • Process:
     • Iteration 1: Pseudo-labels are generated by clustering MFCC features ($K=1000$).
     • Iteration 2: Pseudo-labels are refined by clustering embeddings from the 9th layer of the Iteration 1 model.
     • The model is trained to predict these discrete pseudo-labels from masked audio frames.

2. Phase 3 (Distillation & Compression):

   • Data: A smaller, Arabic-only dataset (~1,100 hours from QASR).
   • Target: Distill knowledge from the Iteration 2 teacher into compact student models.
   • Compression Strategies:
     • Shallow (HArnESS-S): Reduces Transformer depth from 24 to 4 layers.
     • Shallow & Thin (HArnESS-ST): Reduces depth to 4 layers and embedding dimension from 1024 to 512.
     • PCA-based Supervision: To better match the capacity of thin students, the authors apply Principal Component Analysis (PCA) to the teacher's embeddings before clustering. This reduces the dimensionality of the supervision signal (from $D$ to $D'$), removing noise and creating a simpler target space for the student.

B. Downstream Evaluation Strategy

To isolate the quality of the learned representations, the authors employ a frozen encoder approach:

• The SSL encoder is frozen and used strictly as a feature extractor.
• Frame-level representations from all Transformer layers are averaged to form utterance-level vectors.
• Lightweight task-specific classifiers (for DID/SER) or Encoder-Decoder models (for ASR) are trained on top of these fixed features.

3. Key Contributions

1. HArnESS Model Family: Introduction of the first Arabic-centric SSL foundation models trained from scratch, comprising a large teacher (HArnESS-L) and two lightweight student variants (HArnESS-S and HArnESS-ST).
2. Iterative Self-Distillation Framework: A novel pipeline that progressively refines pseudo-labels and compresses the model architecture, effectively transferring knowledge from a large bilingual teacher to compact Arabic-specialized students.
3. PCA for Supervision Compression: An investigation into using PCA to compress the teacher's supervision signal prior to clustering. The study demonstrates that this simplifies the distillation target, leading to faster convergence and better performance for highly compressed students.
4. Comprehensive Benchmarking: Establishment of a new benchmark for Arabic SSL models across three diverse tasks: ASR (content), DID (dialect), and SER (paralinguistics), filling a gap in standardized Arabic evaluation.
5. Open Source Release: Public release of the distilled models and benchmark resources to facilitate future research.

4. Results

The models were evaluated on ASR (MGB2/MGB3), DID (ADI5), and SER (KSUEmotion).

• Performance vs. Baselines:

  • HArnESS-L significantly outperforms both English-only (HuBERT-L) and Multilingual (XLS-R) baselines across all Arabic tasks.
  • Example (ASR on MGB2): HArnESS-L achieves 15.50% WER, compared to 22.60% for both HuBERT-L and XLS-R.
  • Example (DID on ADI5): HArnESS-L achieves 84.98% accuracy, vastly outperforming XLS-R (42.35%) and HuBERT-L (64.14%).

• Efficiency vs. Performance Trade-off:

  • HArnESS-S (79.4% structural compression) maintains strong performance, outperforming multilingual baselines despite being much smaller.
  • HArnESS-ST (93.7% compression) shows a performance drop but remains competitive, particularly in ASR and SER.
  • DID Sensitivity: Dialect identification is the most sensitive to compression. While HArnESS-L achieves 84.98%, the highly compressed HArnESS-ST drops to 69.77%. This suggests dialectal cues require deeper or wider architectures to preserve.

• Ablation Findings:

  • PCA Supervision: Applying PCA to teacher embeddings before clustering improved training stability and efficiency for the smallest models.
  • Initialization: Weight initialization strategies (random vs. blocked-averaging) had minimal impact on final performance compared to the quality of the distillation signal.
  • Dimension Reduction: Aggressive reduction of embedding dimensions (e.g., to 256) caused sharp performance drops, indicating a limit to how thin the model can be made.

5. Significance

• Practical Deployment: HArnESS provides a practical solution for deploying high-performance speech AI in Arabic-speaking regions, particularly in scenarios with limited compute (mobile devices, edge servers).
• Linguistic Specificity: The work proves that language-specific pretraining, even with bilingual data, yields superior results for low-resource, high-dialectal-diversity languages compared to generic multilingual models.
• Methodological Insight: The paper highlights that suppression signal compression (via PCA) is a viable strategy to match the capacity of extremely small student models, offering a new direction for efficient model distillation beyond just architectural pruning.
• Community Resource: By releasing the models and benchmarks, the authors address the scarcity of standardized evaluation tools for Arabic speech processing, encouraging further research into dialect-aware and resource-efficient speech technologies.