Thaka at KSAA-2026 Task 2: Regularized Fine-Tuning for Arabic Speech Diacritization

Technical Summary: Thaka at KSAA-2026 Task 2: Regularized Fine-Tuning for Arabic Speech Diacritization

Problem Definition
The paper addresses Task 2 of the KSAA-2026 Shared Task, which focuses on Arabic Speech Dictation with Automatic Diacritization. The objective is to generate fully diacritized Arabic text from speech audio and undiacritized transcripts. This task is particularly challenging due to the inherent ambiguity of Arabic orthography, where short vowels and phonological markers (diacritics) are typically omitted, leading to multiple potential meanings for a single string of characters. While speech signals provide prosodic and phonetic cues to resolve these ambiguities, the shared task presents a low-resource scenario: only 2,327 training samples are available across multiple Arabic dialects, and the use of external data is strictly prohibited.

Methodology
The authors' system is built upon the CATT-Whisper architecture, a multimodal model that fuses a character-level CATT text encoder with a Whisper speech encoder. The technical approach emphasizes rigorous training regularization and inference-time ensembling rather than complex architectural changes.

• Architecture:

  • Text Encoder: A 6-layer CATT Transformer (512 dimensions, 16 heads) pretrained on Arabic diacritization, predicting one of 15 diacritic classes per letter.
  • Speech Encoder: A frozen Whisper-base encoder (6 blocks, 512 dimensions).
  • Fusion: The system employs prefix addition. 1,500 Whisper speech frames are mean-pooled into 150 tokens, projected, and inserted as 150 dedicated prefix tokens preceding the text input. The model contains approximately 39M parameters, with roughly 19M trainable.

• Training Strategy:

  • Regularization: The core innovation lies in the training recipe. The system utilizes R-Drop consistency regularization, where each input passes through the model twice with different dropout masks, penalizing divergent predictions via a symmetric KL divergence penalty ($\alpha = 2.08$).
  • Loss Function: The supervised loss combines Focal Loss ($\gamma = 0.34$) with label smoothing ($\epsilon = 0.018$) to handle class imbalance and hard examples.
  • Hyperparameter Optimization: All hyperparameters (learning rate, weight decay, dropout rates, etc.) were selected via Optuna over 30 trials. Notably, the system employs high weight decay (0.098) and specific dropout rates for speech embeddings.
  • Data Augmentation: Techniques include SpecAugment (frequency and time masking) and Gaussian noise injection.
  • Checkpoints: Four distinct model checkpoints were trained: three using the primary configuration with different random seeds (42, 7, 123) and a fourth using an alternative configuration (different learning rate, batch size, and partially unfrozen Whisper blocks) to increase ensemble diversity.

• Inference:

  • The system employs Monte Carlo (MC) Dropout ensembling. During inference, dropout remains active in the CATT encoder.
  • Each of the four checkpoints performs 50 stochastic forward passes (totaling 200 passes).
  • The final prediction is derived by averaging the softmax probabilities across all 200 passes before taking the argmax.
  • Post-Processing: A direct positional insertion method ensures that the output maintains a 1:1 correspondence with input characters, enforcing invariants such as recovering the original input upon stripping diacritics.

Key Contributions

1. Winning System: The proposed system achieved 1st place in the KSAA-2026 Shared Task.
2. Regularization over Architecture: The authors demonstrate that in low-resource settings (2,187 effective training samples), optimization strategies (R-Drop, Focal Loss, high weight decay) yield significantly larger performance gains than architectural modifications (e.g., cross-attention, CRF decoding, or RL fine-tuning).
3. Inference Ensembling: The application of MC Dropout ensembling at inference time provided a measurable improvement, though the authors note the computational cost (200 forward passes).

Results
The system was evaluated on the test set using Word Error Rate (WER) with case endings (including no-diacritic positions) as the primary metric.

• Performance: The system achieved a 23.26% WER, outperforming the second-place participant (24.39%) and the text-only baseline (49.85%).
• Ablation Study: A cumulative ablation on the development set revealed the source of improvements:
  • Fine-tuning the pretrained CATT-Whisper with a standard recipe yielded 30.43% WER.
  • Adding the regularized recipe (R-Drop, Focal Loss, high weight decay) reduced WER to 27.18% (a 3.25 percentage point gain).
  • Adding MC Dropout ensembling further reduced WER to 26.02% (an additional 1.16 percentage point gain).
  • The authors conclude that the training recipe was the primary driver of improvement.

Significance and Claims
The paper claims that for the specific constraints of the KSAA-2026 Task 2 (limited data, no external resources), the most effective path to high performance is not architectural complexity but rather robust training regularization. The authors assert that their approach successfully leverages the complementary nature of speech signals to resolve orthographic ambiguities, confirming that audio input is critical for this task. They modestly suggest that future work should focus on disentangling the individual contributions of each regularization component and analyzing per-dialect performance to identify remaining challenges, rather than claiming universal superiority across all Arabic diacritization scenarios.