💬 NLP

Constrained CTC Decoding for Efficient Diacritic Restoration

This paper proposes an efficient non-autoregressive CTC-based method for Arabic speech diacritic restoration that utilizes hard constraints on a character-level lattice to achieve statistically significant error rate reductions compared to more complex multi-modal baselines.

Original authors: Rufael Marew, Amr Keleg, Hanan Aldarmaki

Published 2026-07-22

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Rufael Marew, Amr Keleg, Hanan Aldarmaki

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to read a secret message written in a language where the vowels are invisible. In the Arabic script, the letters are like the skeleton of a word, but the short vowels, stress marks, and other tiny accents (called diacritics) are often left out in everyday writing. Without these hidden clues, a single string of letters can sound like three completely different words, changing the meaning from "the girl drank" to "the girl was made to drink." This is a huge headache for computers trying to turn spoken Arabic into text. While computers are getting better at hearing speech, they often stumble when trying to guess these missing accents, especially because there isn't enough training data where every single accent is written down. Scientists have been trying to fix this by combining what a computer hears with what it reads, but the old ways of doing this are like trying to solve a puzzle while wearing heavy, clunky armor—they work, but they are slow and complicated.

This paper introduces a clever, lightweight trick to help computers restore those missing accents in Arabic speech transcripts. The researchers propose a method called "Constrained CTC Decoding." Think of the computer's standard way of listening as a wild explorer who might accidentally swap a letter, delete a word, or invent a new one while trying to write down what was said. The new method acts like a strict but helpful guide. It takes the computer's best guess at the basic letters (the skeleton) and builds a "lattice," or a map, that says, "You must keep these letters exactly where they are, but you are free to choose the correct accent for each one." Instead of asking the computer to relearn how to speak or read from scratch, this method simply restricts the computer's choices to only valid combinations of the known letters and possible accents.

The team tested this idea on two different types of Arabic speech: Classical Arabic (like the language of ancient religious texts) and Modern Standard Arabic (used in news and formal settings). They compared their new "guide" method against a more complex, heavy-duty system that tries to fuse speech and text data in a multi-step process. The results were promising. The new method didn't just match the performance of the heavy-duty system; it actually made fewer mistakes in restoring the accents, particularly when the computer had to handle speech it hadn't seen before. In fact, the improvement was statistically significant, meaning it wasn't just a lucky fluke. The paper suggests that by forcing the computer to stick to the known letters and only guessing the accents, the system becomes both faster and more accurate. It's a streamlined approach that proves you don't need a massive, complex machine to solve a tricky puzzle; sometimes, you just need to know which paths are off-limits.

Technical Summary: Constrained CTC Decoding for Efficient Diacritic Restoration

Problem Statement
Arabic script is an abjad system where short vowels and other phonological marks (diacritics) are typically omitted in standard text, leading to under-specified input for speech applications. This omission causes ambiguity in downstream tasks such as Text-to-Speech (TTS) and assistive reading, particularly for homographs and morphologically complex forms. While automatic speech recognition (ASR) models can generate transcripts, they typically lack the capability to produce fully diacritized outputs due to the scarcity of large-scale, fully diacritized speech data. Existing approaches to speech-to-text diacritization often rely on complex multi-modal frameworks that combine acoustic models with separate text-based decoders. These methods, while effective in specific contexts like Classical Arabic, often suffer from poor generalization to Modern Standard Arabic (MSA) and dialects, and they introduce computational overhead through multi-modal training and text fusion.

Methodology
The authors propose an efficient, non-autoregressive approach for speech-to-text diacritization that leverages the Connectionist Temporal Classification (CTC) decoding mechanism directly, eliminating the need for additional text-based modules or multi-modal training.

Acoustic Model Training: The system utilizes a standard CTC-based ASR model (specifically Wav2vec2-XLSR) trained on speech data with fully diacritized transcripts. The model learns to predict a sequence containing both base alphabetic characters and their corresponding diacritics.
Constrained Decoding Lattice: The core innovation lies in the inference phase. Instead of allowing the decoder to generate arbitrary character sequences, the authors construct a character-level diacritization lattice ( $G_{char}(u)$ $G_{c ha r} (u)$ ) based on the input undiacritized transcript ( $u$ $u$ ).
- Structure: This lattice is a linear-chain Weighted Finite-State Transducer (WFST). It enforces a "hard constraint" where the base alphabetic characters must appear in the output in the exact order and positions as the input transcript.
- Wildcard States: Between each base character, the lattice inserts "wildcard" states. These states allow the model to predict only the valid diacritic tokens (or a blank token representing no diacritic) associated with that specific character position.
- Forced Alignment: This creates a partially forced-alignment scenario. The decoding graph forces the base letters to remain unchanged, restricting the model's hypothesis space to only valid diacritic insertions. This is implemented by composing the standard CTC decoding graph with the constraint lattice or by restricting beam search expansions to symbols permitted by the lattice's outgoing arcs.

Key Contributions

Efficient Architecture: The paper introduces a streamlined approach that integrates diacritic restoration directly into the CTC decoding process. Unlike prior multi-modal methods, it does not require a separate text encoder, cross-attention mechanisms, or the fusion of ASR hypotheses with text representations.
Hard Constraints via Lattice: The method proposes constructing a specific WFST lattice that restricts decoding hypotheses to valid diacritized realizations of a given undiacritized transcript, ensuring the base character skeleton remains intact.
Generalization: The approach is designed to generalize better across different Arabic varieties (Classical and Modern Standard) compared to complex multi-modal baselines.

Experimental Results
The method was evaluated on two datasets: ClArTTS (Classical Arabic) and ArVoice (Modern Standard Arabic). The proposed model was compared against two baselines: a Text-only model (Transformer-based sequence labeling) and a Text+ASR multi-modal baseline (which fuses text and ASR hypotheses via cross-attention).

Performance: The proposed method achieved statistically significant reductions in Diacritic Error Rate (DER) compared to the Text+ASR baseline on both test sets.
- In the matched setting (trained and tested on the same dataset), the proposed method achieved performance comparable to the Text+ASR baseline.
- In cross-dataset settings (e.g., trained on ClArTTS, tested on ArVoice), the Text+ASR baseline degraded sharply, whereas the proposed method demonstrated superior robustness and generalization.
Combined Training: When trained on a combination of both datasets, the proposed method achieved the best performance on both test sets, outperforming the baselines.
Statistical Significance: Non-parametric bootstrap resampling confirmed that the improvements in DER were statistically significant (95% confidence intervals lay entirely below zero).
Efficiency: The proposed approach is noted for being more computationally efficient and streamlined, as it avoids the complexity of multi-modal fusion.

Significance and Claims
The paper claims that this constrained CTC decoding approach offers a dual advantage of performance and efficiency. By leveraging the ASR model directly as a constrained diacritic restoration tool, the method avoids the data scarcity issues and architectural complexity of multi-modal systems. The authors assert that this approach is particularly valuable for Arabic speech data curation, as the majority of existing speech datasets contain undiacritized transcripts. The model can serve as a robust method to generate fully diacritized transcripts for these datasets, thereby enabling more accurate phonological modeling and improving downstream applications like TTS and assistive reading without requiring extensive retraining or additional text-only corpora.

Technical Summary: Constrained CTC Decoding for Efficient Diacritic Restoration

More like this