Technical Summary: CANDLE – Character-level Arabic Noise Deduplication using Lightweight Encoder

Problem Definition

The paper addresses the challenge of character-level noise deduplication in Arabic text, specifically targeting the removal of invalid consecutive character repetitions common in informal social media writing. While writers often repeat characters to express emphasis or emotion (e.g., elongating "sooooo" or Arabic equivalents), these repetitions can also be linguistically valid and essential to word meaning.

The core difficulty lies in distinguishing between noisy elongation (which should be reduced) and valid morphological repetition (which must be preserved). For instance, in Arabic, the word for "kingdom" (al-mamlaka) contains a valid double 'm', whereas "queen" (al-malika) does not. A naive reduction system that collapses all repetitions would incorrectly alter the meaning of valid words. Furthermore, while valid repetitions in Arabic rarely exceed two consecutive characters, the theoretical maximum is five, creating a need for a system that can handle context-dependent decisions without relying on rigid rules.

Methodology

The authors propose CANDLE, a lightweight system that frames character deduplication as a sequence alignment problem rather than a classification task.

1. Core Formulation: CTC Alignment

Unlike previous approaches that rely on handcrafted rules, dictionaries, or morphological analyzers, CANDLE utilizes Connectionist Temporal Classification (CTC).

• Input Transformation: The input text is first processed using a reduce_to_2() operation, which ensures every character appears exactly twice. This creates a consistent input structure where the sequence length is roughly double the original.
• Alignment Task: The model learns to align this doubled input sequence to the target "clean" sequence (where valid repetitions are kept and noisy ones are collapsed).
• Advantage: This formulation satisfies the CTC assumption that the input sequence length ($M$) is greater than the output sequence length ($N$), allowing the model to learn the alignment directly from data without explicit boundary labels.

2. Model Architecture and Training

• Base Architecture: The system is initialized using a pretrained character-based BERT architecture (6 transformer layers, $d_{model}=512$, 16 attention heads).
• Training Strategy:
  • Classification Baseline: A baseline approach was tested where the model classifies each character as "duplicate" or "don't duplicate" after a reduce_to_1() operation. This suffered from class imbalance and poor performance on ambiguous cases.
  • CTC Training: The CTC model was trained with a progressive unfreezing strategy. Initially, only the output layer was trained; subsequently, the last two, then three, and finally all transformer layers were unfrozen to ensure convergence on the harder alignment task.
• Model Distillation: To reduce inference overhead, a 6-layer "teacher" model was distilled into a 2-layer "student" model. The student was initialized using the first and last layers of the teacher and trained using a combination of soft targets (KL divergence from the teacher) and hard targets (ground truth CTC loss).

3. Datasets

The system was evaluated on three distinct benchmarks:

• NewsText: A clean corpus from the Sabq newspaper used to measure errors on formally written text (where the model should ideally make no changes).
• AmbigText: A manually curated dataset of 273 sentences containing ambiguous cases where reduced forms could map to multiple valid words (e.g., "grandfather" vs. "renewed").
• WildSAText: A real-world dataset of 142,280 sentences from Saudi forums, representing noisy, informal, and dialectal Arabic.

Key Results

The CTC-based approach significantly outperformed the classification-based baseline across all metrics and datasets.

• Performance on Clean Text (NewsText): The CTC model achieved a Sentence Error Rate (SER) of 5.37%, compared to 63.79% for the classification baseline. Crucially, both models achieved 0% error on sentences without duplicates, confirming they do not alter already-clean text.
• Performance on Ambiguous Text (AmbigText): The CTC model achieved an SER of 23.08%, vastly outperforming the classification baseline (69.96%). This demonstrates the alignment formulation's superior ability to leverage context to resolve ambiguity.
• Performance on Real-World Text (WildSAText): The CTC model achieved an overall SER of 9.93%, compared to 53.93% for the baseline.
• Distillation Efficiency: The 2-layer distilled model maintained high performance, with SERs of 6.31% (NewsText), 25.27% (AmbigText), and 12.43% (WildSAText). This represents a 3× reduction in model depth with minimal performance degradation.

Downstream Impact: Tokenizer Fertility

Beyond accuracy, the paper highlights a practical downstream benefit: reduced tokenizer fertility. By normalizing text before it enters Large Language Models (LLMs), the number of tokens required to represent the text decreases.

• Across diverse Arabic LLM tokenizers (including JAIS, Gemma, Llama, and Qwen), normalization reduced fertility scores by 4.8% to 12.8% relative to raw input.
• This reduction directly translates to lower inference costs and more efficient utilization of context windows.

Significance and Contributions

The paper claims the following contributions:

1. Novel Application of CTC: This is the first known application of Connectionist Temporal Classification to the specific task of character-level deduplication, framing normalization as a sequence alignment problem.
2. Resource Efficiency: The system operates without handcrafted rules, dictionaries, or morphological analyzers, relying solely on a lightweight encoder.
3. Practical Deployment: Through model distillation, the authors demonstrate that a high-performing model can be compressed into a 2-layer student, making it suitable for production environments.
4. Cost Reduction: The work establishes a direct link between text normalization and reduced LLM inference costs via improved tokenization efficiency.

Limitations

The authors acknowledge two primary limitations:

• Pure Arabic Input: The system assumes input contains only Arabic letters. It does not support diacritics, numerals, or special characters, which must be handled by a dedicated preprocessing stage to avoid distorting sentence structure.
• Bounded Repetition: The system assumes valid repetitions do not exceed two consecutive characters. Consequently, valid Arabic words with three or more consecutive identical characters (though rare) may be incorrectly collapsed.