💻 computer science

An Optimized Approach for Handling Limited Vocabulary Constraints and Word Order Divergence in English-Urdu Neural Machine Translation

This study improves English–Urdu neural machine translation by compiling a large-scale parallel corpus and demonstrating that optimizing subword tokenization strategies (specifically Subword-NMT BPE and SentencePiece-BPE) combined with Dynamic Position Encoding effectively addresses vocabulary sparsity and word-order divergence to outperform baseline models.

Original authors: Rabail Asghar, Junaid Akram, Yella Mehroze, Zaheer Ahmad Gondal

Published 2026-11-05 (v2)

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Rabail Asghar, Junaid Akram, Yella Mehroze, Zaheer Ahmad Gondal

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Language is the bridge that connects human experience, yet for billions of people, that bridge is broken by the sheer difficulty of translating between languages that think in fundamentally different ways. In the field of machine translation, computers have become remarkably good at converting text from one language to another, largely by learning from massive collections of existing translations. However, when the languages involved are very different in their structure or when one of them has fewer digital resources available, the computers often stumble. They struggle with words they have never seen before and, more critically, with the fact that languages arrange their sentences in different orders. For instance, English typically places the verb in the middle of a sentence, while Urdu places it at the end. This structural gap creates a specific kind of confusion for automated systems, leading to translations that are grammatically awkward or simply wrong.

A team of researchers at COMSATS University Islamabad and Huazhong University of Science and Technology has tackled these specific hurdles in the translation between English and Urdu. They approached the problem by building a massive, clean library of paired sentences in both languages and then testing different ways to teach the computer how to break down words into smaller, manageable pieces. Their work demonstrates that the way a computer learns to see a word matters just as much as the algorithm it uses to translate it. By carefully adjusting how the system handles word order and vocabulary, they were able to produce significantly smoother and more accurate translations than previous standard methods.

The researchers began by addressing a fundamental issue: the computer's dictionary. In machine translation, if a word appears in the text that the computer has never seen during its training, it cannot translate it. This is a common problem with names, new terms, or rare words. To solve this, the team compiled a vast collection of English and Urdu text from diverse sources, including religious texts, news articles, political speeches, and movie subtitles. They cleaned this data meticulously, removing errors and standardizing how numbers and punctuation appeared, resulting in a final dataset of nearly 5.8 million paired sentences. This is one of the largest and most refined collections of English-Urdu translations ever made available to the public.

With this data in hand, the team tested several different strategies for "tokenization," which is the process of chopping sentences into smaller units that the computer can understand. Instead of treating every word as a single, unbreakable block, modern systems often break words into sub-parts. For example, the word "running" might be split into "run" and "ning." The researchers tried five different methods for doing this, including techniques that learn from the text itself to find the most useful chunks. They found that for translating English into Urdu, a method called Byte-Pair Encoding worked best, allowing the system to handle rare words effectively. Conversely, when translating Urdu into English, a slightly different version of this technique performed better. This discovery highlights that the best way to teach a computer to read one language is not always the same as the best way to teach it to read another.

The second major challenge was the difference in sentence structure. Because English and Urdu arrange their words in different orders, a computer trained to follow English patterns often produces Urdu sentences that sound unnatural, placing the verb in the wrong spot. To fix this, the researchers used a specialized type of computer model that can learn to rearrange the internal logic of a sentence without actually shuffling the words around in the input. They added a specific training step where the model was taught to pay attention to how words in English correspond to words in Urdu, effectively learning the "rules" of the language switch. By fine-tuning the balance between learning the translation itself and learning the correct word order, they found a specific setting that worked perfectly for this language pair.

The results of these adjustments were clear and measurable. The new system, which combined the best word-splitting method with the order-aware model, significantly outperformed the standard baseline models. In the English-to-Urdu direction, the translation quality improved by more than three points on a standard scoring scale, and in the Urdu-to-English direction, it improved by nearly two and a half points. When the researchers looked at specific examples, the difference was visible: the old models often left time phrases like "after several rounds of discussion" at the end of the Urdu sentence, mimicking English grammar. The new model correctly moved these phrases to the beginning, creating sentences that flowed naturally for a native Urdu speaker.

This work does not claim to have solved machine translation entirely, but it offers a concrete path forward for languages that are often left behind. The researchers showed that by paying close attention to how words are broken down and by explicitly teaching the system to respect the unique grammar of the target language, translation quality can be greatly enhanced. Their findings suggest that for linguistically distant pairs like English and Urdu, a one-size-fits-all approach is insufficient. Instead, tailored strategies that address both vocabulary limitations and structural differences are necessary to build systems that truly bridge the gap between speakers of different tongues. The large dataset they created and the methods they tested are now available for other researchers to use, ensuring that this progress can be built upon by the wider scientific community.

Technical Summary: An Optimized Approach for Handling Limited Vocabulary Constraints and Word Order Divergence in English-Urdu Neural Machine Translation

Problem Statement

Neural Machine Translation (NMT) has largely superseded statistical and rule-based approaches, yet low-resource and morphologically rich language pairs like English–Urdu continue to face significant hurdles. This study identifies two primary challenges:

Vocabulary Sparsity: The presence of Out-of-Vocabulary (OOV) words, including named entities, domain-specific terms, and neologisms, which degrades translation fluency and adequacy.
Syntactic Word-Order Divergence: A fundamental structural mismatch between English (Subject–Verb–Object, SVO) and Urdu (Subject–Object–Verb, SOV). This divergence complicates the alignment of source and target sequences, often leading to structural errors in the generated output.

While previous research has addressed these issues through multilingual transfer learning or architectural modifications, there remains a gap in the systematic comparative evaluation of subword tokenization strategies specifically for this language pair, alongside the explicit modeling of word-order divergence within a Transformer framework.

Methodology

The authors propose a comprehensive framework involving data curation, subword tokenization evaluation, and a reordering-aware Transformer architecture.

1. Dataset Collection and Preprocessing

To ensure reproducibility and support large-scale training, the authors compiled a massive parallel corpus from diverse sources, including religious texts (Quran, Bible), news archives (PIB, The Wire), subtitles, and legal documents.

Preprocessing Pipeline: The data underwent rigorous cleaning, including the removal of symbolic content (hashtags, URLs), whitespace normalization, expansion of English contractions, and standardization of Urdu numerals.
Filtering: Advanced filters were applied to remove inconsistent sentence pairs, duplicates, and misalignments. A specific non-zero numeral filter was implemented to ensure consistent formatting of quantities and dates.
Scale: The final cleaned dataset comprises 5,775,799 parallel sentence pairs, making it the largest publicly available English–Urdu dataset to date.

2. Subword Tokenization Strategies

To mitigate vocabulary sparsity, five distinct tokenization algorithms were trained and evaluated on the preprocessed corpora with a fixed vocabulary size of 30,000 tokens:

Subword-NMT BPE: Byte-Pair Encoding trained via the Subword-nmt library.
SentencePiece Variants: Including BPE, Word-based, Unigram, and Character-based tokenizers.
The goal was to identify which segmentation strategy best balances the decomposition of rare words with the preservation of semantic meaning for both translation directions (English-to-Urdu and Urdu-to-English).

3. Model Architecture: Dynamic Position Encoding (DPE)

The study utilizes a Transformer variant incorporating Dynamic Position Encoding (DPE) to address syntactic divergence without explicitly reordering the input sequence.

Mechanism: Unlike standard Transformers that use fixed sinusoidal positional encodings, the DPE model learns to refine source positional embeddings during training.
Reordering Loss: An auxiliary objective is introduced using word alignments generated by FastAlign. The model minimizes the Mean Squared Error (MSE) between the refined source positional embedding and the target positional embedding derived from the aligned target word.
Optimization: The total loss function is a weighted sum of the translation loss and the reordering loss: $L_{total} = \lambda L_{translation} + (1 - \lambda)L_{reorder}$ . The hyperparameter $\lambda$ is tuned to find the optimal balance between translation accuracy and positional refinement.

Key Contributions

Public Dataset Release: The compilation and public release of a high-quality, standardized English–Urdu parallel corpus (5.77M sentences) on Mendeley Data to facilitate reproducible research.
Tokenization Analysis: A systematic evaluation of multiple subword tokenization strategies, identifying that different tokenizers perform optimally depending on the translation direction.
Architecture Optimization: The application and tuning of a DPE-based Transformer architecture specifically for the English–Urdu pair, demonstrating the efficacy of auxiliary reordering objectives in handling SVO-SOV divergence.
Performance Benchmarking: A comprehensive performance analysis comparing the optimized model against standard Transformer baselines and existing literature.

Experimental Results

The experiments were conducted using the Fairseq toolkit on an NVIDIA Tesla T4 GPU.

Tokenization Performance

English-to-Urdu: Subword-NMT BPE achieved the highest BLEU score (30.30). The authors attribute this to its ability to preserve semantically meaningful subword units during target-language generation.
Urdu-to-English: SentencePiece-BPE yielded the best results (26.90 BLEU). This suggests that for this direction, corpus-driven segmentation that encodes word-boundary information provides richer source-side representations.
General Trend: BPE-based approaches consistently outperformed word-level, unigram, and character-level methods.

Hyperparameter Tuning ( $\lambda$ )

Fine-tuning the reordering loss weight $\lambda$ significantly improved performance:

English-to-Urdu: The optimal $\lambda$ was 0.5, resulting in a BLEU score of 31.95.
Urdu-to-English: The optimal $\lambda$ was 0.7, resulting in a BLEU score of 27.90.
Baseline Comparison: The optimized DPE Transformer outperformed the standard Transformer Base baseline by +3.17 BLEU points (EN-UR) and +2.40 BLEU points (UR-EN).

Qualitative Analysis

Qualitative examples demonstrated that the optimized model successfully corrected word-order errors common in the baseline. For instance, in English-to-Urdu translation, the model correctly moved temporal phrases (e.g., "after several rounds of discussion") from the sentence-final position (English style) to the pre-verbal slot (Urdu SOV style), resulting in more fluent and syntactically accurate outputs.

Significance and Claims

The paper claims that the proposed framework effectively addresses the dual challenges of vocabulary sparsity and syntactic divergence in English–Urdu machine translation. The authors emphasize that:

Targeted Adaptation is Key: Simply applying standard Transformer models is insufficient; specific adaptations in tokenization and positional encoding are necessary for linguistically distant pairs.
Data Quality Matters: The creation of a large, clean, and publicly available dataset is a foundational step for advancing research in low-resource language pairs.
Complementary Strategies: The study demonstrates that combining effective subword tokenization with reordering-aware modeling yields meaningful improvements in translation quality.

The authors maintain a modest stance, acknowledging that while automatic metrics (BLEU, METEOR, TER, NIST) show improvement, they may not fully capture all aspects of human-perceived translation quality. They note that future work could incorporate human evaluation and significance testing to further validate the findings. The study does not claim to solve all challenges in English–Urdu translation but provides a robust, reproducible baseline that significantly outperforms existing Transformer-based approaches on comparable datasets.