arXiv💬 cs.CL

Statistical Machine Translation for Indic Languages

This paper presents the development and evaluation of Statistical Machine Translation (SMT) systems using the MOSES toolkit to translate between English and fifteen low-resource Indian languages, leveraging the Samanantar and OPUS datasets for training, Flores-200 for testing, and various preprocessing and reordering techniques to optimize translation quality as measured by BLEU, METEOR, and RIBES metrics.

Sudhansu Bala Das, Divyajoti Panda, Tapas Kumar Mishra, Bidyut Kr. Patra

Published 2026-03-04
📖 6 min read🧠 Deep dive
View on arXiv ↗PDF ↗

Imagine you are trying to have a conversation with a friend who speaks a completely different language. You want to share a story, but you don't speak their tongue, and they don't speak yours. You need a translator.

For a long time, computers tried to be that translator by following a giant, rigid rulebook written by human linguists. But languages are messy, full of exceptions and quirks, so these "rulebook" computers often sounded like robots reading a dictionary.

This paper is about a different approach: Statistical Machine Translation (SMT). Instead of memorizing rules, the computer acts like a super-observant student who learns by reading millions of examples of the same story told in two different languages. It looks for patterns and guesses, "Based on what I've seen a million times before, when this English word appears, it's usually followed by this Indian word."

Here is a simple breakdown of what the researchers did, using everyday analogies:

1. The Mission: Bridging the Gap

India is like a giant potluck dinner with 15 different regional dishes (languages) like Hindi, Bengali, Tamil, and Urdu. While everyone speaks English at the "head table," many people struggle to access global information because they can't read the menu in English.

The researchers wanted to build a translation machine that could speak English and 15 specific Indian languages (both ways: English to Indian, and Indian to English). They focused on "low-resource" languages, which are like rare ingredients—there aren't many cookbooks (data) available for them, making it hard to teach the computer.

2. The Ingredients: The Datasets

To teach the computer, you need "training data." Think of this as a massive library of parallel books: one page in English, the exact same page in the target language.

  • The Big Library: They used two huge online collections called Samanantar and OPUS. Imagine these as massive digital warehouses containing millions of sentence pairs.
  • The Test Kitchen: To see if the computer actually learned, they used a smaller, high-quality set of sentences called Flores-200. This is like a final exam where the teacher (the researchers) knows the correct answers.

3. The Prep Work: Cleaning the Data

Raw data from the internet is messy. It's like trying to cook a gourmet meal with ingredients that have dirt, rocks, and old labels on them.

  • The Cleaning Crew: Before training, the researchers had to scrub the data. They removed weird symbols, fixed broken punctuation, and made sure numbers looked the same in both languages.
  • The Tokenizer: They chopped sentences down into individual words (tokens), just like a chef chopping vegetables into uniform pieces so they cook evenly.

4. The Cooking Process: How the Model Learned

The researchers used a tool called MOSES, which is like a high-tech kitchen appliance designed specifically for translation. Here is how it worked:

  • The Alignment (Matching): The computer looked at an English sentence and an Indian sentence side-by-side. It tried to draw invisible strings connecting "Dog" to "Kutta" or "Run" to "Daorna." It did this millions of times to build a map of how words relate.
  • The Reordering (The Dance): This is the tricky part. English is like a dance where you say the Subject first, then the Verb, then the Object (e.g., "I eat apples"). But most Indian languages dance differently: Subject, Object, Verb (e.g., "I apples eat").
    • The computer had to learn to shuffle the words. They used a "Distance Reordering" technique. Imagine a line of people; if the person at the back needs to move to the front, it's a long, expensive move. The computer learned that moving words a little bit is cheap, but moving them across the whole sentence is expensive, so it tries to keep things in the most logical order.
  • The Fine-Tuning: After the initial cooking, they tasted the dish. If it was too salty (bad translation), they adjusted the spices (mathematical weights) and tried again, using the "Flores-200" test set to guide them.

5. The Taste Test: Did it Work?

How do you know if the translation is good? You can't just ask the computer; you need a scorecard. They used three different judges:

  • BLEU: Checks how many words match the human translation exactly.
  • METEOR: Looks for synonyms and meaning, not just exact word matches.
  • RIBES: Checks if the order of the words makes sense, even if the exact words are slightly different.

The Results:

  • The Stars: Languages like Hindi and Bengali performed very well. Why? Because there were huge, clean libraries of data for them. It's like having a million practice exams to study from.
  • The Strugglers: Languages like Sinhala and Tamil had lower scores.
    • Why? Even though they had lots of data, the data was "noisy" (bad translations). It's like having a million practice exams, but half of them have the wrong answers written in the back. The computer learned the wrong patterns.
    • Also, some languages are very complex (agglutinative), meaning they stick many small meaning-pieces together into one word, making them harder for the computer to break apart and translate.

6. The Conclusion

The researchers found that while this "statistical student" (SMT) is quite good at translating common languages, it still struggles with the messier, less common ones.

The Big Takeaway:
You can't just throw a huge pile of data at a computer and expect magic. Quality matters more than quantity. If the training data is full of errors, the computer learns those errors.

What's Next?
The researchers plan to:

  1. Clean the data even better (remove the "rocks" from the ingredients).
  2. Try mixing this statistical method with newer "Neural" methods (deep learning) to see if they can get the best of both worlds.
  3. Focus on the complex grammar of languages where words are built like Lego blocks, to help the computer understand how to take them apart and put them back together correctly.

In short, they built a bridge between English and 15 Indian languages. It's a sturdy bridge for the busy highways (Hindi, Bengali), but for the smaller, winding roads (Sinhala, Sindhi), they need to do some more roadwork to make the journey smoother.

More like this

Summarize Before You Speak with ARACH: A Training-Free Inference-Time Plug-In for Enhancing LLMs via Global Attention Reallocation

The paper introduces ARACH, a training-free inference-time plug-in that enhances large language models by aggregating context and reallocating attention through an adaptive context hub, thereby mitigating the attention sink phenomenon and improving performance across various tasks without requiring parameter updates or costly training.

Jingtao Wang, Yucong Wang, Jun Ding, Rui Cai, Xun Wang2026-03-13💬 cs.CL

DeReason: A Difficulty-Aware Curriculum Improves Decoupled SFT-then-RL Training for General Reasoning

The paper proposes DeReason, a difficulty-aware curriculum that optimizes post-training for general reasoning by strategically allocating non-reasoning-intensive data to supervised fine-tuning and challenging reasoning-intensive problems to reinforcement learning, thereby significantly outperforming baseline methods across STEM and mathematical benchmarks.

Hanxu Hu, Yuxuan Wang, Maggie Huan, Jannis Vamvas, Yinya Huang, Zhijiang Guo, Rico Sennrich2026-03-13💬 cs.CL

MDER-DR: Multi-Hop Question Answering with Entity-Centric Summaries

The paper introduces MDER-DR, a novel Retrieval-Augmented Generation framework that combines a Map-Disambiguate-Enrich-Reduce indexing strategy with a Decompose-Resolve retrieval mechanism to significantly improve multi-hop question answering on Knowledge Graphs by preserving contextual nuance and enabling robust reasoning without explicit graph traversal.

Riccardo Campi, Nicolò Oreste Pinciroli Vago, Mathyas Giudici, Marco Brambilla, Piero Fraternali2026-03-13💬 cs.CL

Markovian Generation Chains in Large Language Models

This paper defines the iterative inference process of large language models as Markovian generation chains and demonstrates through experiments and modeling that such processes can either increase or reduce sentence diversity depending on factors like temperature and initial input, with outcomes ranging from convergence to a recurrent set to the production of novel sentences.

Mingmeng Geng, Amr Mohamed, Guokan Shang, Michalis Vazirgiannis, Thierry Poibeau2026-03-13💬 cs.CL