💬 NLP

Assessing and Improving Punctuation Robustness in English-Marathi Machine Translation

This paper introduces "Viram," a diagnostic benchmark for punctuation-ambiguous English-Marathi sentences, and demonstrates that both cascade-based punctuation restoration and direct fine-tuning significantly improve translation robustness compared to current Large Language Models.

Original authors: Kaustubh Shivshankar Shejole, Sourabh Deoghare, Pushpak Bhattacharyya

Published 2026-02-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kaustubh Shivshankar Shejole, Sourabh Deoghare, Pushpak Bhattacharyya

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to translate a recipe from English to Marathi for a friend. If the recipe says, "Add salt pepper and sugar," your friend might mix everything together in a bowl and ruin the dish. But if the recipe says, "Add salt, pepper, and sugar," the instructions are clear, and the meal turns out perfect.

This paper is about how computer translation systems (like Google Translate) often get confused when sentences are missing their "commas, periods, and semicolons." The authors, researchers from IIT Bombay, wanted to fix this problem specifically for translating English into Marathi.

Here is a breakdown of their work using simple analogies:

1. The Problem: The "Run-On" Sentence Disaster

Neural Machine Translation (NMT) systems are like brilliant students who have read millions of books, but they rely heavily on punctuation to understand who is doing what to whom.

When users type sentences without punctuation (which happens a lot in text messages or quick notes), the computer gets lost.

The Analogy: Imagine a fire extinguisher instruction that says: "Pull pin aim at fire."
- With punctuation: "Pull pin, aim at fire." (Do this, then do that).
- Without punctuation: "Pull pin aim at fire." The computer might think "Pin" is the name of the fire, or that you should aim the pin at the fire.
The Result: The translation might be grammatically smooth but semantically disastrous (e.g., telling a user to eat their grandmother instead of inviting her to dinner).

2. The Solution: Building a "Training Gym" (The Viram Benchmark)

To test how bad this problem is, the researchers created a special test set called Viram (which means "punctuation" in Marathi).

The Analogy: Think of this as a "gym" for translation models. They created 54 specific sentences that are tricky puzzles. Each sentence has two versions:
1. The Written Version: The messy, punctuation-free sentence (the "trap").
2. The Meant Version: The clear, punctuated version with the correct meaning (the "goal").
They used this gym to see if current AI models could figure out the correct meaning even when the clues (punctuation) were missing.

3. The Two Strategies: How to Fix the Models

The researchers tried two different ways to teach the AI to handle messy sentences.

Strategy A: The "Editor First" Approach (Restore then Translate)

How it works: Before translating, they use a separate AI tool to act like a strict editor. This tool looks at the messy sentence, adds the missing commas and periods, and then passes the clean sentence to the translator.
The Analogy: It's like hiring a proofreader to fix a student's essay before a teacher grades it.
Result: This worked very well. By cleaning up the sentence first, the translator could understand the meaning perfectly.

Strategy B: The "Hardy Gymnast" Approach (Direct Fine-Tuning)

How it works: Instead of hiring an editor, they took the main translator and trained it on a mix of clean sentences and messy, punctuation-free sentences. They forced the model to learn how to guess the meaning even when the clues were missing.
The Analogy: This is like training a gymnast to walk a tightrope in the rain. They don't rely on the rope being dry (punctuation); they learn to balance using their own internal sense of balance (context).
Result: This also worked great. The model learned to "read between the lines" and didn't panic when punctuation was missing.

4. The Surprise: Big AI Models (LLMs) Struggled

The researchers also tested huge, famous AI models (like Llama and Gemma) to see if they could solve this puzzle just by being asked nicely (using "prompts").

The Finding: Surprisingly, these massive, general-purpose models performed worse than the specialized, smaller models the researchers built.
The Analogy: It's like asking a world-famous chef (the Big LLM) to cook a specific local dish without a recipe. They might make something tasty, but a local chef (the specialized model) who has practiced this specific dish a thousand times will do a much better job. The big models were too "general" and didn't have the specific training to handle these punctuation traps.

5. The Trade-off: Perfection vs. Robustness

One interesting discovery was that making the model "punctuation-robust" (good at handling messy text) sometimes made it slightly worse at standard tests that use perfect, clean text.

The Analogy: If you train a car to drive off-road (handle mud and rocks), it might feel slightly bumpier when driving on a perfect highway. But the trade-off is worth it because the car won't get stuck in the mud.

The Bottom Line

This paper shows that for languages like Marathi, we can't just rely on big, general AI models to handle real-world, messy text. We need to:

Test specifically for punctuation errors (using tools like Viram).
Train models to understand context even when punctuation is missing.
Accept that specialized, smaller models might be more reliable for critical tasks than the biggest, flashiest AI models.

The researchers have made their "gym" (the Viram dataset) and their "training methods" available to everyone so other developers can build better, safer translation tools for India and beyond.

1. Problem Statement

Neural Machine Translation (NMT) systems rely heavily on explicit punctuation cues to resolve semantic and structural ambiguities. However, user-generated text often lacks punctuation or contains errors. When such "noisy" input is fed into NMT systems, the output can be fluent but semantically disastrous.

Specific Context: The paper focuses on the English-to-Marathi language pair. Marathi is a low-to-mid-resource Indo-Aryan language spoken by over 80 million people.
The Gap: Current models (including standard NMT and Large Language Models) are highly sensitive to punctuation. A missing comma can completely alter the meaning of a sentence (e.g., changing a safety instruction into a dangerous one), yet there is a lack of diagnostic benchmarks and robust strategies specifically for Indic languages to handle this variability.

2. Methodology

The authors propose and evaluate two primary strategies to improve punctuation robustness, alongside an evaluation of Large Language Models (LLMs).

A. The Viram Benchmark

To stress-test existing systems, the authors created Viram (meaning "punctuation" in Marathi), a diagnostic benchmark consisting of 54 manually curated, punctuation-ambiguous sentence pairs.

Structure: Each instance follows the format: English (Written) $\rightarrow$ English (Meant) $\rightarrow$ Marathi (Meant).
Source: Sentences were derived from Kirkman (2006) and curated by two native Marathi speakers with advanced academic backgrounds.
Goal: To isolate and test the model's ability to infer correct semantic structure from ambiguous, unpunctuated input.

B. Proposed Approaches

The study evaluates two distinct paradigms for improving robustness:

Approach 1: Restore-then-Translate (Pipeline)
- Concept: A decoupled approach where punctuation is restored in the English source text before translation.
- Implementation:
  - Token Classification: Fine-tuning bert-large-uncased and microsoft-mpnet-base for sequence labeling.
  - Text-to-Text Generation: Fine-tuning google-t5-base to generate punctuated text from unpunctuated input.
  - Zero-shot: Using the pre-trained AI4Bharat Cadence model without fine-tuning.
- The restored text is then fed into the baseline NMT model (IndicTrans2).
Approach 2: Direct Fine-Tuning
- Concept: Training the NMT model directly on noisy data to learn robustness without a separate restoration step.
- Implementation: Fine-tuning the IndicTrans2 model on four data variants:
  - Original data with punctuation.
  - Data with all source punctuation removed.
  - A combined dataset (2x size) containing both punctuated and unpunctuated versions.
  - An alternating dataset (x size) where punctuation is randomly retained or removed per sentence.

C. LLM Evaluation

The authors evaluated three LLMs (Sarvam-2b-v0.5, Gemma-2-9b, LLaMA-3.1-8b) using Zero-shot and Three-shot prompting. They tested both "Restore-then-Translate" and "Direct Translation" prompting strategies to see if general-purpose models could handle the ambiguity better than task-specific fine-tuning.

3. Key Contributions

First Study on English-Marathi Punctuation Robustness: The paper provides the first systematic analysis of how punctuation errors affect English-to-Marathi translation.
The Viram Benchmark: A novel, high-quality diagnostic dataset of 54 ambiguous instances designed specifically to test semantic disambiguation capabilities.
Comparative Analysis of Strategies: A rigorous comparison between pipeline-based restoration and direct fine-tuning, revealing that while both improve performance, they have different trade-offs.
LLM Performance Gap: A demonstration that current sub-10B parameter LLMs (even with few-shot prompting) underperform compared to task-specific fine-tuned models on punctuation-ambiguous tasks.

4. Results and Analysis

Quantitative Performance (Viram Benchmark)

Baseline: The original IndicTrans2 model performed poorly on unpunctuated inputs (BLEU ~21.72).
Approach 1 (Pipeline): Consistently outperformed the baseline. The t5-base and mpnet restorers achieved BLEU scores approaching the "Oracle Upper Bound" (where the model is given the correct meaning directly), reaching ~24.74 BLEU. This indicates that high-quality punctuation restoration directly translates to better semantic recovery.
Approach 2 (Direct Fine-tuning): Training on mixed data (punctuated + unpunctuated) yielded significant gains (~24.27 BLEU), outperforming training on punctuated data alone. It showed superior robustness to variability.
LLMs: Even the best-performing LLM (Gemma-2-9b) achieved a BLEU of 14.84 (3-shot), significantly lagging behind the fine-tuned NMT models. This highlights that general LLMs struggle with specific structural ambiguities without specialized training.

Performance on Standard Benchmarks (IN22, FLORES-22)

Trade-off: While the proposed methods drastically improved performance on the Viram benchmark (punctuation-sensitive), they resulted in slight drops in standard metrics (BLEU, COMET) on general-purpose datasets like IN22 and FLORES-22.
Implication: There is a cost to robustness; optimizing for punctuation ambiguity may slightly reduce performance on clean, standard text. However, the mixed-data approach (Approach 2) minimized this gap better than the pipeline approach.

Qualitative Analysis

Error Patterns: The baseline model frequently failed on headline-style constructions and procedural instructions, misidentifying clause boundaries (e.g., confusing who confessed and who denied a claim).
Success of Proposed Methods:
- Approach 1 successfully restored clause boundaries but occasionally propagated errors from the restoration step.
- Approach 2 proved most stable, correctly inferring latent sentence structures and conditional logic even without punctuation, suggesting the model learned to rely on contextual cues rather than surface punctuation.

5. Significance and Conclusion

Critical for Safety: The paper underscores that punctuation errors in machine translation can lead to real-world disasters (e.g., misinterpreting safety instructions on fire extinguishers).
Research Direction: The findings suggest that for low-resource Indic languages, task-specific fine-tuning on mixed data is currently the most effective strategy for punctuation robustness, outperforming both standard NMT and general-purpose LLMs.
Future Work: The authors recommend developing metrics that better capture semantic nuance (beyond BLEU) and exploring hybrid architectures that handle punctuation ambiguity natively rather than through multi-stage pipelines.

In summary, this work establishes that while current NMT and LLM systems are fragile regarding punctuation, targeted data augmentation and fine-tuning strategies can significantly mitigate these risks, ensuring more reliable translation for critical applications in India.