💬 NLP

Paramanu: Compact and Competitive Monolingual Language Models for Low-Resource Morphologically Rich Indian Languages

The paper introduces Paramanu, a family of compact, cost-effective monolingual language models trained from scratch on open-source data for five major Indian languages, which utilize specialized tokenizers and training techniques to outperform larger multilingual models despite their small size and single-GPU training budget.

Original authors: Mitodru Niyogi, Eric Gaussier, Arnab Bhattacharya

Published 2026-01-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Mitodru Niyogi, Eric Gaussier, Arnab Bhattacharya

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the world of Artificial Intelligence as a massive, bustling library. For a long time, this library has been almost entirely filled with books written in English. While there are some books in other languages, they are often just translations of the English ones, or they are written in a way that assumes the reader thinks in English. If you try to read a book about Indian languages in this library, you might find the pages are torn, the words are broken into tiny, confusing fragments, or the story just doesn't make sense.

The paper you're asking about introduces a new, specialized collection of books called PARAMANU. Here is the story of how they built it and why it matters, explained simply.

The Problem: The "One-Size-Fits-All" Suit Doesn't Fit

Current big AI models are like giant, heavy suits made for English speakers. When researchers try to force these suits onto speakers of Indian languages (like Hindi, Tamil, or Bengali), the suit doesn't fit right.

The "Broken Word" Problem: Indian languages are "morphologically rich," meaning words change shape a lot to add meaning (like adding "s" for plural or "ed" for past tense, but much more complex). Big models often chop these words into tiny, useless pieces, like trying to eat a whole apple by biting off just the skin. This makes the AI slow and inefficient.
The "English Brain" Problem: Even when these models speak Indian languages, they often "think" in English underneath, leading to awkward phrasing or bias.
The "Too Expensive" Problem: Building a new AI from scratch usually costs millions of dollars and requires supercomputers. This leaves researchers in India or with smaller budgets unable to build their own tools.

The Solution: The PARAMANU "Tailor-Made" Suits

The authors created PARAMANU, a family of five small, custom-tailored AI models. Each one is designed specifically for a single major Indian language: Bengali, Hindi, Marathi, Tamil, and Telugu.

Think of them not as giant, heavy suits, but as lightweight, custom-fitted uniforms made specifically for the people who wear them.

1. Built from Scratch, Not Patched Up

Instead of taking an English model and trying to "teach" it Indian languages (which is like trying to teach a French speaker to speak Hindi by only using French words), they built these models from the ground up using only Indian data. It's like building a house using only local bricks and wood, rather than importing materials that don't fit the climate.

2. The "Smart Shredder" (Tokenization)

One of the biggest innovations is a new way of breaking down words, called a tokenizer.

The Old Way: Imagine a shredder that cuts a word like "unbelievable" into "un," "believ," "able." It misses the root meaning.
The PARAMANU Way: They created a "smart shredder" that understands the grammar of Indian languages. It keeps the root of the word intact (like "unbeliev") and only separates the endings. This makes the AI understand the word faster and with fewer "pieces" to process. This is called low-fertility, meaning it doesn't create as many unnecessary fragments.

3. The "Budget-Friendly" Build

The most surprising part is the cost. Usually, training an AI is like trying to launch a rocket; it requires a massive budget.

PARAMANU's Trick: They managed to train these models on a single graphics card (a standard computer part) with a budget of less than $1,000.
The Analogy: It's like building a high-performance race car in a garage using a regular wrench and a few hundred dollars, instead of needing a factory and a million-dollar budget. This allows researchers with limited resources to build competitive tools.

4. Stretching the Memory (Context Scaling)

AI models have a "memory limit" (context window) on how much text they can read at once. Usually, if you want to read a longer book, you need a bigger computer.

The Innovation: The authors developed a mathematical trick (using something called RoPE interpolation) that allows the model to "stretch" its memory.
The Analogy: Imagine you have a short ruler. Instead of buying a longer one, they invented a way to mark the ruler so that every inch on the short ruler represents a mile on a map. This lets the model read longer sequences of text without needing more expensive hardware.

The Results: Small but Mighty

When they tested these small models (which range from 108 million to 367 million "parameters"—think of these as the model's brain cells) against much larger models (some with billions of brain cells):

The Underdogs Won: The small PARAMANU models often performed better than the massive, expensive multilingual models in their specific languages.
Efficiency: They achieved a better balance between performance and cost. They are like a compact, fuel-efficient car that gets better mileage than a gas-guzzling limousine.

The "Instruction" Library

To make these models helpful for real tasks (like answering questions or following commands), the team created a set of "instruction" examples in Bengali. They then carefully translated these into the other four languages. This is like giving the AI a recipe book in its native language, teaching it exactly how to cook the dishes it's supposed to serve.

Summary

The paper argues that for languages that are rich in grammar and have fewer digital resources, the best strategy isn't to build a bigger, more expensive model. Instead, the best strategy is to build smaller, specialized models that are:

Native: Trained only on that specific language.
Smart: Using a tokenizer that understands the language's structure.
Accessible: Trainable by anyone with a modest budget.

They proved that you don't need a billion-dollar budget to build a great AI for Indian languages; you just need the right tools and a focus on the specific needs of the language.

Technical Summary: PARAMANU

Problem Statement

Current Large Language Models (LLMs) are predominantly trained on English and high-resource European languages, leading to significant performance disparities for low-resource, morphologically rich languages like those spoken in India. Multilingual models often suffer from "English-centric bias," where they "think in English," resulting in degraded performance on non-Latin scripts. Furthermore, these models frequently utilize a "language-fits-all" tokenizer that causes over-segmentation in morphologically rich languages, leading to high token fertility, increased inference costs, and reduced fluency. Adapting existing English-centric models to Indian languages via continual pretraining or fine-tuning presents challenges such as vocabulary extension inefficiencies, embedding misalignment, and catastrophic forgetting, often requiring computational resources beyond the reach of under-resourced researchers.

Methodology

The authors introduce PARAMANU, a family of monolingual, autoregressive decoder-only language models trained from scratch specifically for five major Indian languages: Bengali, Hindi, Marathi, Tamil, and Telugu. The models range in size from 108M to 367M parameters and are designed to be trained on a single GPU with a budget under $1,000.

Key Technical Components:

Data Curation and Cleaning:
- The pretraining corpus consists of ~54.6 GB of UTF-8 text (25–66 billion tokens) derived from web-scraped news, blogs, and Wikipedia (IndianCorp v2, Vacaspati).
- Rigorous cleaning includes regex filtering of HTML/XML, removal of non-literal characters, language identification (using cld3 and IndianLID-FTN), toxicity filtering, and deduplication via MurmurHash5.
- Sentence splitting utilizes language-specific punctuation (e.g., the "danda" | for Bengali and Devanagari scripts).
Morphology-Aligned Tokenization:
- To address over-segmentation, the authors developed a hybrid tokenizer combining Byte Pair Encoding (BPE) with Unigram Language Model tokens.
- The Unigram tokens, which capture global morphological structures through probabilistic pruning, are merged into the BPE vocabulary.
- This approach preserves stems and frequent suffixes as atomic subwords, significantly reducing token fertility (the number of tokens per word) compared to standard BPE or multilingual models like Sarvam 2B and Llama-3.
Context Scaling via Positional Interpolation:
- To train on longer sequences (up to 4096 tokens) on hardware with limited context windows (e.g., 256 tokens on a single A100 GPU), the authors employ a shrinking factor for Rotary Positional Embeddings (RoPE).
- Token position indices $p$ are scaled to $p' = p \cdot (L/y)$ , where $L$ is the hardware-permissible context length and $y$ is the target length. This allows the model to learn long-range dependencies by mapping absolute positions into a narrower, hardware-supported range while preserving relative distances.
Instruction Tuning:
- A dataset of 23,000 instructions was created for Bengali, comprising human-authored samples, translated Dolly instructions, and self-instruct samples.
- These were machine-translated (with manual post-editing) to Hindi, Marathi, Tamil, and Telugu to align the models with human instructions.
Model Architecture:
- Based on the Transformer decoder architecture, utilizing RMSNorm, SwiGLU activation, and weight tying.
- Configurations follow Chinchilla scaling laws, with models ranging from 108M to 367M parameters.

Key Contributions

PARAMANU Family: The first open-source, Indian-only, sub-400M decoder language models trained from scratch on fully transparent, language-specific data for Bengali, Hindi, Marathi, Tamil, and Telugu.
Low-Fertility Tokenizers: Novel morphology-aligned tokenizers that combine BPE and Unigram methods to reduce token counts and improve morphological coherence for Indian scripts.
Efficient Training Techniques: An interpolation-based method for RoPE position indices enabling long-context training on single-GPU setups, and the use of $\mu$ P transfer for hyperparameter optimization.
Instruction Datasets: Novel instruction-tuning datasets in Bengali, translated to other target languages, facilitating better alignment with human instructions.

Results

The PARAMANU models were evaluated against 21 multilingual and Indian-adapted LLMs (ranging from 200M to 8B parameters) using benchmarks such as MMLU, ARC, Belebele, XNLI, and language-specific datasets (HellaSwag, XCOPA, XStoryCloze).

Performance vs. Size: Despite their small size (<400M parameters) and limited training data (<70B tokens), PARAMANU models outperform most larger multilingual models (up to 3B parameters) across all five languages.
Comparison with Larger Models: The models outperform 6 out of 10 larger models (>3B parameters), including those trained with up-sampled low-resource data (e.g., xGLM). While models like LLaMA-3-8B and Bloomz-7B lead overall due to massive instruction tuning, PARAMANU remains competitive.
Ablation Studies:
- The hybrid tokenizer improved downstream task performance by 0.56% to 2.02% over standard BPE.
- Data cleaning further boosted scores by 1% to 3.5%.
- Position interpolation allowed for effective training on longer sequences, with performance generally increasing with context size.
Cross-Lingual Transfer: The models demonstrated strong zero-shot transfer capabilities within the Devanagari script family (e.g., a Marathi model performing well on Hindi benchmarks), suggesting utility for cross-lingual tasks.

Significance and Claims

The paper argues that for low-resource, morphologically rich languages, the optimal strategy is not necessarily maximizing model scale or relying on massive multilingual corpora. Instead, building language-specific models with low-fertility, morphologically aligned tokenizers trained on cleaned, specific data yields a superior performance-efficiency tradeoff.

PARAMANU demonstrates that small monolingual models can be competitive with much larger multilingual alternatives when trained under tight computational and budget constraints. This approach makes advanced language modeling accessible to under-resourced researchers and communities, providing a practical foundation for NLP research and downstream applications in Indian languages. The authors emphasize that their open-source release of models, tokenizers, and datasets aims to bridge the gap for languages historically underrepresented in AI research, fostering transparency and community-driven advancement.