Raising Bars, Not Parameters: LilMoo Compact Language Model for Hindi

This paper introduces LilMoo, a 0.6-billion-parameter Hindi language model trained from scratch on a high-quality, transparently curated corpus, which outperforms similarly sized multilingual baselines and demonstrates that specialized pretraining can effectively address low-resource language gaps without relying on opaque foundation models.

The Gist

Imagine the world of Artificial Intelligence (AI) as a massive library. For a long time, the librarians (AI researchers) have been building huge, all-encompassing encyclopedias that try to teach every language in the world at once. These are the "Multilingual Giants" like Qwen or Llama. They are incredibly smart, but because they are trying to cover 7,000+ languages, the section dedicated to any single low-resource language (like Hindi) is often thin, dusty, and full of gaps.

Enter "LilMoo."

Think of LilMoo not as a giant encyclopedia, but as a specialized, hyper-local community guide for Hindi speakers. It's a small, compact AI model (only 0.6 billion parameters, which is tiny compared to the giants) that was built from the ground up specifically to understand the nuances, culture, and rhythm of the Hindi language.

Here is the story of how they built it, explained simply:

1. The Problem: The "Jack of All Trades" Trap

The paper argues that while big AI models are great, they often treat low-resource languages like an afterthought. It's like hiring a tour guide who knows a little bit about every country in the world but doesn't know the secret shortcuts or local slang of your specific village.

• The Old Way: Take a giant, opaque model (like Qwen) and try to "fine-tune" it for Hindi. It's like trying to teach an adult who speaks only English how to speak Hindi by just showing them a few phrases. They might get the basics, but they'll miss the soul of the language.
• The LilMoo Way: Build a new model from scratch, using only the best Hindi materials available. It's like raising a child who grows up speaking only Hindi, surrounded by the best books and teachers.

2. The Ingredients: "GigaLekh" (The Great Book)

To train this AI, you need a massive library of text. The team created a dataset called GigaLekh (which sounds like "Giga" + "Lekh," meaning "Great Writing").

• The Filter: The internet is full of junk—spam, ads, and nonsense. The team didn't just download everything; they acted like strict librarians. They used a "smart judge" (a powerful AI called Qwen2.5) to read thousands of documents and rate them: "Is this educational? Is it toxic?"
• The Result: They ended up with a pristine library of 90 billion words of high-quality Hindi text. They even kept a separate "trash bin" of toxic content to study it later, but they didn't feed that to the baby AI.

3. The Training Recipes: Two Ways to Cook

The team tried two different "recipes" to train LilMoo, like a chef testing two different ways to make a perfect curry.

• Recipe 1: The Pure Hindi Dish (LilMoo-v0.1)

  • The Idea: Feed the AI only Hindi. No English, no code, just pure Hindi.
  • The Goal: See how deep the model can go if it focuses entirely on one language.
  • The Result: It became very good at Hindi, but it was a bit rigid.

• Recipe 2: The Fusion Dish (LilMoo-v0.2)

  • The Idea: Feed the AI Hindi plus high-quality English (like textbooks, science articles, and math problems).
  • The Analogy: Think of this like a bilingual child who grows up in a Hindi-speaking home but goes to an English-speaking school. They learn the logic and structure of English, which helps them understand complex concepts better, but they still speak Hindi fluently at home.
  • The Result: This version (v0.2) turned out to be the superstar. It learned to reason better and handle complex tasks because the English data acted as a "boost" for its brain, without making it forget its Hindi roots.

4. The Surprise: Small is Beautiful

The most exciting part of the paper is the "David vs. Goliath" moment.

• The Giants: The team compared their tiny 0.6B model against massive models like Qwen3 (which has 0.6B parameters but was trained on 100 times more data and computing power).
• The Outcome: LilMoo-v0.2 beat the giants on almost every Hindi test.
• The Metaphor: Imagine a local expert (LilMoo) who knows every street in Mumbai, beating a world-famous tourist guide (Qwen) who has visited 100 countries but doesn't know the local bus routes.
• The Efficiency: The team calculated that to train the giant Qwen model, they needed about 100 times more electricity and computer power than they used for LilMoo. If that same amount of power were used to build 100 different small models for 100 different languages, the world would have 100 specialized experts instead of one overworked generalist.

5. Why This Matters

This paper is a manifesto for democratizing AI.

• Transparency: Unlike many big companies that keep their training secrets, the LilMoo team released everything: the code, the data, the recipes, and the models. It's like giving everyone the cookbook, not just the finished cake.
• Sustainability: It proves you don't need a supercomputer the size of a city to build a smart AI. With smart data curation and careful training, you can build powerful tools for low-resource languages without breaking the bank or the planet.

In a nutshell:
LilMoo is a proof-of-concept that says: "Don't just add more languages to a giant model; build small, focused, high-quality models for each language." It's a shift from "bigger is better" to "smarter and more specific is better," ensuring that Hindi speakers (and speakers of other low-resource languages) finally get an AI that truly understands them.

Technical Summary

Here is a detailed technical summary of the paper "RAISING BARS, NOT PARAMETERS: LILMOO COMPACT LANGUAGE MODEL FOR HINDI".

1. Problem Statement

The field of Natural Language Processing (NLP) is currently dominated by large, multilingual foundation models (e.g., Qwen, LLaMA) trained on massive datasets. While these models offer broad coverage, they often leave low-resource languages like Hindi underrepresented due to data imbalances and the "dilution" of language-specific capabilities in multilingual training.

• The Gap: Existing Hindi models typically rely on Continual Pretraining (CPT) of opaque, proprietary multilingual foundations. This approach lacks transparency, reproducibility, and deep insight into the specific learning dynamics of Hindi.
• The Challenge: Building a high-quality, native Hindi model from scratch is difficult due to the scarcity of clean, high-quality digital text and the prevalence of "Hinglish" (code-mixed Hindi-English) and English contamination in available data.
• The Goal: To demonstrate that a compact, language-specific model trained from scratch with a transparent, reproducible pipeline can outperform larger, multilingual baselines, even with significantly fewer parameters and compute resources.

2. Methodology

The authors developed LilMoo, a 0.6-billion-parameter (670M) Hindi language model. The project involved a full-stack approach from data curation to training and evaluation.

A. Data Curation: GigaLekh

The team constructed GigaLekh, a high-quality Hindi corpus containing ~90 billion tokens from 83 million documents.

• Pipeline: Based on the FineWeb2 framework, adapted for Hindi.
• Filtering:
  • Heuristic: Rule-based filters for structural coherence (punctuation, repetition, length).
  • Learned Filtering (LLM-as-a-Judge): Used Qwen2.5-32B to annotate a sample of 320k documents for Educational Quality and Toxicity. These annotations were distilled into lightweight classifiers (using Hindi-RoBERTa) to filter the entire corpus at scale.
• Composition: The final dataset includes diverse domains with a knowledge cutoff of December 2025. Toxic documents were separated into an "excluded" subset rather than discarded, preserving them for future safety research.

B. Tokenizer Design

A custom tokenizer was trained from scratch using SentencePiece on a mixture of Hindi (40%), English (40%), and Code (20%).

• Vocabulary Size: 49,152 tokens.
• Optimization: Designed to align with GPU-friendly dimensions (multiples of 128/256) to maximize tensor core utilization.
• Performance: Achieved a Subword Fertility of 1.46, outperforming multilingual baselines like Qwen2.5 and Llama-3.2 in token efficiency for Hindi.

C. Model Architecture

LilMoo is based on the Llama architecture (Decoder-only Transformer) with specific optimizations for efficiency:

• Parameters: ~670M (Hidden size: 1,536; Layers: 28; Attention Heads: 16).
• Features: RMSNorm, RoPE positional embeddings, SwiGLU activations, and Grouped-Query Attention (GQA).
• Design Philosophy: A "deep and slim" configuration to improve generalization and reasoning at small scales.

D. Training Recipes

Two distinct training strategies were explored:

1. LilMoo-v0.1 (Single-Stage): Trained exclusively on Hindi data using a cosine learning rate decay. This serves as a monolingual baseline.
2. LilMoo-v0.2 (Multi-Stage): Trained on a mixture of high-quality Hindi and English data (approx. 50:50 ratio).
   • Curriculum: Three stages (Warmup+Stable, Stable, Stable+Decay) progressively up-sampling higher-quality educational and reasoning data (e.g., FineMath, Big-Reasoning-Traces, OpenScience).
   • Rationale: Leveraging the symbiotic relationship between Hindi and English to enhance cross-lingual transfer and reasoning capabilities.

E. Infrastructure & Optimization

• Hardware: Trained on 8x NVIDIA A100 GPUs (2 nodes).
• Optimizations: Used PyTorch DDP, BF16 precision with TF32, FlashAttention-2, and Liger kernels.
• Efficiency: Achieved ~70% Model FLOP Utilization (MFU).

3. Key Contributions

1. GigaLekh Corpus: A high-quality, filtered Hindi dataset (~90B tokens) with associated quality/toxicity classifiers and metadata.
2. LilMoo Models: Two open-weight models (v0.1 and v0.2) trained from scratch, fully transparent in their data and training recipes.
3. Evaluation Harness: A comprehensive, reproducible benchmark suite specifically for Hindi, analyzing which metrics provide reliable signals for small models.
4. Open Release: All code, datasets, models, and training recipes are released under Apache 2.0, promoting reproducibility in low-resource NLP.

4. Results

The models were evaluated against strong multilingual baselines (Qwen2.5-0.5B, Qwen3-0.6B, Llama-3.2-1B, OpenHathi-7B) using a suite of benchmarks (ARC, HellaSwag, MMLU, CSQA, Global PIQA, MILU).

• Performance:
  • LilMoo-v0.2 achieved a Normalized Preferred Metric (NPM) score of 9.94, significantly outperforming Qwen3-0.6B (4.08) and Qwen2.5-0.5B (1.92).
  • LilMoo models surpassed the baselines on 5 out of 6 primary benchmarks.
  • LilMoo-v0.2 (Bilingual) generally outperformed LilMoo-v0.1 (Monolingual), except on Global PIQA (a culturally specific task), where the monolingual model performed better, suggesting that English data can sometimes dilute cultural nuance.
• Compute Efficiency:
  • LilMoo models were trained with ~1.3 × 10²¹ FLOPs.
  • Qwen3-0.6B required ~1.3 × 10²³ FLOPs (approx. 100x more compute) yet achieved a lower performance score.
  • Implication: The compute budget required for one large multilingual model could theoretically train ~100 specialized, language-specific models of similar size with superior performance.

5. Significance and Conclusion

This paper challenges the prevailing "scale-first" paradigm in NLP. It demonstrates that careful data curation and targeted, language-specific pretraining are more effective than simply scaling up multilingual models for low-resource languages.

• Scientific Impact: It provides a blueprint for building transparent, reproducible models for languages other than English, countering the "black box" nature of proprietary foundation models.
• Practical Impact: It proves that high-performance NLP for Hindi is achievable with limited compute resources, making advanced AI accessible to researchers and developers in low-resource settings.
• Future Work: The authors suggest scaling these recipes to larger models (1B–3B parameters) and applying the methodology to other Indic and low-resource languages to further validate the "Raising Bars, Not Parameters" hypothesis.

In summary, LilMoo serves as a proof-of-concept that specialized, compact models can rival and exceed the capabilities of massive multilingual giants when the training pipeline is optimized for the specific linguistic and cultural context of the target language.