💬 NLP

Multilingual TinyStories: A Synthetic Combinatorial Corpus of Indic Children's Stories for Training Small Language Models

This paper introduces Multilingual TinyStories, a large-scale synthetic corpus of over 132,000 children's stories in 17 Indic languages, generated via a hybrid pipeline of native model generation and cross-lingual translation to address data scarcity for training Small Language Models.

Original authors: Deepon Halder, Angira Mukherjee

Published 2026-03-17

📖 4 min read☕ Coffee break read

CC0 1.0

Original authors: Deepon Halder, Angira Mukherjee

Original paper dedicated to the public domain under CC0 1.0 (http://creativecommons.org/publicdomain/zero/1.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a tiny, eager puppy how to speak. If you throw a massive, chaotic encyclopedia at it, the puppy gets confused. But if you give it simple, short, and clear stories about rabbits, forests, and lost toys, the puppy learns the rules of language quickly and happily.

This paper is about building a massive library of those simple stories, but with a twist: it's written in 17 different Indian languages, many of which don't have enough good books for computers to learn from.

Here is the story of how they built this library, explained simply:

1. The Problem: The "Empty Bookshelf"

For a long time, the smartest computer brains (AI models) have been trained mostly on English. They are like students who only read English textbooks. If you try to teach them Hindi, Tamil, or Urdu, they struggle because there aren't enough clean, simple, high-quality stories available in those languages. The existing data is often messy, like a pile of old newspapers mixed with random internet comments.

The researchers wanted to fix this for Small Language Models (SLMs). Think of these models as "puppy-sized" AIs. They are smaller, cheaper, and faster than the giant AIs, but they need very specific, clean training data to learn properly.

2. The Solution: A "Story Factory"

Instead of trying to find existing stories (which are rare), the team decided to manufacture them. They built a "Story Factory" with two main machines:

The Master Chef (Sarvam-M): For 7 of the languages, they used a powerful AI called Sarvam-M. But they didn't just ask it to "write a story." That would be boring and repetitive.
The Slot Machine (Combinatorial Prompts): Instead, they built a "slot machine" for the AI. They created a template with empty slots, like a Mad Libs game:
- Character: [A curious rabbit]
- Setting: [A magical forest]
- Problem: [A lost lantern]
- Lesson: [Bravery]
They mixed and matched these slots millions of times. This forced the AI to write millions of unique stories that were simple enough for a 5-year-old to understand, ensuring the computer learned the basics of grammar and storytelling without getting confused by complex words.

3. The Expansion: The "Translator Bridge"

They had great stories in 7 languages, but they needed 17. To get the other 10 languages, they used a Translator Bridge.

They picked Gujarati as their "base camp" because the AI wrote the cleanest stories in that language.
They used Google Translate to carefully translate those Gujarati stories into the other 10 languages (like Urdu, Sanskrit, and Manipuri).
The Safety Net: They added a strict filter to make sure no English words or Latin letters "leaked" into the stories. If a story had a stray English word, it got thrown in the trash. They wanted the stories to be pure and native.

4. The Result: A Massive Library

The result is a dataset called Multilingual TinyStories.

Size: It contains over 132,000 stories.
Scale: That's roughly 94 million words (tokens).
Variety: It covers 17 languages, including some that are rarely seen in AI research.

5. Why This Matters

Think of this dataset as a gym for small AI models.

Before this, small AI models trying to learn Indian languages were trying to lift heavy, rusty weights (messy data).
Now, they have a set of perfectly balanced, lightweight dumbbells (clean, simple stories).
This allows researchers to train smaller, more efficient AI models that can understand and speak these languages fluently, making technology accessible to millions more people across India.

The Catch (Limitations)

The authors are honest about the flaws. Since the stories were made by a computer, sometimes the logic might be slightly weird (like a rabbit flying to the moon without a rocket). Also, the translated stories might not sound exactly like how a human grandmother would tell a story; they might feel a bit "robotic" in their phrasing. But for teaching a computer the basics of language, it's a fantastic starting point.

In a nutshell: They built a factory to print millions of simple, clean children's stories in 17 Indian languages, giving small computer brains the perfect training ground to learn how to speak and understand their local tongues.

1. Problem Statement

The development of robust Small Language Models (SLMs) for low-resource languages, particularly within the Indian subcontinent, is hindered by a scarcity of high-quality, coherent, and structurally simple training corpora.

Current Limitations: Existing resources (e.g., IndicCorp, Samanantar) are often massive, web-scraped, noisy, and formally complex. These characteristics make them suboptimal for the initial syntactic alignment and pre-training of small-parameter models.
The Gap: While Large Language Models (LLMs) excel in high-resource languages like English, they struggle with zero-shot capabilities in Indic languages due to a lack of clean, domain-appropriate data. There is a specific need for datasets that restrict vocabulary complexity to match the reading level of a young child (approx. 5 years old) to help SLMs learn core linguistic structures without being overwhelmed by jargon or noise.

2. Methodology

The authors propose a hybrid generation pipeline to create a synthetic corpus of children's stories across 17 Indic languages. The process involves two main stages:

A. Native Synthetic Generation (Core Languages)

For a foundational set of 7 languages, the authors used the Sarvam-M language model (an Indic LLM) combined with a Combinatorial Prompt Engineering framework.

Dynamic Prompting: To avoid mode collapse and ensure semantic variance, they avoided static prompts. Instead, they utilized a slot-based template system with over 15 narrative archetypes (e.g., Adventure, Moral, Mystery).
Variable Slots: Prompts were dynamically populated by uniformly sampling from curated conceptual pools:
- Characters ( $N \ge 50$ ): e.g., rabbit, robot, princess.
- Settings ( $N \ge 40$ ): e.g., forest, space station.
- Objects/Problems ( $N \ge 50$ ): e.g., lost lantern, hidden treasure.
- Themes ( $N \ge 30$ ): e.g., bravery, honesty.
Constraints: Generation was strictly constrained to:
- Length: 5 to 8 sentences.
- Token Count: 80 to 200 tokens per story.
- Decoding: Stochastic decoding with Temperature = 0.9 and Top-p = 0.95.
- Target Audience: Language simple enough for a 5-year-old.

B. Cross-Lingual Expansion (Remaining Languages)

To scale to the remaining 10 underrepresented languages, the authors employed a machine translation pipeline:

Pivot Language: The Gujarati split was selected as the source corpus due to its high structural fidelity and minimal script contamination during preliminary testing.
Translation: The validated Gujarati dataset was programmatically translated into 10 target languages (Assamese, Dogri, Konkani, Maithili, Manipuri, Nepali, Sanskrit, Santali, Sindhi, and Urdu) using the Google Translate API.

C. Data Cleaning and Normalization

Script Normalization: A programmatic filter removed all characters outside the specific Unicode blocks of the target language to eliminate Latin character leakage and embedded English tokens.
Filtering: Strict filtering ensured the final corpus contained only clean, native-script text.

3. Key Contributions

Multilingual TinyStories Dataset: A large-scale, open-source synthetic corpus containing 132,942 stories and approximately 93.9 million tokens across 17 Indic languages.
SLM-Tailored Resource: The dataset is specifically designed for pre-training and aligning Small Language Models (10M–200M parameters) by providing clean, structurally simple narratives that avoid the noise of web-scraped data.
Hybrid Generation Framework: A novel pipeline combining native LLM generation with combinatorial prompt engineering and machine translation, effectively bridging the gap for heavily underrepresented languages.
Exclusion of High-Resource Languages: Unlike previous efforts, this dataset explicitly excludes languages like Hindi, Bengali, and Marathi to focus resources on underrepresented demographics.

4. Results and Statistics

Corpus Size: The final release includes 132,942 stories. The largest splits are Punjabi (13,144 stories) and Odia (13,004 stories), while the smallest is Urdu (3,374 stories).
Lexical Diversity:
- Distinct-n Metrics: Native-generated languages (e.g., Kannada, Malayalam, Tamil, Telugu) showed high lexical diversity with Distinct-2 scores exceeding 0.65.
- Translation Impact: Languages generated via the translation pipeline showed slightly lower diversity (Distinct-2 ranging from 0.25 to 0.69), attributed to normalization effects in machine translation.
- Semantic Variability: Average cosine similarity between sentence embeddings ranged from 0.11 to 0.38, indicating significant semantic variation and low repetition.
Script Purity: The normalization filter successfully removed Latin characters. The modification rate varied by language, with Punjabi (41.1%) and Odia (31.3%) showing the highest initial contamination, which was successfully cleaned.

5. Significance and Future Impact

Democratizing AI: This dataset addresses a critical barrier to democratizing AI in the Indian linguistic sphere by providing a foundational resource for languages that lack extensive, clean corpora.
Efficient Training: The constrained vocabulary and simple narrative structure allow SLMs to converge rapidly while learning syntax, semantics, and basic reasoning without the computational cost required for massive, noisy datasets.
Research Catalyst: The release includes the full prompt templates and filtering scripts, facilitating reproducibility and further research into synthetic data generation and cross-lingual NLP.
Limitations: The authors note that as a synthetic dataset, it carries a risk of subtle semantic hallucinations. Furthermore, the translated languages may lack strict idiomatic nuance compared to native human speech, making the dataset best suited for pre-training and structural alignment rather than tasks requiring high factual accuracy or complex reasoning.

In conclusion, Multilingual TinyStories represents a significant step forward in low-resource NLP, demonstrating that high-quality, synthetic, combinatorial data can effectively train capable language models for underrepresented Indic languages.