Oral to Web: Digitizing 'Zero Resource'Languages of Bangladesh

This paper introduces the Multilingual Cloud Corpus, the first national-scale, parallel, multimodal dataset for Bangladesh's predominantly oral and endangered minority languages, which was systematically collected through extensive fieldwork and is now publicly available to advance digital preservation and low-resource natural language processing.

The Gist

Imagine Bangladesh as a giant, colorful tapestry. Most people see it as a single, solid block of blue thread because almost everyone speaks Bengali. But if you look closer, you'll see hundreds of tiny, intricate threads woven in the corners—threads of different colors and textures. These are the languages of Bangladesh's ethnic minorities.

For a long time, these "tiny threads" were at risk of unraveling and disappearing forever. They existed only in people's mouths (oral tradition) and had no written form, no books, and no presence on the internet. If the elders who spoke them passed away, the languages would vanish like smoke.

This paper describes a massive rescue mission called "Oral to Web." It's like building a digital time capsule to save these disappearing voices before they are gone.

Here is the story of how they did it, explained simply:

1. The Problem: The "Silent" Languages

Think of the internet as a giant library. English, Spanish, and even Bengali have huge, well-organized shelves full of books, movies, and websites. But for many of Bangladesh's 40+ minority languages, that library shelf is completely empty. They are "Zero Resource" languages.

Without digital records, these languages can't be used on phones, computers, or in modern education. They are slowly fading away because the younger generation is switching to Bengali. The researchers wanted to stop this fading by turning spoken words into digital data.

2. The Plan: Building a "Digital Museum"

The team didn't just want to write down words; they wanted to build a Multilingual Cloud Corpus. Imagine a massive, interactive museum where every exhibit is a language.

• The Blueprint: Before they went out, they designed a strict "recipe" to make sure every language was recorded in the same way. They created a checklist of 2,200 specific things to record:
  • The "Word Jar": 475 basic words (like "mother," "tiger," "rain," "to eat").
  • The "Sentence Builder": 887 sentences showing how grammar works (like "I am eating," "I will eat," "I ate").
  • The "Storyteller": 46 real-life scenarios (like "buying groceries," "telling a folk tale," or "comforting a sick child").

This ensured that they could compare Language A directly with Language B, like comparing apples to apples, rather than just guessing.

3. The Adventure: The Fieldwork

The team sent out 16 "language detectives" to nine different districts across Bangladesh. They traveled to remote villages, tea gardens, and hill tracts to find the speakers.

• The Process: They sat with native speakers (77 of them) and recorded them.
  • First, they asked for words.
  • Then, they asked for sentences.
  • Finally, they asked the speakers to have natural conversations or tell stories based on the scenarios they prepared.
• The Safety Net: To make sure the recordings were accurate, they had "validators"—other people from the same village who listened to the recordings and said, "Yes, that sounds right," or "No, that's not how we say it."

4. The Result: A Digital Treasure Chest

After 90 days of hard work, they created something incredible:

• 85,792 structured entries: A huge database of words and sentences.
• 107 hours of audio: Recordings of people speaking naturally.
• 42 different languages: Covering four major language families (like the Tibeto-Burman, Austro-Asiatic, Dravidian, and Indo-European families).

Every single entry has three layers:

1. Bengali: What the speaker was asked to say.
2. English: What it means.
3. IPA (International Phonetic Alphabet): A scientific code that shows exactly how the sounds are made, so anyone can learn to pronounce them correctly.

5. Why This Matters (The "So What?")

You might ask, "Why do we need a computer to record people talking?"

• For the Speakers: It gives them a voice in the digital world. Now, they can type their language on a keyboard, use it in apps, and teach their children using digital tools. It's like giving them a key to the modern world.
• For the World: It saves knowledge. Some of these languages, like Rengmitcha, have only six elderly speakers left. If they pass away without this recording, the language dies forever. This project captures the last breath of those languages so future generations can study them.
• For Computers: Artificial Intelligence (AI) needs data to learn. Right now, AI is great at English but terrible at these small languages. This database is like "food" for AI, teaching computers how to understand and speak these languages.

6. The "Multilingual Cloud"

All this data is now live on a website called multiling.cloud. Think of it as a public park where anyone can walk in, listen to the songs of these languages, read the stories, and learn the words. It's the first time these languages have been organized and shared on a national scale.

The Bottom Line

This paper isn't just about linguistics; it's about preservation. It's like a fire department rushing to save a burning house, but instead of water, they are using data. They are taking the oral traditions of Bangladesh's most vulnerable communities and turning them into a permanent, digital legacy that will survive long after the speakers are gone.

They proved that even in a developing country, with limited resources, you can build a bridge between the ancient oral past and the digital future.

Technical Summary

Here is a detailed technical summary of the paper "Oral to Web: Digitizing 'Zero Resource' Languages of Bangladesh."

1. Problem Statement

Bangladesh is often perceived as linguistically homogeneous, with Bangla spoken by 98% of the population. However, it harbors approximately 40 ethnic minority languages, 14 of which are classified as endangered. Despite their existence, these languages suffer from:

• Lack of Digital Resources: They fall into "Class 0" or "Class 1" of the NLP resource taxonomy (Joshi et al., 2020), meaning they have virtually no digital footprint, labeled datasets, or pre-trained models.
• Fragmented Documentation: Prior efforts have been descriptive, community-specific, and non-systematic. No parallel corpus existed across all four major language families (Tibeto-Burman, Indo-European, Austro-Asiatic, Dravidian) at a national scale.
• Oral-Only Existence: Many of these languages lack standardized orthographies or written traditions, making them inaccessible to modern computational linguistics and digital preservation tools.

The core challenge is to transform these "zero-resource" oral languages into structured, web-accessible digital resources to enable both computational research and community-driven revitalization.

2. Methodology

The project, titled "Multilingual Cloud," followed a three-phase methodology to construct a large-scale, multimodal parallel corpus:

Phase 1: Pre-Field Preparation

• Template Design: A standardized elicitation template was created to ensure cross-linguistic comparability. It included:
  • Words: 475 unique lexical items across 22 semantic domains (e.g., verbs, kinship, nature).
  • Sentences: 887 unique items covering grammatical constructions, including verbal conjugation paradigms (varying person, number, tense, aspect) and 16 sentence types (e.g., interrogative, imperative, complex).
  • Directed Speech: 46 conversational scenarios (e.g., domestic life, health, folklore) totaling 862 prompts to elicit naturalistic discourse.
• Tooling: A custom fieldwork management platform was developed for real-time annotation and audio segmentation.
• Pilot: A pilot session with the Hajong community refined the protocols and technical pipeline.

Phase 2: Fieldwork (Data Collection)

• Scope: Data was collected over 90 days across 9 districts in 7 zones of Bangladesh, covering 41 target languages.
• Personnel: 16 data collectors worked with 77 native speakers and 43 validators.
• Process:
  1. Elicitation: Speakers were presented with Bengali stimuli (words/sentences) and recorded producing equivalents in their native language.
  2. Validation: A separate validator from the same community verified the accuracy, naturalness, and representativeness of the data to prevent individual idiosyncrasies or code-switching.
  3. Demographics: The corpus includes 188 unique speaker identities, with speaker counts per language ranging from 1 to 10.

Phase 3: Post-Field Lab Work

• Transcription: 10 linguists transcribed all audio into the International Phonetic Alphabet (IPA) over 4 months.
• Adjudication: 6 adjudicators performed independent quality reviews to resolve discrepancies and ensure consistency.
• Community Review: Finalized datasets were reviewed by community members for cultural and linguistic accuracy.
• Publication: The data was published on the Multilingual Cloud (multiling.cloud) platform.

3. Key Contributions

• The Multilingual Cloud Corpus: The first national-scale, parallel, multimodal linguistic dataset for Bangladesh's ethnic languages.
  • Scale: 85,792 structured textual entries.
  • Audio: Approximately 107 hours of transcribed audio recordings.
  • Coverage: 42 language varieties (including dialects) from four language families and two unclassified languages.
• Tri-Modal Alignment: Every entry contains a Bengali stimulus, an English translation, and an IPA transcription, aligned with segmented audio.
• Taxonomic Breadth: Covers the Tibeto-Burman (21 varieties), Indo-European (9), Austro-Asiatic (8), and Dravidian (5) families, including critically endangered languages like Rengmitcha (6 speakers) and Kharia (5 speakers).
• Digital Infrastructure: The launch of a public web platform featuring interactive maps, audio playback, and universal keyboard support for indigenous scripts.

4. Results and Data Statistics

• Data Completeness:
  • Bengali text: 100% complete.
  • English translation: 99.76% complete (missing only 210 entries in directed speech).
  • IPA Transcription: 99% complete (missing 860 entries, primarily in the Mundari language).
• Per-Language Variance:
  • Median entries per language: ~1,961.
  • Range: 1,217 (Soura) to 4,145 (Telugu, due to multiple dialects).
  • Audio duration ranges from 64 minutes (Gurkhali) to 352 minutes (Telugu/Madrasi).
• Geographic Distribution: The largest dataset share came from Moulvibazar (tea garden communities) and the Chittagong Hill Tracts (Bandarban, Rangamati, Cox's Bazar).
• Dialect Handling: The corpus distinguishes between sociolinguistic varieties (e.g., Sadri Oraon vs. Sadri Srimangal; Telugu vs. Madrasi Telugu) while grouping mutually intelligible dialects (e.g., Marma North/South) where appropriate.

5. Significance and Impact

• Computational Linguistics:
  • Provides the foundational seed data (107 hours) required for Automatic Speech Recognition (ASR) and Machine Translation (MT) in extremely low-resource settings, addressing the "left-behind" status of these languages.
  • Enables Cross-Lingual Transfer Learning, particularly within the Tibeto-Burman family, which has been historically neglected in NLP research.
  • Supports morphological analysis and lexicon induction through its parallel structure.
• Language Revitalization:
  • Serves as a "digital ark" for critically endangered languages (e.g., Rengmitcha) where intergenerational transmission has ceased, preserving the language even if it ceases to be spoken.
  • Empowers communities by providing the first structured digital representation of their languages, including fonts and keyboards, facilitating digital literacy.
• Policy and Preservation:
  • Demonstrates that large-scale, government-supported digital documentation is feasible in developing nations.
  • Offers a replicable model for other linguistically diverse regions in South and Southeast Asia.

6. Limitations and Future Work

• Limitations: Uneven data coverage due to speaker availability; potential inter-annotator variation in IPA transcription; limited speaker diversity for some languages (e.g., Gurkhali has only one speaker).
• Future Directions: Expanding speaker diversity, developing specific NLP tools (ASR, TTS, MT), creating language learning applications, and capturing additional cultural narratives and ceremonial language.