💬 NLP

Bangla Key2Text: Text Generation from Keywords for a Low Resource Language

This paper introduces Bangla Key2Text, a large-scale dataset of 2.6 million Bangla keyword–text pairs created via a BERT-based pipeline, and demonstrates that fine-tuning sequence-to-sequence models like mT5 and BanglaT5 significantly outperforms zero-shot large language models for keyword-driven text generation in this low-resource language.

Original authors: Tonmoy Talukder, G M Shahariar

Published 2026-04-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Tonmoy Talukder, G M Shahariar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to write a story in Bangla (the language of Bangladesh), but you only give it a handful of scattered words like "rain," "umbrella," and "sad." You want the robot to turn those words into a beautiful, coherent sentence like, "It was raining, so I felt sad because I forgot my umbrella."

This is the challenge the authors of this paper tackled. They created a massive new tool called Bangla Key2Text to help computers do exactly that.

Here is the story of their work, broken down into simple concepts:

1. The Problem: The "Empty Library"

Bangla is spoken by hundreds of millions of people, but in the world of AI, it's considered a "low-resource" language. Think of it like a library that has millions of books, but none of them have a "Table of Contents" or an index.

While big AI models (like the ones you chat with) are great at English, they often stumble when asked to write Bangla sentences just from a list of random keywords. They might get the words right but the grammar wrong, or they might invent facts that don't make sense.

2. The Solution: Building a Giant "Keyword-to-Story" Dictionary

The researchers decided to build a massive training manual for the AI. They didn't just write a few examples; they created 2.6 million pairs of "Keywords" and "Full Texts."

How they did it: They took millions of real Bangla news articles.
The Magic Trick: They used a smart computer program (a "keyword extractor") to act like a highlighter. It read the news, picked out the most important words (the "keywords"), and saved the original sentence next to them.
The Result: A giant dataset called Bangla Key2Text. It's like having a dictionary where every entry says: "If you give me these 3 words, here is how a human would write a sentence using them."

3. The Training: Teaching the Robot

Once they had this giant dataset, they had to teach two specific AI models (called mT5 and BanglaT5) how to use it.

The Analogy: Imagine you have a student who is very smart but hasn't studied for a specific test.
- Zero-Shot (The old way): You just ask the student, "Write a sentence about 'rain' and 'umbrella'." They might guess, but they aren't sure.
- Fine-Tuning (The new way): You give them the 2.6 million examples from the Bangla Key2Text dataset. You say, "Look at how these sentences are built. Learn the pattern."
The Outcome: After studying this dataset, the models became much better at turning messy lists of words into smooth, natural-sounding Bangla sentences.

4. The Showdown: Small Specialists vs. Big Generalists

The researchers wanted to see if their "specialized" models were better than the famous, giant AI models (like Llama or Mistral) that everyone uses.

The Race: They tested their trained models against several huge, powerful AI models that hadn't been trained specifically on this task.
The Winner: The smaller, specialized models (trained on the Bangla Key2Text dataset) won easily.
Why? The big models are like general doctors who know a little about everything but might miss the details of a specific local disease. The researchers' models are like local specialists who know the exact dialect, grammar, and flow of Bangla because they studied the local "textbook" (the dataset) extensively.

5. The Results: What Can It Do?

The paper shows that this new system can:

Handle Chaos: It works even if you give the keywords in a random order (e.g., "Friday," "crowded," "yesterday" vs. "yesterday," "crowded," "Friday"). The AI figures out the correct sentence structure.
Mix Languages: If you give it a mix of English and Bangla words, it usually figures out how to write the final sentence in pure Bangla.
Create Variety: It can write short sentences or longer paragraphs depending on how many keywords you give it.

6. The Catch (Limitations)

No system is perfect. The authors admit:

Dialects: The AI is great at standard Bangla (like the news), but it sometimes struggles with heavy regional accents or slang (like the specific dialects of rural villages).
Missing Words: Sometimes, the AI forgets to use one of the keywords you gave it, though it usually gets the meaning right.
Safety: Like any AI, it needs guardrails to ensure it doesn't generate harmful content.

The Big Picture

The most important thing this paper gives us isn't just the AI models; it's the dataset (the 2.6 million examples).

Think of it like the researchers digging up a massive gold mine of "Bangla writing examples" and leaving it open for everyone to use. Now, other scientists can use this gold to build better translators, better summarizers, and smarter chatbots for the Bangla-speaking world. They proved that for languages like Bangla, specialized training on good data beats just using a giant, generic AI.

1. Problem Statement

The paper addresses the scarcity of resources for Keyword-to-Text (Key2Text) generation in Bangla, a low-resource language despite having over 230 million speakers. While large language models (LLMs) have shown success in general text generation, the authors observe that:

Existing LLMs (even quantized versions) perform poorly on Bangla Key2Text tasks in a zero-shot setting, often failing to generate coherent, keyword-faithful text from unordered inputs.
There is no large-scale, structured dataset available for training supervised models specifically for Bangla keyword-driven generation.
General-purpose models struggle with the specific nuances of Bangla morphology and syntax when constrained by unordered keyword inputs without fine-tuning.

2. Methodology

The authors propose a three-stage pipeline: Dataset Construction, Keyword Extraction, and Text Generation.

A. Dataset Construction (Bangla Key2Text)

Source: The authors utilized the Bangla Newspaper Dataset (Nazi, 2020), selecting 2.6 million news articles.
Scale: The final dataset contains 2.6 million keyword-text pairs.
- Training Set: 2.0 million pairs.
- Validation Set: 500,000 pairs.
- Test Set: 100,000 pairs.
Annotation: A separate evaluation set of 10,000 pairs was manually curated by human annotators to verify the quality of automatic keyword extraction and to serve as a gold standard for decoding technique selection.
Statistics: The average text length is ~15 words, with an average of ~9.4 extracted keywords per text.

B. Keyword Extraction Pipeline

Since manual extraction for 2.6 million texts is infeasible, the authors developed an automatic BERT-based keyword extractor:

Scoring: They utilized a pre-trained BanglaBERT model. For each word in a text, they calculated the cosine similarity between the word's token embedding and the mean embedding of the entire text. Words with higher similarity (closer to the text's central context) were deemed more important.
Handling Sub-words: To address BERT's sub-word tokenization, embeddings of sub-tokens were averaged to form a single word embedding.
Selection: A dynamic threshold was applied based on text length:
- Texts with $\ge$ 10 words: Top 60% selected.
- Texts with 5–9 words: Top 70% selected.
- Texts with $\le$ 4 words: Top 80% selected.

Validation: The extractor was benchmarked against Bangla-BERT-Base, TextRank, and YAKE. The proposed BanglaBERT approach achieved the highest MRR (33.59%) and mAP (33.56%), outperforming traditional unsupervised methods.

C. Text Generation Models

The authors fine-tuned two encoder-decoder sequence-to-sequence models on the Bangla Key2Text dataset:

mT5: A multilingual T5 model.
BanglaT5: A T5 model specifically pre-trained on Bangla.

Training Details: Fine-tuned for 2 epochs using AdamW optimizer, OneCycleLR scheduler, and a learning rate of $1e^{-4}$ .
Decoding: Extensive experiments were conducted with Greedy, Beam Search, Top-k, and Top-p sampling.
- mT5 performed best with Top-p sampling ( $p=0.95$ ).
- BanglaT5 performed best with Beam Search (beam size 2).

3. Key Contributions

Bangla Key2Text Dataset: The first and largest dataset (2.6M pairs) for keyword-to-text generation in Bangla, publicly released to the community.
Novel Extraction Method: A lightweight, BERT-based keyword extraction pipeline that effectively identifies contextually salient keywords without manual supervision at scale.
Baseline Models: Fine-tuned mT5 and BanglaT5 models that serve as strong baselines for the task.
Comprehensive Evaluation: A rigorous benchmark comparing fine-tuned models against multiple 4-bit quantized open-source LLMs (e.g., LLaMA 3.1, Phi-3.5, Mistral, Gemma, Qwen) in a zero-shot setting.

4. Results and Findings

The models were evaluated using BERTScore, ROUGE, BLEU, WER (Word Error Rate), and WIL (Word Information Lost), alongside human evaluation.

Fine-tuning vs. Zero-shot: Task-specific fine-tuned models (mT5 and BanglaT5) significantly outperformed all zero-shot LLMs.
- BanglaT5 achieved the highest BERTScore (91.51%) and ROUGE-L (43.20%).
- Zero-shot LLMs (even large ones like LLaMA-3.1) showed very low ROUGE scores (often < 10%) and high WER, indicating they struggle to generate coherent Bangla text from unordered keywords without specific training.
Decoding Techniques: The choice of decoding strategy was critical. Beam search favored BanglaT5, while Top-p sampling favored mT5.
Temperature Sensitivity: Lowering the temperature ( $\tau=0.7$ ) generally improved performance for BanglaT5, while mT5 showed mixed results.
Keyword Order Sensitivity: The models demonstrated robustness to keyword reordering. While syntactic structures varied, the semantic meaning remained consistent (high BERTScore), though minor variations in output were observed.
Cross-Lingual Transfer: When provided with mixed Bangla-English keywords, the models predominantly generated Bangla text, demonstrating cross-lingual understanding, though BanglaT5 occasionally retained English words.
Dialectal Performance: The models struggled with regional dialects (e.g., Mymensingh, Barisal) due to a lack of dialect-specific training data, highlighting a gap for future research.
Human Evaluation: Annotators rated the generated texts as highly relevant (Fleiss' Kappa scores of 0.87 for mT5 and 0.84 for BanglaT5).

5. Significance and Future Work

Resource for Low-Resource NLP: This work provides a critical infrastructure (dataset + baselines) for advancing Natural Language Generation (NLG) in Bangla, a language often underserved by large-scale resources.
Efficiency: It demonstrates that smaller, task-specific fine-tuned models are more effective and efficient for low-resource tasks than relying on massive, general-purpose LLMs in zero-shot modes.
Future Directions: The authors suggest future work should focus on:
- Fine-tuning on dialectal data to improve generalization.
- Exploring few-shot learning.
- Improving constrained decoding to ensure all keywords are included without sacrificing fluency.
- Addressing safety concerns (filtering hate speech/harmful content).

In conclusion, Bangla Key2Text establishes a new benchmark for keyword-driven text generation in Bangla, proving that targeted data curation and model fine-tuning are essential for unlocking the potential of low-resource languages in complex generative tasks.