💬 NLP

Polite on the Surface, Wrong in Practice: A Curated Dataset for Fixing Honorific Failures in Multilingual Bangla Generation

This paper introduces BLADE, a curated dataset of 4,196 interaction pairs designed to address honorific and pragmatic failures in multilingual Bangla generation, demonstrating that fine-tuning open-weight models on this resource significantly improves structural fidelity and cultural alignment.

Original authors: Md. Asaduzzaman Shuvo, Mahedi Hasan, Md. Tashin Parvez, Azizul Haque Noman, Md. Shafayet Hossain Ovi

Published 2026-05-22

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Md. Asaduzzaman Shuvo, Mahedi Hasan, Md. Tashin Parvez, Azizul Haque Noman, Md. Shafayet Hossain Ovi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Core Problem: The "Polite but Clueless" Robot

Imagine you ask a very smart, well-read robot to write a formal letter to your school principal asking for a few days off. The robot writes something that looks perfect at first glance. It has big words, it's grammatically correct, and it flows well.

But then, you read closer. The robot starts the letter saying, "Dear Sir," but halfway through, it switches to talking to the principal like they are your best friend, using slang and informal pronouns. It might say, "Hey, can you give me a break?" right after saying, "I respectfully request your permission."

To a native speaker, this is a disaster. It's like wearing a tuxedo to a funeral but then doing a backflip on the way in. The robot sounds fluent, but it fails the most important test: social competence.

The paper argues that current AI models (LLMs) are great at learning vocabulary but terrible at learning cultural rules, especially for languages like Bangla where the way you speak changes based on who you are talking to (honorifics).

The Solution: The "BLADE" Dataset

To fix this, the researchers built a special training manual called BLADE (BangLa Applications and DialoguEs).

Think of existing AI training data as a giant library of random books, news articles, and internet comments. It's huge, but it's messy. If you want to learn how to write a formal letter, you might find a few examples, but they might be mixed with casual text or contain errors.

BLADE is different. It's like a strict, expert-led writing workshop.

The Content: It contains 4,196 carefully crafted examples of formal applications (like leave requests) and dialogues.
The Teachers: These weren't just written by computers. They were curated by humans, checked against government textbooks, and verified by language experts to ensure the "politeness levels" (honorifics) were perfect.
The Rule: Every single example teaches the AI that in a formal setting, you must always use the "respectful" version of "you," never the "casual" version, and you must follow a specific structure (Date → Recipient → Subject → Body → Sign-off).

The Experiment: Teaching the Robot the Rules

The researchers took several popular AI models (some huge, some small) and gave them a choice:

The "Zero-Shot" Test: Ask the AI to write a letter without any special training. (Result: The AI sounded fluent but made rude mistakes, like mixing formal and informal language).
The "Fine-Tuning" Test: Feed the AI the BLADE dataset to learn the specific rules of formal Bangla writing.

The Results:

Before Training: Even the biggest, most expensive AI models failed to write a usable letter. They got low scores because they didn't understand the social rules.
After Training: The models improved dramatically.
- The "Small" Winner: A tiny model (only 1.5 billion parameters) trained on BLADE actually performed better than the massive, untrained models.
- The Analogy: It's like taking a small, local chef who knows the exact recipe for a traditional dish and giving them a perfect cookbook. They will cook a better meal than a world-famous chef who has never seen that specific recipe before.

Key Takeaways

Data Quality > Model Size: For languages with complex social rules (like Bangla), having a massive brain isn't enough. You need the right training data. A small model with high-quality, culturally specific data beats a giant model with generic data.
The "Pragmatic Gap": There is a huge difference between a model that can speak a language and a model that can use it correctly in real life. The paper shows that without specific training, AI fails at the "real life" part.
Structure Matters: The AI didn't just learn words; it learned the shape of a formal document. It learned that a formal letter needs a date at the top and a specific sign-off at the bottom, not just random paragraphs.

What the Paper Does Not Claim

It does not claim this fixes all AI problems.
It does not claim this works for informal slang or casual texting (the dataset was focused on formal, institutional writing).
It does not claim the models are now perfect; they are just much better at this specific task than before.

In a nutshell: The paper says, "If you want an AI to write a formal letter in Bangla, don't just give it a bigger brain. Give it a better teacher and a strict rulebook."

Technical Summary: BLADE – Fixing Honorific Failures in Multilingual Bangla Generation

1. Problem Statement

Despite advances in Multilingual Large Language Models (MLLMs), a critical "pragmatic gap" persists in low-resource languages like Bangla. While models often achieve surface-level fluency, they fail to adhere to the deep cultural and structural conventions required for functional usability. Specifically, general-purpose models struggle with:

Honorific Inconsistency: Bangla encodes social relationships grammatically through a two-tier honorific system (Apni for formal/respectful, Tumi for informal). Zero-shot models frequently mix these registers within a single document (e.g., addressing a school principal with informal pronouns), rendering the output institutionally unacceptable.
Structural Deficiencies: Models often omit mandatory document components (date, subject line, salutation, signature) or misplace them, failing to follow the canonical order of Bangla institutional writing.
Register Blindness: Current pretraining data, often derived from web-scale corpora or machine-translated instructions, lacks the "register-aware supervision" necessary to internalize format-sensitive and culturally appropriate discourse markers.

The authors argue that this is not a vocabulary issue but a structural limitation affecting any language where institutional writing grammatically encodes social relationships.

2. Methodology

The BLADE Dataset

To address these gaps, the authors introduce BLADE (BangLa Applications and DialoguEs), a curated instruction-tuning dataset comprising 4,196 high-quality prompt-response pairs across 2,008 unique topics. The dataset covers educational, professional, administrative, and conversational domains.

Data Acquisition Pipeline (Three-Tier):

Institutional Textbooks (1,972 samples): Extracted from government-approved NCTB textbooks to establish a "gold standard" for canonical document formatting.
Public Web Portals (1,382 samples): Curated from verified Bangla educational portals, with ~23% discarded due to register inconsistencies or malformed structures.
Author-Synthesized Examples (842 samples): Purpose-built to address complex, underrepresented scenarios (e.g., multi-turn dialogues requiring sustained honorific consistency), cross-validated by native linguistic experts.

Key Design Principles:

Register Awareness: Enforcing consistent alignment of salutations, pronouns, verb morphology, and closings.
Structural Fidelity: Strict adherence to canonical document order (Date → Addressee → Subject → Salutation → Body → Closing → Signature).
Topical Diversity: Focusing on real-world institutional writing rather than generic web content.

Experimental Setup

Zero-Shot Baseline: Five state-of-the-art models (including Gemini-2.5-Flash, LLaMA-4-Scout-17B, Kimi-K2-32B) were evaluated on BLADE without fine-tuning.
Supervised Fine-Tuning (SFT): Four architectures (DeepSeek-8B, Qwen2-1.5B, LLaMA-3.2-3B, TigerLLM-1B) were fine-tuned using Parameter-Efficient Fine-Tuning (PEFT).
- Technique: LoRA (Low-Rank Adaptation) with rank $r=16$ , $\alpha=32$ .
- Quantization: 4-bit NormalFloat (NF4).
- Hardware: Dual NVIDIA Tesla T4 GPUs (32 GB VRAM).
- Configuration: 2 epochs, 2048 token sequence length (with sliding-window packing), cosine learning rate decay ( $2\times10^{-5}$ ), and a format-aware prompt template.

Evaluation Metrics

The study employs a multi-faceted evaluation framework:

Automatic Metrics: BLEU, chrF (character n-gram F-score), ROUGE-L, WER (Word Error Rate), and BERTScore.
LLM-as-Judge: GPT-4.1 scored 450 samples on a 1–10 scale based on structural correctness, register consistency, and semantic relevance.
Human Evaluation: Three native Bangla philology experts rated 100 stratified samples on a 5-point Likert scale for Structural Integrity, Fluency, and Cultural Alignment.

3. Key Results

Zero-Shot Performance

Even the strongest zero-shot baselines (e.g., Gemini-2.5-Flash) failed to produce functionally usable Bangla documents, achieving a maximum BLEU of only 7.88.

Failure Modes: Systematic structural displacement (missing dates/subjects) and honorific mismatch (oscillating between Apni and Tumi).
Human Scores: Zero-shot baselines scored an average of 1.85/5 on Structural Integrity.

Fine-Tuning Performance

Fine-tuning on BLADE yielded substantial improvements across all models, demonstrating that data quality outweighs model scale for low-resource pragmatic generation.

DeepSeek-8B: Showed the most dramatic improvement, with BLEU increasing 22-fold (0.78 → 17.73) and chrF rising from 9.85 to 45.87. The LLM-judge score jumped from 3.8 to 8.9.
Qwen2-1.5B: Despite its small size, it achieved the lowest post-SFT WER (0.84) and highest BERTScore (0.748), outperforming zero-shot Gemini-2.5-Flash on these metrics.
Human Evaluation: Post-SFT models achieved average scores exceeding 4.6/5 across all dimensions (Structure, Fluency, Cultural Alignment).
Convergence: Automatic metrics, LLM-judge scores, and human expert evaluations converged, confirming genuine improvements in functional usability rather than superficial metric inflation.

Ablation Study

The study identified several critical factors for success:

Context Length: Extending to 2048 tokens (vs. 512) was decisive for preserving long-form structural integrity.
Prompt Design: Format-aware templates and explicit role tags significantly reduced WER and improved structural scores.
Adapter Configuration: Applying LoRA to both attention and MLP layers outperformed attention-only adaptation.

4. Significance and Claims

The paper claims that cultural specificity of instruction-tuning data is more critical than model scale for low-resource pragmatic generation.

Data over Scale: A compact model (Qwen2-1.5B) fine-tuned on expert-curated, register-aware data outperformed zero-shot frontier models (Gemini-2.5-Flash) that are orders of magnitude larger.
Functional Usability: The primary contribution is shifting model outputs from "surface-level fluency" to "institutionally acceptable" documents. The authors argue that without register-aware supervision, general-purpose pretraining cannot substitute for the specific cultural conventions required in languages like Bangla.
Benchmarking: BLADE serves as a rigorous benchmark for measuring pragmatic gaps and evaluating the efficacy of fine-tuning strategies in low-resource, high-context environments.

The authors conclude that BLADE provides a pathway to bridging pragmatic disparities in multilingual text generation, suggesting that future progress in underserved languages relies on high-quality, domain-specific supervision rather than merely increasing parameter counts.