NCTB-QA: A Large-Scale Bangla Educational Question Answering Dataset and Benchmarking Performance

This paper introduces NCTB-QA, a large-scale Bangla educational question-answering dataset featuring a balanced distribution of answerable and unanswerable questions with adversarial distractors, and demonstrates that fine-tuning transformer models on this domain-specific data significantly improves performance in low-resource settings.

The Gist

Imagine you are trying to teach a very smart, but slightly confused, robot how to read a textbook written in Bengali. You want this robot to be able to answer questions about the book. But there's a catch: sometimes the book doesn't have the answer to the question you ask.

If the robot is too eager, it might just make up a fake answer to look smart. This is called "hallucinating," and in a school setting, that's a disaster. It could confuse a student or make them lose trust in the system.

This paper introduces a new tool called NCTB-QA to solve this problem. Here is the story of how they built it and what they found, explained simply:

1. The Problem: The "Eager Beaver" Robot

Before this study, most question-answering systems for Bengali were like eager beavers. They were trained only on questions where the answer was in the text. So, when they encountered a question the text couldn't answer, they panicked and guessed anyway.

Also, the existing datasets (collections of practice questions) were too small. Imagine trying to learn to play the piano by practicing on just three songs. You wouldn't be ready for a real concert. The old Bengali datasets were like those three songs—too small and too simple to train a really smart AI.

2. The Solution: The "NCTB-QA" Training Ground

The researchers built a massive new training ground called NCTB-QA. Think of it as a giant library of 50 official textbooks from Bangladesh's National Curriculum and Textbook Board, covering grades 1 through 10.

They didn't just copy-paste the books; they turned them into a sophisticated video game for AI:

• The Scale: They created nearly 88,000 question-and-answer pairs. That's huge!
• The Twist (The "Trick" Questions): This is the most important part. They made sure that about 43% of the questions were "trick questions." These are questions where the answer is not in the text.
  • Analogy: Imagine a teacher asking, "What is the capital of France?" while showing a picture of a banana. A good student should say, "I don't know, that's not in the picture." A bad student would guess "Paris" anyway. NCTB-QA forces the AI to learn to say, "I don't know," when the answer isn't there.
• The Distractors: They even added "plausible distractors." These are trick questions that almost have the answer, but not quite. It's like asking, "Who won the 2024 World Cup?" when the text only talks about the 2022 World Cup. The AI has to be sharp enough to spot the difference.

3. The Experiment: Teaching the Robots

The researchers took three famous AI models (think of them as three different types of student brains: BERT, RoBERTa, and ELECTRA) and put them through a training camp using this new dataset.

• Before Training: The robots were okay at reading, but they were terrible at knowing when not to answer. They were confident but often wrong.
• After Training: They studied the NCTB-QA dataset.
  • The Result: The improvement was massive. One model, BERT, went from being barely able to answer correctly (15% success) to being very reliable (62% success). That's a 313% improvement!
  • The "Don't Know" Skill: The robots learned to stop guessing. When the answer wasn't in the text, they successfully admitted they didn't know, rather than making things up.

4. Why This Matters

This isn't just about getting better scores on a test. It's about trust.

• For Students: If a student uses an AI tutor to study for a math or science exam, they need to know that if the AI says "I don't know," it's because the answer really isn't there, not because the AI is broken.
• For Low-Resource Languages: Bengali is spoken by over 230 million people, but in the world of AI, it's considered "low-resource" (not enough data). This paper shows that by using official textbooks and creating smart "trick" questions, we can build powerful AI tools for these languages without needing millions of dollars in data.

The Bottom Line

The researchers built a giant, tricky practice test based on real school books. They used it to teach AI models how to read Bengali and how to know when to stop talking. The result is a much smarter, more honest AI that won't try to fake its way through a student's homework.

In short: They taught the AI to say "I don't know" when it's the right thing to do, making it a much safer and more useful tool for education.

Technical Summary

Here is a detailed technical summary of the paper "NCTB-QA: A Large-Scale Bangla Educational Question Answering Dataset and Benchmarking Performance."

1. Problem Statement

Reading comprehension systems for low-resource languages, particularly Bangla, face two critical challenges:

• Lack of Large-Scale Educational Data: Existing Bangla Question Answering (QA) datasets are small (typically 3,000–15,000 pairs), lack domain specificity (mostly general news or factoid questions), and are insufficient for training robust transformer-based models.
• Inability to Handle Unanswerable Questions: Most existing datasets contain only answerable questions. Consequently, models trained on them tend to "hallucinate" plausible but incorrect answers when the required information is missing from the context. This is particularly detrimental in educational settings where accuracy is paramount.
• Imbalanced Distributions: Existing datasets often have a skewed ratio of answerable to unanswerable questions (e.g., 3:1 or 100% answerable), failing to train models to effectively abstain from answering when appropriate.

2. Methodology

A. Dataset Construction (NCTB-QA)

The authors introduced NCTB-QA, a large-scale dataset derived from official educational materials.

• Source: 50 textbooks published by the National Curriculum and Textbook Board (NCTB) of Bangladesh, covering grades 1–10 across 17 subjects (e.g., Science, History, Bangla Literature).
• Scale: The dataset contains 87,805 question-answer pairs derived from 10,070 contextual segments.
• Pipeline:
  1. Automated Scraping: Textbooks were scraped from the NCTB website and cleaned to remove formatting noise, page numbers, and metadata.
  2. Segmentation: Text was segmented into coherent contexts (mean length ~290 words) with a 500-character overlap to preserve continuity.
  3. QA Generation: Using Gemini 2.5 Pro, the system generated diverse question types (Factual, Causal, Comparative, Negation, etc.).
  4. Adversarial Design: A key feature is the balanced inclusion of unanswerable questions (42.75%). These were adversarially designed to be topically relevant but lacking the specific answer in the context, forcing the model to learn abstention rather than guessing.
• Distribution: The dataset maintains a near-balanced split: 57.25% answerable and 42.75% unanswerable.

B. Experimental Setup

• Models: Three encoder-only transformer architectures were fine-tuned: BERT-base-cased, RoBERTa-base, and ELECTRA-base-discriminator.
• Preprocessing:
  • Answer spans were corrected via exact and partial matching.
  • Long contexts were handled using a sliding window approach (2,000 tokens, stride 300).
  • Unanswerable questions were treated as negative examples (empty string output).
• Training: Models were trained on a single 12GB VRAM GPU using mixed-precision (FP16), AdamW optimizer, and linear learning rate decay with warmup.
• Metrics:
  • Exact Match (EM): Strict token-level matching.
  • F1 Score: Token-level overlap.
  • BERTScore: Semantic similarity measurement using the Bengali language code (bn).

3. Key Contributions

1. NCTB-QA Dataset: The first large-scale (87k+ pairs), domain-specific (educational) Bangla QA dataset that explicitly includes a balanced distribution of unanswerable questions and adversarial distractors.
2. Comprehensive Benchmarking: Establishes baseline performance for BERT, RoBERTa, and ELECTRA on Bangla educational QA, providing a standard for future research.
3. Chain-of-Thought (CoT) Annotations: The dataset includes CoT reasoning annotations (though not utilized in the current extractive experiments), paving the way for future generative and reasoning-focused research.
4. Analysis of Low-Resource Adaptation: Provides empirical evidence on how English-pretrained models adapt to typologically distinct languages (Bangla) through fine-tuning.

4. Results

Overall Performance

Fine-tuning yielded dramatic improvements across all models, highlighting the necessity of domain-specific adaptation:

• BERT: Showed the most significant relative improvement.
  • F1 Score: Increased from 0.150 (zero-shot) to 0.620 (fine-tuned), a 313% relative improvement.
  • Exact Match: Improved from 0.059 to 0.509.
  • BERTScore: Improved from 0.665 to 0.880, indicating better semantic understanding.
• RoBERTa: Started with the strongest zero-shot performance (F1 0.428) due to its larger pre-training corpus but showed more modest gains after fine-tuning (F1 0.469, +9.6%).
• ELECTRA: Improved F1 from 0.406 to 0.550 (+35.5%).

Performance by Question Type

• Comparative Questions: BERT showed the largest gain here (F1 from 0.126 to 0.767), suggesting it struggled initially with comparative reasoning but adapted well.
• Unanswerable Detection: All models achieved very high F1 scores (0.82–0.98) on unanswerable questions, even before fine-tuning. This suggests that learning to "abstain" is a simpler task than accurate answer extraction.

5. Significance and Future Directions

• Educational Impact: NCTB-QA fills a critical gap in Bangla NLP resources, enabling the development of reliable automated assessment tools and personalized learning systems aligned with the national curriculum.
• Robustness: The dataset proves that domain-specific fine-tuning is critical for low-resource settings, as zero-shot performance from English-centric models is often insufficient for complex educational reasoning.
• Future Work:
  • Leveraging the included CoT annotations with generative models (e.g., Qwen 2.5) for interpretable reasoning.
  • Exploring monolingual Bangla pre-training or adapter-based methods to close the performance gap.
  • Extending the benchmark to multimodal tasks (incorporating diagrams and tables from textbooks).

In conclusion, NCTB-QA establishes a new standard for Bangla reading comprehension, demonstrating that while pre-trained models have potential, rigorous fine-tuning on high-quality, balanced educational data is essential to prevent hallucinations and ensure accuracy in real-world applications.