BengaliMCQ: Automatic Generation and Answer Prediction of Academic Multiple-Choice Questions in a Low-Resource Language

In the world of artificial intelligence, large language models have become powerful tools for reading, writing, and reasoning. These systems can process vast amounts of text, but they face a practical limit: they cannot read an entire library at once. When asked to generate a specific type of question based on a long textbook chapter, the model often gets lost in the sheer volume of words, missing the most important details. To solve this, researchers use a technique called retrieval-augmented generation. This approach acts like a librarian who first finds the exact pages a student needs before handing the book to the teacher. However, standard methods for finding these pages often treat a textbook like a flat list of sentences, ignoring the way chapters, headings, and paragraphs are organized. This is particularly difficult when working with languages like Bengali, where fewer digital resources exist to train these systems, leading to results that are often inaccurate or irrelevant.

A team of researchers at BRAC University in Bangladesh set out to fix this problem by teaching computers to understand the structure of a book, not just its words. They developed a system called BengaliMCQ, designed to automatically create multiple-choice questions for students and to answer those questions with high accuracy, all while using the Bengali language. Instead of feeding the entire textbook to the artificial intelligence, their system first builds a map of the book. It treats the textbook as a hierarchy, connecting chapters to headings, headings to paragraphs, and paragraphs to individual sentences. This map allows the computer to see how ideas flow from one section to another, much like a family tree shows relationships between relatives.

To make this map useful, the researchers trained a specialized computer program to navigate it. They created a dataset of thousands of sample questions and the specific passages in the books that answered them. Using this training, the system learned to identify which parts of a book were most relevant to a specific topic. When a user asks the system to generate a question about a topic like "photosynthesis" or "Bengali literature," the system does not scan the whole book. Instead, it uses its map to retrieve only the five most important passages. These short, focused excerpts are then sent to a large language model, which uses them to write a high-quality question or find the correct answer. This process drastically reduces the amount of information the computer has to process, making the task faster and more precise.

The results of this approach were striking. In tests comparing their method against standard techniques, the new system proved far better at finding the right information. While other methods often struggled to locate the correct passage, the researchers' system consistently identified the most relevant text. When they tested the system's ability to generate questions, the output was not only more accurate but also more diverse and pedagogically sound. Perhaps most impressively, the system achieved an answer prediction accuracy of 91.41 percent. This means that when given a question from a past exam, the system could correctly identify the answer more than nine times out of ten. In contrast, other methods that tried to read the entire book at once or used less structured search methods performed significantly worse, often getting lost in the noise of irrelevant text.

The researchers also discovered that the structure of the book was the most critical factor in their success. When they removed the connections that represented the hierarchy of the text—such as the link between a chapter and its subheadings—the system's performance collapsed. This proved that understanding the organization of the material was far more important than simply having access to the words themselves. They also found that the system worked best when it was trained to look for a specific number of related sentences and to ignore connections that were too weak. By fine-tuning these settings, they ensured the system focused only on the strongest relationships between ideas.

To ensure the quality of their work went beyond computer scores, the team enlisted human experts to review the generated questions. Two specialists in literature and science evaluated a random sample of the questions created by the system. The experts rated the vast majority of the questions as relevant and useful, with biology questions receiving the highest marks. This human validation confirmed that the system was not just mimicking patterns but was actually producing content that met the standards of a real classroom. The study also highlighted a significant efficiency gain: by using this structured approach, the system reduced the amount of text it needed to process from over 43,000 words down to just 1,651 words, without losing any of the necessary context.

Despite these successes, the researchers acknowledge that their work is not without limitations. The training data used to teach the system was created by other artificial intelligence models, which means there is a small chance of errors or "hallucinations" being built into the foundation. Additionally, the human evaluation was limited to a small group of experts and a specific set of topics. The team suggests that future work should involve larger datasets curated entirely by humans to further improve reliability. Nevertheless, this study demonstrates a clear path forward for educational technology in low-resource languages. By respecting the natural structure of textbooks and using it to guide artificial intelligence, it is possible to create tools that are both highly accurate and deeply relevant to the learning process, bridging the gap between complex technology and the everyday needs of students and teachers.