BanglaDiaHallu: Hallucinations of large language models and the effect of guideline-grounded retrieval-augmented generation in Bangla diabetes question answering
This study demonstrates that while large language models exhibit significant hallucination rates in Bangla diabetes question answering, grounding a locally deployable model with Bangladesh's National Guideline on Diabetes Mellitus via retrieval-augmented generation substantially improves medical accuracy and safety, though clinical oversight remains essential as no configuration is entirely free of errors.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer
In the modern world, people increasingly turn to artificial intelligence for answers about their health, hoping for quick, reliable guidance. These computer programs, known as large language models, are trained on vast amounts of text and can speak fluently in many languages, including Bangla, which is spoken by hundreds of millions of people. However, these systems have a well-known flaw: they can sound perfectly confident while stating things that are simply not true. In the medical field, where a wrong word can lead to dangerous choices, this tendency to invent facts is called a hallucination. It is a particular worry in languages like Bangla, where the training data for these computer programs is often thinner and less reliable than in English. For millions of people living with diabetes in Bangladesh, getting accurate, safe advice in their own language is not just a convenience; it is a critical need, especially when local hospitals are often overwhelmed and remote consultations are becoming the norm.
A team of researchers set out to measure just how often these computer programs make up medical facts when asked about diabetes in Bangla, and to see if connecting them to a trusted source could stop the errors. They gathered forty common questions that patients might ask, covering topics from diet and medication to managing the disease during pregnancy or religious fasting. They then asked four different open-source computer models to answer these questions. These models varied in size and design, with some being general-purpose tools and others specifically tuned for the Bangla language. The researchers also built a special system that forced one of the models to look up answers in the official National Guideline on Diabetes Mellitus for Bangladesh before speaking, a method known as retrieval-augmented generation. Two doctors then read every single answer, rating them on accuracy, safety, and clarity, and marking any response that contained a false or unsupported medical claim as a hallucination.
The results revealed a startling difference between the models. The most advanced general-purpose model made up facts in only 2.5% of its answers, while a model designed specifically for Bangla but not for medicine performed almost as well. However, one of the general models, which was smaller and less specialized, hallucinated in a staggering 77.5% of its responses. It invented foods that do not exist in Bangladesh, suggested dangerous changes to medication, and offered advice that could harm a patient. This showed that simply having a model that speaks the language is not enough; the underlying intelligence of the model matters just as much as the language it uses.
The researchers then tested whether grounding the weakest model in the official national guidelines could fix its problems. They gave the model access to the correct medical rules and instructed it to answer only from that text. The improvement was dramatic. The rate of hallucinations dropped from 77.5% down to 11.3%. The answers became much more accurate, complete, and safe, and the model's performance rose to match or even exceed the Bangla-adapted model on most criteria, though it remained below the best-performing model in safety, readability, and reliability. The system successfully turned a dangerously unreliable tool into a much safer assistant for patients.
Yet, the study also found that the solution was not perfect. Even with the official guidelines, the model still made mistakes in about one out of every nine answers. These remaining errors were particularly concerning because they were not obvious nonsense. Instead, they were fluent, well-structured sentences that contained specific, confident, and potentially harmful advice, such as telling a patient to stop taking insulin or to use a non-existent medical device. The model sometimes even cited the official guidelines to support these false claims, making the errors harder to spot. This suggests that while connecting a computer to a trusted source helps immensely, it does not eliminate the risk entirely. The researchers concluded that for these tools to be safe for patients, they must be built on a strong base model, grounded in official guidelines, and always checked by a human doctor before being used to give medical advice. The work provides a clear path forward for creating health assistants that are both accessible and trustworthy for millions of Bangla speakers.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.