Development and Evaluation of HEAL A Bilingual English-Luganda RAG Chatbot for Disease Surveillance in Uganda

Imagine the world of artificial intelligence as a massive, super-smart library where a robot librarian can answer any question you ask. But here's the catch: most of these librarians only speak English, and they are trained on books from all over the world, not the specific rulebooks used by doctors in a particular country. This creates a problem for "Retrieval-Augmented Generation" (RAG). Think of RAG not as a robot that just guesses answers from its memory, but as a robot that is forced to open a specific textbook, find the exact page, and read the answer out loud before speaking. This stops the robot from making things up. Then, there's the issue of "Large Language Models" (LLMs), which are the brains behind these robots. While they are brilliant, they often struggle with languages that don't have millions of digital books written in them, like Luganda, a language spoken by millions in Uganda. Why does this matter? Because when a disease outbreak happens, health workers need to know exactly what to do right now. If the only guide is a thick, technical English book that they can't read quickly, or if a robot gives them a wrong answer because it doesn't know Uganda's specific rules, people could get sick or even die.

This paper introduces a new project called HEAL (AI for Health Equity), a bilingual chatbot designed to be that perfect librarian for Uganda. The team wanted to see if they could build a robot that doesn't just guess, but actually reads Uganda's official disease surveillance guidelines (the IDSR manual) and answers questions in both English and Luganda. They took the official guidelines, which are hundreds of pages long and written in complex English, and translated them into Luganda. Then, they built a system that uses a powerful AI brain (GPT-4-turbo) to find the right page in those guidelines and generate an answer, while a special translation tool ensures the answer makes sense in Luganda.

The researchers tested this new HEAL bot against three famous commercial robots (GPT-4, GPT-3.5, and Gemini) using 50 real questions that health workers in Uganda had actually asked during their daily jobs. Six experts from the Ministry of Health and the Infectious Diseases Institute acted as judges, scoring the answers on how relevant they were, how complete the information was, and how well they were written. The results showed that HEAL was the clear winner. It scored higher than all the commercial models in the overall analysis, and the experts rated it the highest most consistently. The paper suggests that grounding the AI in the specific national guidelines and adding a local language layer made it much more useful than the generic, off-the-shelf alternatives.

However, the story isn't a perfect fairy tale. While the experts liked HEAL, the translation from English to Luganda wasn't flawless. The team measured the translation quality and found that while it was good enough to understand the medical facts, the Luganda phrasing sometimes felt a bit clunky or unnatural. The paper explicitly rules out the idea that a simple English-only tool would work for these workers; they found that without the Luganda layer, the tool fails to solve the access problem. Furthermore, the study highlights that while the AI is smart, it still relies on the internet and cloud servers, which can be a barrier for health workers in rural areas with poor connectivity. The authors are careful to say that while the tool is promising and accurate, it hasn't been proven yet to actually stop outbreaks or save lives in the long run; they only measured how well it answered questions and how much the workers liked using it.

In the end, the paper suggests that the future of health AI in places like Uganda isn't about having the biggest, most expensive brain, but about having a brain that knows the local rules and speaks the local language. The team found that by combining a powerful AI with a specific set of national guidelines and a translation layer, they could create a tool that outperforms the big commercial giants in a real-world setting. But they also warn that before this can be used everywhere, the translation needs to be smoother, and the system needs to work without needing a constant internet connection. It's a successful prototype that shows the way forward, but the journey to a fully operational, life-saving tool is still ongoing.