BM25 and Dense Retrieval Are Complementary for Portuguese Clinical Text: An Empirical Study of Hybrid RAG Across 500 Clinical Queries

In the high-stakes world of medicine, doctors rely on vast libraries of knowledge to make split-second decisions that affect human lives. For decades, these libraries were physical books and journals, but today, they are digital. A new technology called retrieval-augmented generation acts as a digital librarian for artificial intelligence. When a doctor asks a question, this system does not just guess an answer based on what the computer has memorized; instead, it first searches a specific database of trusted medical documents, finds the most relevant pages, and then uses those pages to construct a precise, evidence-based response. The critical challenge lies in the search itself: how does the computer know which documents are the right ones? For years, the assumption in English-speaking medicine has been that the best way to find answers is to understand the deep meaning and context of a question, much like a human reader does. This approach, known as dense retrieval, uses complex mathematics to map the "vibe" or semantic connection between words. However, a simpler, older method called keyword matching, which looks for exact word overlaps, has long been the standard for many systems. The question facing researchers is whether the sophisticated, meaning-based search is truly superior for every language, or if the simpler method still holds vital ground, especially in regions where medical terminology is highly standardized and specific.

This question became the focus of a rigorous study conducted by an independent researcher investigating how these digital libraries function for Portuguese-speaking doctors in Brazil. While artificial intelligence tools for healthcare are growing rapidly across Latin America, most of the research guiding their design has been done using English text. The researcher set out to test whether the prevailing wisdom—that deep semantic search is always better—held true for the specific, structured language of Brazilian clinical protocols. To do this, the team built a test environment using a massive, curated collection of real medical documents, including drug guides, emergency protocols, and national treatment guidelines. They then generated five hundred distinct clinical questions that a doctor might ask, covering six different medical specialties ranging from pharmacology to emergency care. The goal was to see which search strategy would successfully find the correct documents to answer these questions.

The researchers compared three main approaches. The first relied entirely on the simple keyword matching method. The second used only the complex, meaning-based search. The third was a hybrid system that combined both methods, merging their results to see if they could cover more ground together. The findings challenged the common belief that the sophisticated, meaning-based search was the ultimate solution. When the researchers tested the complex, meaning-based search on its own, it failed to find the correct documents for more than twenty-two percent of the clinical questions, even when the system was set to be very permissive in what it accepted as a match. In contrast, the simpler keyword method was remarkably effective, finding the right documents for ninety-nine percent of the queries. This high success rate for the simple method is likely due to the nature of Brazilian medical writing, where doctors and protocols often use the exact same standardized terms for drugs and conditions, making word-for-word matching highly effective.

However, the study did not declare the simple method the sole winner. In fact, the most significant discovery was that the two methods are not redundant; they are complementary. When the researchers looked at the documents found by the keyword method versus the documents found by the hybrid system, they found that the two approaches were pulling from largely different pools of information. The hybrid system uncovered hundreds of unique documents that the keyword method missed entirely, while the keyword method found hundreds of unique documents that the hybrid system overlooked. This means that relying on just one strategy leaves a doctor with an incomplete picture. The hybrid approach, which combines the precision of keyword matching with the contextual understanding of the complex search, provided the most comprehensive coverage, ensuring that no critical piece of information was left behind.

The study also examined how the system handles the authority of the sources it finds. In medicine, a national government guideline is often considered more authoritative than a single research study. The researchers tested whether boosting the ranking of these high-authority documents improved the system's ability to find the right information. They found that while this weighting changed the order in which documents appeared, pushing the most trusted sources to the top, it did not actually increase the total number of correct documents found. This distinction is crucial for system designers: if the goal is to ensure the most trusted information is seen first, ranking adjustments work, but if the goal is to find every possible relevant document, these adjustments do not help.

Finally, the researchers tackled a major hurdle in testing these systems: who decides if an answer is actually good? Traditionally, this requires hiring busy doctors to read and judge thousands of results, a slow and expensive process. The study tested whether an artificial intelligence could act as a judge to evaluate the quality of the search results. Two independent AI systems were asked to rate the relevance of the documents for each query. The two AIs agreed with each other on nearly every single judgment, achieving a level of consistency that is considered almost perfect. This suggests that for Portuguese medical text, automated evaluation is a reliable and scalable way to improve these systems without needing constant human oversight. Furthermore, the study demonstrated that by using a strict, programmatic method to link answers to their source documents, the system could completely eliminate the problem of "hallucinations," where an AI invents fake citations. While a standard approach resulted in hundreds of incorrect citations, the strict method ensured that every citation pointed to a real document that the system had actually retrieved.

The implications of these findings are clear for the future of medical AI in Portuguese-speaking regions. The assumption that a single, sophisticated search method is sufficient for all medical queries is incorrect. Instead, the most robust systems will likely be those that blend the reliability of simple keyword matching with the contextual depth of semantic search. This hybrid approach ensures that the system captures both the exact terminology of standardized protocols and the broader, nuanced connections between medical concepts. By validating that automated judges can reliably assess these systems, the study also paves the way for faster, more continuous improvements in clinical decision support tools, ultimately helping doctors access the right information more quickly and safely.