Bilingual Retrieval-Augmented Access to ANVISA Regulatory Documents: A Technical Evaluation Using Drug-Development and Health-Product Questions
This study evaluates a bilingual retrieval-augmented generation system built on ANVISA regulatory documents, demonstrating its high accuracy in locating, citing, and summarizing Portuguese and English regulatory requirements for drug development while effectively handling unanswerable queries after calibration.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Regulatory science is the quiet engine that keeps medicines, vaccines, and medical devices safe for the public. Before a new drug reaches a pharmacy shelf, a company must navigate a labyrinth of rules written by government health authorities. In Brazil, this authority is ANVISA, which publishes thousands of pages of regulations in Portuguese. These documents dictate everything from how clinical trials are run to how a company must report a side effect. For scientists and regulators working in English, or for teams juggling multiple languages, these rules present a significant barrier. The information is there, but finding the exact sentence that answers a specific question can feel like searching for a single needle in a stack of hay bales, especially when the needle is written in a different language than the question.
To solve this, researchers have begun testing a type of computer system that acts like a highly skilled librarian. Instead of just guessing an answer based on what it has read before, this system first goes to a specific library of documents, finds the exact pages that contain the answer, and then uses those pages to write a response. This method, known as retrieval-augmented generation, is designed to prevent the computer from making things up. The goal is not to replace human experts, but to help them find the right rule quickly and accurately, ensuring that every answer can be traced back to its original source.
A team of researchers recently put this idea to the test using a collection of six key Brazilian health regulations. They built a prototype system that could understand questions asked in either English or Portuguese and search through the official Portuguese documents to find the answers. The challenge was twofold: the system had to bridge the language gap, and it had to be precise enough to handle complex legal details where a missing word could change the meaning entirely. The researchers wanted to know if a machine could not only find the right article but also summarize it correctly, cite its source, and, crucially, admit when it did not know the answer.
The researchers started by cleaning and organizing the raw text of the regulations. They did not just feed the documents into the computer as giant blocks of text. Instead, they broke the documents down into their natural building blocks: individual articles. This was a deliberate choice because regulatory rules often hinge on specific clauses within a single article. By treating each article as a distinct unit, the system could pinpoint exactly where a rule lived. They then taught the computer to recognize key terms in both English and Portuguese, creating a bridge that allowed an English question to find a Portuguese answer. The system was designed to work within a closed library; it was strictly forbidden from searching the live internet or guessing facts that were not present in the documents it was given.
When the researchers tested the system with 200 different questions, the results were encouraging but revealed important boundaries. For questions where the answer was clearly written in the documents, the system performed with remarkable accuracy. It successfully located the correct articles, cited them properly, and summarized the rules in the language the user asked. Whether the question was about the timing of safety reports for clinical trials or the rules for registering biological products, the system found the right evidence nearly every time. In these cases, the computer acted as a reliable guide, narrowing down the search and presenting a clear, verifiable summary.
However, the study also highlighted where the system needed to be careful. The most significant challenge arose when the system was asked questions that sounded like they should have an answer in the documents but actually did not. For instance, if a user asked for the current version of a specific electronic form or a phone number for a regulator, the system sometimes tried to answer using a related but incomplete rule from the text. This is a dangerous error in a regulatory context, where an incomplete answer could lead a company to make a costly mistake. The researchers found that the system initially struggled to say "I don't know" when the specific fact was missing.
To fix this, the team adjusted the system's instructions to be more cautious. They taught it to refuse to answer whenever the exact information was not explicitly present, even if a related rule existed. After this adjustment, the system became much better at knowing its limits. It correctly refused to answer nearly all the questions that fell outside its library, while still answering the valid questions with high accuracy. This trade-off was essential: it is safer for a regulatory tool to say it cannot help than to offer a partial or misleading answer.
The researchers concluded that this bilingual system is a powerful tool for navigating complex health regulations, provided it is used correctly. It is not a magic oracle that can solve every problem or give legal advice. Instead, it is a specialized assistant that helps users locate and summarize evidence within a defined set of documents. The study showed that when the answer exists in the text, the system can find it and explain it clearly in multiple languages. But its true value lies in its ability to recognize when the answer is missing and to stop there, leaving the final decision to a human expert who can verify the citation. In the high-stakes world of drug development and public health, knowing exactly what you do not know is just as important as finding what you are looking for.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.