Ask Twice, Look Twice: Training-Free Consistency Filtering for Reliable Medical VQA

In the high-stakes world of modern medicine, doctors increasingly rely on artificial intelligence to help interpret complex medical images, from X-rays to MRI scans. One specific type of AI, known as a visual question answering system, is designed to look at a picture and answer a natural language question about it, such as "What organ is visible here?" or "Is there a fracture?" While these systems hold immense promise for supporting clinical decisions, they face a critical hurdle: reliability. Just as a human might hesitate or give a different answer if a question is phrased slightly differently, these computer programs can sometimes stumble when the wording of a question changes, even if the meaning remains exactly the same. If an AI gives a confident but wrong answer because a doctor asked the question in a new way, the consequences in a hospital setting could be severe. The core challenge, therefore, is not just making these models smarter, but making them consistent—ensuring they give the same correct answer regardless of how the question is asked or how the image is slightly altered.

A team of researchers from the University of Sydney and Macquarie University has developed a new method to solve this problem without needing to retrain the artificial intelligence models from scratch. They call their approach "Ask Twice, Look Twice." Instead of trying to force the AI to learn new things, they built a safety filter that checks the model's confidence before it is allowed to give an answer. The system works by taking a single medical image and a question, then automatically generating several different versions of that question that mean the exact same thing. For example, if the original question is "What disease is shown on the left of the lung?", the system might create variations like "What is observed in the left lung?" or "What finding is present on the left side?" It then asks the AI all these different versions of the question. If the AI gives the same answer to every single variation, the system accepts the result as reliable. If the AI's answers waver or contradict each other, the system rejects the output, flagging it as uncertain rather than risking a wrong diagnosis.

To test this idea, the researchers first had to create a way to generate these question variations automatically. They used a large language model, a type of advanced AI trained on vast amounts of text, to rewrite the questions found in existing medical datasets. They set strict rules for this process: the new questions had to keep the exact same meaning as the original, and they could not introduce any new facts that weren't already in the image or the original question. This ensured that the AI was being tested on its ability to understand the core concept, not on its ability to guess new information. They also checked the quality of these generated questions to ensure they were grammatically correct and truly equivalent to the originals. This process allowed them to build a much richer testing ground, where every single medical image was paired with dozens of different ways to ask about it, rather than just one.

The researchers then put this method to the test using two different medical AI models on a dataset called SLAKE, which contains thousands of medical images and questions. They created four different versions of this dataset to simulate real-world challenges. One version used only the new, varied questions to test how the models handled language changes. Another version used images that had been slightly altered with standard computer techniques, like changing the brightness or adding a bit of blur, to test visual robustness. They also combined these changes and even included images that were subtly modified to trick the AI, known as adversarial attacks. When they ran the models on these datasets without their new safety filter, the results were revealing. The models were surprisingly fragile when it came to language; even small changes in how a question was phrased caused their accuracy to drop significantly. In contrast, the models were somewhat more resilient to changes in the images themselves, though they still struggled when both text and image were altered at the same time.

The true power of the "Ask Twice, Look Twice" method emerged when the researchers applied their consistency filter. They set a rule that the AI would only be allowed to give an answer if it agreed with itself across a certain number of question variations. For instance, if they required the model to give the same answer to at least eleven different versions of a question, the system became much more trustworthy. On the dataset with varied language, the reliability of one of the top models jumped from roughly 77% to 89% when this strict rule was applied. On the dataset with visual changes, reliability rose from about 80% to 87%. This improvement came with a trade-off: because the system was now rejecting answers it wasn't sure about, the number of questions it actually answered dropped. In the language-varying dataset, the acceptance rate fell by about 29%, meaning the system chose to stay silent rather than guess. However, the answers it did provide were far more likely to be correct, and the uncertainty in its predictions dropped by about a quarter.

The study also highlighted a crucial difference in how these medical AI models handle different types of errors. The researchers found that the models were far more sensitive to changes in language than to changes in the images. Even when the images were made slightly blurry or had their contrast adjusted, the models performed relatively well. But when the questions were rephrased, the models often failed to recognize that the question was the same. This suggests that current medical AI systems rely heavily on the specific words used in a question rather than truly understanding the visual information in the image. The researchers noted that traditional image changes actually hurt performance more than the subtle, computer-generated attacks designed to fool the AI, a finding that challenges some assumptions about how these systems are vulnerable.

By using this consistency filter, the researchers demonstrated that it is possible to significantly boost the trustworthiness of medical AI without needing to train the models on new data or change their underlying architecture. The method acts as a gatekeeper, ensuring that the system only speaks up when it is confident. While this means the system will sometimes choose not to answer a question, the researchers argue that in a clinical setting, a silent system is far safer than one that confidently gives a wrong answer. The approach works across different types of models and different kinds of data, suggesting it could be a practical tool for improving safety in real-world hospitals. The study concludes that while these systems still need work to become fully robust, especially regarding how they handle different ways of asking questions, this simple "ask twice" strategy offers a powerful way to filter out uncertainty and ensure that the answers given to doctors are reliable.