A knowledge-guided agentic framework for mitigating patient-context ambiguity in health queries
This paper introduces a knowledge-guided agentic framework that mitigates patient-context ambiguity in health queries by using a knowledge graph to identify missing information and ask targeted follow-up questions, thereby significantly improving the accuracy of downstream language models in diagnosis retrieval and dietary safety classification compared to direct answering.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the quiet space between a person's worry and a doctor's advice, a new kind of conversation is taking place. For years, people have turned to digital assistants and chatbots to ask about their health, typing short questions like "What should I take for a headache?" or "Is this food safe for me?" These tools are built on vast libraries of medical knowledge, capable of reciting facts about diseases and treatments. Yet, a fundamental gap remains. A computer can know everything about headaches, but it cannot know the specific person asking the question. It does not know if the person is pregnant, if they take blood thinners, or if they have a liver condition. Without these hidden details, the most medically accurate answer might be the wrong one for that specific individual. This is not a failure of the computer's memory, but a missing piece of the puzzle that only the patient holds.
Researchers have long understood that language can be unclear, but this study focuses on a different kind of confusion. It is not that the words are hard to understand; it is that the story is incomplete. Imagine a detective trying to solve a crime with only half the clues. The detective might guess, but the guess could be dangerous. In healthcare, guessing is risky. The team behind this research, led by Mahyar Abbasian and colleagues, set out to see if a computer could learn to ask the right questions before it tries to give an answer. They wanted to know if an intermediary step—where the system actively seeks out the missing personal details—could make the final advice safer and more accurate, without needing to retrain the powerful computers that generate the text.
To test this, the researchers built a framework that sits between the person asking the question and the large language model that provides the answer. Think of this framework as a skilled triage nurse who listens to the initial query and then consults a structured map of medical knowledge. This map, known as a knowledge graph, helps the system understand what information is missing. If a patient asks about a headache, the system looks at its map and realizes that the answer depends on factors like pregnancy or medication use. Instead of guessing, the system asks a targeted follow-up question: "Are you currently pregnant?" or "Do you take blood-thinning medication?" Once the patient answers, the system combines that new information with the original question and passes the complete, clarified story to the language model. The model then generates a response based on a full picture, rather than a partial one.
The team tested this approach using two distinct scenarios: diagnosing illnesses from a list of symptoms and determining if a specific food is safe for a person with a particular health condition. They created thousands of test cases where crucial information was deliberately hidden from the initial question. In the diagnosis tests, they used 1,034 synthetic patient records where about half of the symptoms were hidden. In the food safety tests, they used 487 queries where the patient's health condition was omitted. They compared three methods: letting the computer answer the incomplete question directly, having the computer rephrase the question without adding new facts, and using their new system to ask for the missing details first.
The results were striking. When the computer tried to answer the incomplete questions directly, it was often wrong. For the diagnosis task, the direct approach got the exact correct answer less than 12 percent of the time, even with the most advanced models. When the researchers used their clarification system, the accuracy jumped dramatically. Across five different language models, the system improved the exact correct diagnosis rate by at least 57 percentage points. In some cases, the improvement was even larger, pushing accuracy from near zero to over 70 percent. The system did not just get the broad category right; it identified the specific illness much more reliably. Similarly, for the food safety task, the clarification system helped the models correctly identify unsafe foods while avoiding unnecessary restrictions, achieving the highest balance of safety and accuracy for four out of the five models tested.
Beyond just getting the right answer, the study found that this method made the computers more consistent. When the same question was asked repeatedly, the direct approach often gave different answers each time, sometimes suggesting a safe food one moment and an unsafe one the next. The clarification system, however, produced the same decision every time once the missing context was gathered. This stability is crucial in healthcare, where a user needs to trust that the advice will not change randomly. The researchers also observed that the system worked well across different types of language models, suggesting that the benefit comes from the process of gathering information rather than from the specific computer program used to generate the text.
The researchers were careful to note the limits of their work. The tests were conducted using synthetic data and simulated answers, not real conversations with actual patients. In the real world, people might misunderstand a question, give vague answers, or forget to mention a key detail. The study showed that the method works when the missing information can be reliably recovered, but it does not yet solve the messiness of human conversation. Furthermore, the system relies on the patient being able to report their own symptoms and conditions; it cannot yet pull data from wearable devices or medical records automatically.
Despite these limitations, the findings offer a clear path forward. The study suggests that the biggest barrier to accurate health advice from computers is not a lack of medical knowledge, but a lack of context. By treating clarification as a necessary step to gather missing facts rather than just a way to rephrase words, these systems can become significantly more reliable. The research demonstrates that an intermediary agent, acting as a bridge to fill in the gaps of a patient's story, can transform a potentially dangerous guess into a well-supported recommendation. This approach does not require changing the underlying technology of the language models themselves; it simply changes how they are asked to think, ensuring they have the full story before they begin to speak.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.