💬 NLP

Linking Knowledge to Care: Knowledge Graph-Augmented Medical Follow-Up Question Generation

This paper introduces KG-Followup, a knowledge graph-augmented large language model that leverages structured medical expertise to generate high-quality pre-diagnostic follow-up questions, outperforming state-of-the-art methods by 5% to 8% in recall.

Original authors: Liwen Sun, Xiang Yu, Ming Tan, Zhuohao Chen, Anqi Cheng, Ashutosh Joshi, Chenyan Xiong

Published 2026-03-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Liwen Sun, Xiang Yu, Ming Tan, Zhuohao Chen, Anqi Cheng, Ashutosh Joshi, Chenyan Xiong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery. Usually, you have a witness (the patient) who gives you a vague story: "I feel sick." A good detective knows that "I feel sick" isn't enough. You need to ask the right follow-up questions: "Did you eat anything strange?" "Do you have a fever?" "Where exactly does it hurt?"

If you ask the wrong questions, you might solve the wrong case. If you miss a crucial question, the patient might get the wrong treatment.

This paper introduces a new AI tool called KG-Followup that acts like a super-smart detective's assistant. Its job is to help doctors (or AI doctors) ask the perfect follow-up questions to figure out what's wrong with a patient.

Here is how it works, broken down into simple concepts:

1. The Problem: AI is Smart, but Sometimes "Hallucinates"

Large Language Models (LLMs) are like brilliant students who have read almost every book in the library. They are great at talking and writing. However, when it comes to medicine, they sometimes make mistakes because they rely only on what they "remember" from their training.

Think of an LLM without help as a detective who has read a textbook on medicine but has never actually walked the streets or looked at a real map. They might guess the answer, but they might miss a tiny, crucial clue that only a local expert would know.

2. The Solution: The "Medical Map" (Knowledge Graph)

The authors gave their AI a Knowledge Graph. Imagine this as a giant, super-detailed subway map of the human body.

The Stations: These are medical facts (e.g., "Stomach Pain," "Diarrhea," "Dehydration").
The Tracks: These are the connections between them (e.g., "Stomach Pain" often leads to "Dehydration").

When a patient says, "My stomach hurts," the AI doesn't just guess. It looks at the map, sees the tracks connecting "Stomach Pain" to other symptoms, and realizes, "Hey, I should ask about dehydration!"

3. How the AI Works (The Three-Step Detective Routine)

The paper describes a three-step process to generate these questions:

Step 1: The "First Glimpse" (Preliminary Questions)
The AI listens to the patient and asks basic questions based on its own memory. It's like a detective making an initial list of suspects.
- Analogy: "Okay, you have a headache. Do you have a fever?"
Step 2: The "Map Check" (EHR & KG Linking)
The AI takes the patient's words and matches them to the "Medical Map." It looks for hidden connections.
- Analogy: The detective checks the map and sees that "Headache" + "Recent Travel" = "Malaria Risk." The AI then asks, "Have you traveled anywhere recently?" This is a question the AI might have missed without the map.
Step 3: The "What-If" Game (DDX Reasoning)
Doctors often play a game of "What if?" They think, "Could this be the flu? Or could it be something worse?" The AI does the same. It creates a list of possible diseases and asks questions specifically designed to rule them out.
- Analogy: The detective thinks, "If it's a robbery, we need to check the windows. If it's a break-in, we check the back door." The AI asks questions to eliminate the "worst-case" and "best-case" scenarios.

4. The Secret Sauce: Learning from the "Hard Cases"

The most clever part of this system is Active In-Context Learning.
Imagine you are training a new detective. Instead of showing them easy cases, you show them the toughest cases where the clues were tricky. You say, "Look at this case where we almost missed the answer because we didn't ask about the patient's travel history. Remember that next time."

The AI does this by feeding itself examples of difficult patient stories where it previously struggled. This helps it get better at spotting the tricky clues.

5. The Result: A Sharper Detective

The researchers tested this system against other AI models.

The Old Way: Other AIs were like detectives guessing in the dark. They asked questions, but sometimes missed the big clues.
The New Way (KG-Followup): With the "Medical Map" and the "Hard Case" training, this AI found 5% to 8% more of the important questions that a human doctor would ask.

Why Does This Matter?

Saves Time: Doctors are overworked. If an AI can ask the right questions before the doctor walks into the room, the doctor can focus on the solution.
Saves Money: Fewer missed diagnoses mean fewer repeat visits and fewer unnecessary tests.
Better Care: Patients feel heard because the AI (and the doctor) asks the specific questions that matter to their unique situation.

In a nutshell: This paper teaches AI how to use a "Medical Map" and learn from its mistakes to ask better questions, turning a generic chatbot into a helpful, knowledgeable medical assistant.

1. Problem Statement

Clinical diagnosis relies heavily on the physician's ability to gather comprehensive information through effective follow-up questions. However, diagnostic errors often stem from incomplete history taking, and the manual generation of these questions is time-consuming. While Large Language Models (LLMs) show promise in automating this process, they suffer from limited domain-specific knowledge, leading to:

Failure to identify critical information gaps across diverse symptoms.
Generation of generic or clinically irrelevant questions.
Inability to emulate the structured reasoning (e.g., differential diagnosis) used by human physicians.

The core challenge is to automate the generation of clinically relevant, comprehensive, and context-aware follow-up questions that reduce physician workload without compromising diagnostic accuracy.

2. Methodology: KG-Followup

The authors propose KG-Followup, a framework that augments LLMs with a structured Medical Knowledge Graph (KG) and active in-context learning (ICL). The system operates through a multi-stage pipeline:

A. Preliminary Clarification

The LLM first generates a preliminary set of questions ( $\hat{Q}_{pre}$ ) based solely on the patient conversation and its internal knowledge, simulating an initial consultation.

B. EHR-Guided Question Generation (Symptom Linking)

To ground the generation in specific patient data:

Entity Extraction: The LLM extracts key medical entities ( $E$ ) from the conversation to serve as a proxy Electronic Health Record (EHR).
KG Linking & Subgraph Construction: Entities are linked to KG nodes. A breadth-first search builds entity-specific subgraphs, which are intersected to find shared, clinically relevant concepts.
Ranking & Generation: The LLM ranks these intersected concepts by relevance to the specific case. The top- $k$ concepts are used to generate enriched follow-up questions ( $\hat{Q}_{ehr-kg}$ ).

C. DDX-Guided Question Generation (Diagnosis Reasoning)

To mimic physician reasoning for narrowing down diagnoses:

Differential Diagnosis (DDX): The LLM generates a set of potential diagnoses ( $D$ ), including best-case and worst-case scenarios.
Reasoning Path Search: For each source entity–target diagnosis pair, the system samples the $k_2$ shortest reasoning paths in the KG.
Path Selection: The LLM selects the single most relevant path ( $\hat{P}_{i,j}$ ) based on patient context, pruning irrelevant paths.
Targeted Generation: Questions are generated to eliminate specific diagnostic possibilities using these reasoning paths as context ( $\hat{Q}_{ddx-kg}$ ).

D. Active In-Context Learning (ICL) via Hard Cases

To handle edge cases where the KG cannot be traversed (e.g., missing symptoms or unmapped entities):

The system identifies "hard queries" where KG traversal fails.
These difficult cases are treated as active ICL examples ( $T$ ).
The LLM is prompted with these hard examples alongside the current conversation to guide the generation of robust follow-up questions, ensuring coverage beyond explicit symptom queries.

E. Question Consolidation

To prevent redundancy:

All generated questions are embedded using a medical encoder.
K-means clustering groups similar questions.
An LLM merges overlapping clusters and removes duplicates, yielding a concise, controlled set of final questions ( $\hat{Q}$ ).

3. Key Contributions

KG-Followup Framework: A novel architecture integrating EHR-guided concept retrieval, DDX-guided reasoning paths, and KG-informed active ICL to generate high-quality medical follow-up questions.
ClinicalInquiryBench: A new, large-scale benchmark derived from 5,000 HealthBench cases. Unlike previous datasets, it is filtered for "context-seeking" instances and annotated with question weights reflecting diagnostic importance, allowing for a more rigorous evaluation of AI diagnostic reasoning.
State-of-the-Art Performance: Demonstrated that KG-augmented methods significantly outperform standard LLMs and existing baselines, particularly in recall metrics.

4. Experimental Results

The model was evaluated on ClinicalInquiryBench and FollowupBench using Weighted Recall as the primary metric (evaluated via LLM-as-a-judge).

Performance Gains: KG-Followup outperformed state-of-the-art methods (including FollowupQ and Zero-Shot baselines) by 5%–8% in recall.
- On ClinicalInquiryBench, it achieved 70%–75% recall (depending on the backbone).
- On FollowupBench, it achieved 77%–82% recall.
Ablation Studies:
- DDX vs. EHR: The DDX-guided module provided larger performance gains than the EHR-guided module.
- Active ICL: Using KG-informed hard cases for ICL outperformed random ICL selection, confirming the value of targeted example selection.
- Efficiency: While increasing the number of generated questions generally improves recall, KG-Followup achieves comparable results to methods generating 40+ questions while using significantly fewer (controlled sets), enhancing interaction efficiency.
Backbone Analysis: The framework showed that stronger backbones (e.g., Claude Sonnet) leverage external KG knowledge more effectively than weaker ones (e.g., Claude Haiku).

5. Significance and Limitations

Significance:

Clinical Efficiency: Automates the information-gathering phase, potentially reducing physician workload, referral rates, and healthcare costs.
Diagnostic Safety: By integrating structured medical knowledge, the system reduces the risk of missing critical symptoms or making premature diagnoses, a common failure mode of pure LLMs.
Benchmarking: Establishes a new standard for evaluating AI in clinical reasoning through the weighted, context-aware ClinicalInquiryBench.

Limitations:

KG Dependency: The system relies on pre-constructed, high-quality biomedical knowledge graphs (e.g., PrimeKG). If the KG is incomplete, outdated, or lacks coverage for specific domains, performance may degrade.
Real-World Deployment: The benchmarks are offline. Real-world deployment faces challenges such as noisy EHR data, diverse clinical workflows, and the need for real-time integration with clinical decision support systems.
Ethical Risks: Potential biases in the KG or LLM could influence question relevance. Human oversight remains critical to ensure patient safety and fairness.

In conclusion, the paper presents a robust approach to bridging the gap between the generative capabilities of LLMs and the structured reasoning required in clinical medicine, offering a scalable solution for improving pre-diagnostic assessment.