🤖 AI

Knowledge Graph Augmented Large Language Models for Disease Prediction

This paper proposes a knowledge graph-guided chain-of-thought framework that fine-tunes lightweight large language models on electronic health records to achieve strong, clinically preferred disease prediction and zero-shot transfer performance with clear, temporally consistent rationales.

Original authors: Ruiyu Wang, Tuan Vinh, Ran Xu, Yuyin Zhou, Jiaying Lu, Francisco Pasquel, Mohammed K Ali, Carl Yang

Published 2026-08-04

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Ruiyu Wang, Tuan Vinh, Ran Xu, Yuyin Zhou, Jiaying Lu, Francisco Pasquel, Mohammed K Ali, Carl Yang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery, but instead of a crime scene, you are looking at a patient's medical history. This history is a massive, chaotic pile of notes, codes, and lab results called Electronic Health Records (EHRs). For a long time, computers have been good at looking at these piles and saying, "There is a 70% chance this patient will get sick." But they were terrible at explaining why. It was like a detective pointing at a suspect and saying, "I just know," without showing a single clue. This is frustrating for real doctors, who need to understand the logic to make life-or-death decisions.

Recently, a new kind of computer brain called a Large Language Model (LLM) arrived. These are like super-smart students who can read almost everything and write stories. They can explain their thinking step-by-step, a process called "Chain-of-Thought." However, these students sometimes make things up or get confused, especially when dealing with serious medical facts. To fix this, scientists are trying to teach these AI students to use a giant, pre-written encyclopedia of medical facts called a Knowledge Graph. Think of the Knowledge Graph as a map that shows exactly how diseases, genes, and drugs are connected. The big question is: Can we teach a smart AI to use this map to explain its medical predictions clearly and correctly, even when it hasn't seen millions of examples?

This paper introduces a clever new method to answer "yes." The researchers built a system that acts like a strict but helpful tutor for an AI student. Instead of letting the AI guess freely, they first use a massive medical map (PrimeKG) to find the specific clues that link a patient's current symptoms to a future disease. They then ask the AI to write a story explaining the prediction, but they force the AI to follow the path on the map. If the AI's story doesn't match the actual medical facts or the map's logic, the system throws that story away and tries again. This process creates a small, high-quality library of perfect examples.

The team tested this on a dataset of hospital records called MIMIC-III, focusing on predicting ten different diseases. They taught two lightweight AI models (LLaMA-3.1 and Gemma) using only tiny amounts of data—just 400 or 1,000 patient visits. The results were surprising: these small, carefully taught models performed competitively with, and in some cases slightly better than, traditional heavy-duty computer programs. They achieved a score (AUROC) between 0.66 and 0.70 and a macro-AUPR between 0.40 and 0.47, which is a strong result for such a small amount of training data, matching or exceeding strong classical baselines like XGBoost and Random Forest on specific metrics. Even more impressively, when they took these models and tested them on a completely different group of patients (the CRADLE cohort) without any extra training, the models' accuracy jumped from about 0.40–0.51 to 0.72–0.77. This suggests the models learned a general way of thinking about disease, not just memorized facts.

Finally, the researchers asked real doctors to review the AI's explanations. In a blind test, the doctors overwhelmingly preferred the explanations generated by the "map-guided" AI. They found these explanations clearer, more relevant, and medically sound. The doctors noted that the AI stopped making wild guesses and started organizing evidence logically, much like a real clinician would. The paper suggests that by anchoring AI reasoning to a solid medical map, we can create tools that not only predict disease but also explain their reasoning in a way that doctors can trust and use.

Technical Summary: Knowledge Graph Augmented Large Language Models for Disease Prediction

Problem Statement

Electronic Health Records (EHRs) contain longitudinal data suitable for clinical prediction, yet existing models often fail to provide patient-level reasoning that is actionable for clinicians. Classical models (e.g., logistic regression, tree ensembles) offer robustness and global interpretability but typically provide only coarse, post-hoc explanations (e.g., feature importance scores) that lack the step-by-step narrative required for specific clinical decisions. Conversely, while Large Language Models (LLMs) can generate Chain-of-Thought (CoT) narratives, their unconstrained application in medicine is hindered by a scarcity of high-quality, clinically validated reasoning datasets and the risk of hallucinated or medically incorrect intermediate steps. The core challenge is to enable LLMs to perform accurate, visit-level disease prediction while generating explanations that are factually grounded in biomedical knowledge and temporally consistent with patient history.

Methodology

The authors propose a Knowledge Graph (KG)-guided CoT framework designed to scaffold LLM reasoning using the structure of PrimeKG, a large-scale biomedical knowledge graph. The pipeline consists of four primary stages:

KG Entity Mapping:
- ICD-9 codes from MIMIC-III are aligned to PrimeKG nodes via a three-stage procedure:
  - Exact Match: Direct string matching between ICD-9 descriptions and KG node labels.
  - Similarity Match: Embedding-based retrieval (cosine similarity > 0.85) for non-exact matches.
  - LLM Filtering: GPT-4o validates or revises provisional mappings to ensure clinical correctness.
- This process yields 1,513 mapped feature nodes and 10 mapped disease nodes.
KG Relevance and Path Mining:
- For each target disease, the system identifies a compact set of relevant feature nodes ( $R_d$ ) using GPT-4o.
- Reasoning Paths: Shortest paths (bounded by $L=5$ hops) are extracted between relevant feature nodes and the target disease node within PrimeKG.
- Path Selection: GPT-4o filters these paths to retain only those that are clinically relevant and mechanistically sound, pruning spurious chains.
CoT Generation and Filtering:
- The LLM is prompted to generate CoT explanations ( $C_{t,d}$ ) for a specific visit $t$ , incorporating the observed ICD-9 features, the disease-specific relevance set, and the mined KG paths.
- Label-Consistency Filtering: Generated CoT traces are retained only if the explicit binary conclusion ( $\hat{y}$ ) derived from the reasoning matches the ground-truth outcome ( $y$ ). This creates a high-quality, KG-grounded supervision corpus.
Model Fine-Tuning:
- Lightweight open-weight models (LLaMA-3.1-Instruct-8B and Gemma-7B) are fine-tuned on this curated corpus.
- At inference, these models take ICD-9 features and disease-specific KG evidence as input to produce both a disease prediction and a clinician-style reasoning trace.

Key Contributions

KG-Guided CoT Framework: A novel method that integrates biomedical KG structure directly into the CoT generation process, moving beyond unconstrained free-form explanations to clinically grounded reasoning.
Data-Efficient Supervision: The construction of a compact, high-quality supervision corpus via label-consistency filtering allows small LLMs (8B parameters) to learn prognostic reasoning with limited data (400–1,000 samples).
Zero-Shot Transfer: The framework demonstrates the ability to transfer learned reasoning patterns to a distinct cohort (CRADLE) without retraining, maintaining high accuracy in forecasting cardiovascular events post-diabetes diagnosis.
Clinician-Centric Evaluation: A blinded human study validates that KG-guided explanations are preferred over baselines across clarity, relevance, and clinical correctness.

Experimental Results

The framework was evaluated on the MIMIC-III cohort (10 diseases mapped to PrimeKG) and the CRADLE cohort (Type 2 diabetes to cardiovascular events).

Performance on MIMIC-III:
- In low-data settings (400 and 1,000 index visits), KG-guided CoT fine-tuning improved the performance of LLaMA-3.1-8B and Gemma-7B, achieving results competitive with or superior to strong classical baselines (XGBoost, Random Forest, Logistic Regression).
- LLaMA-3.1 (1,000 samples): Achieved an AUROC of 0.6995 and macro-AUPR of 0.4685, slightly exceeding the best classical baseline (Logistic Regression: AUROC 0.6874, AUPR 0.4178; XGBoost: AUROC 0.6799).
- Gemma-7B: Showed similar improvements, with macro-AUPR increasing from 0.25 (original) to 0.4363 (KG-CoT tuned), though its AUROC (0.6609) remained slightly below the top classical baselines.
- Ablation: Removing KG guidance or filtering reduced performance, confirming that both the structural grounding and the label-consistency filter are critical.
Zero-Shot Transfer to CRADLE:
- Models fine-tuned on MIMIC-III were applied to CRADLE without retraining.
- Accuracy improved from 0.40–0.51 (untuned) to 0.72–0.77 (KG-CoT tuned).
- While AUROC and AUPR saw modest gains, the significant jump in accuracy suggests the models learned a more robust decision boundary that generalizes across cohorts.
Human Evaluation:
- In a blinded study with 115 cases, clinicians preferred the KG-guided model over the untuned baseline in 96.5% of cases for Clarity/Coherence, 94.8% for Coverage/Relevance, and 98.3% for Correctness/Soundness.
- Clinicians noted the KG-guided traces were better organized temporally, highlighted meaningful risk factors, and avoided unsupported leaps.

Significance and Claims

The paper claims that KG-anchored CoT supervision enables small, open-weight LLMs to learn data-efficient, clinically grounded prognostic reasoning. The significance lies in bridging the gap between accurate prediction and interpretable justification:

Reliability: By grounding reasoning in PrimeKG and filtering for label consistency, the system mitigates hallucinations and ensures medical plausibility.
Generalizability: The learned reasoning patterns transfer effectively to out-of-distribution cohorts (CRADLE), suggesting the models capture underlying disease mechanisms rather than dataset-specific artifacts.
Clinical Utility: The framework produces explanations that clinicians find clearer and more trustworthy, potentially facilitating the adoption of LLMs in clinical decision-support systems where transparency is paramount.

The authors modestly note that the approach depends on the coverage of PrimeKG and that future work should explore richer EHR views (notes, labs) and multi-visit contexts.