💬 NLP

The Provenance Gap in Clinical AI: Evidence-Traceable Temporal Knowledge Graphs for Rare Disease Reasoning

This paper introduces HEG-TKG, a system that utilizes hierarchical evidence-grounded temporal knowledge graphs to overcome the "Provenance Gap" in clinical AI by ensuring 100% verifiable citations and high resistance to errors for rare disease reasoning, addressing the tendency of frontier large language models to fabricate references.

Original authors: Md Shamim Ahmed, Maja Dusanic, Moritz Nikolai Kirschner, Elisabeth Nyoungui, Jana Zschüntzsch, Lukas Galke Poech, Richard Röttger

Published 2026-04-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Md Shamim Ahmed, Maja Dusanic, Moritz Nikolai Kirschner, Elisabeth Nyoungui, Jana Zschüntzsch, Lukas Galke Poech, Richard Röttger

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Problem: The "Magic Trick" of AI Doctors

Imagine you go to a doctor, and they give you a diagnosis. But instead of saying, "I found this in your blood test," they say, "I just know this because I read a book once."

That is essentially what current "Frontier" Large Language Models (LLMs) do when they act as medical experts. They are incredibly smart and can often give the right answer. However, when you ask them, "Where did you get that information?" they often make up fake references.

The authors call this the "Provenance Gap."

Provenance means "where something came from."
The Gap is the space between the AI's answer and the proof that the answer is true.

The Analogy:
Think of a student taking a history exam.

The AI writes a perfect essay.
The Teacher asks, "Show me your sources."
The AI hands over a list of books that look real but don't actually exist, or books that are about baking instead of history.
The Result: The teacher can't trust the essay because they can't verify the facts. In medicine, this is dangerous. If a doctor trusts a fake citation, a patient could get the wrong treatment.

The Experiment: Testing the "Magic"

The researchers tested five of the smartest AI models on three pairs of rare, confusing diseases (like distinguishing between two types of muscle weakness that look identical at first).

The Test: They asked the AI to diagnose patients and provide citations (proof) from medical journals.
The Result:
- When not asked to cite, the AIs gave zero real proof.
- When forced to cite, the best AI only got 15% of the citations right. The rest were "hallucinations"—real-sounding numbers pointing to the wrong topics or fake papers.
- The Shock: Even if you ask the AI to "be honest," it still lies about the sources because it's trying to guess what a citation looks like, not actually finding one.

The Solution: HEG-TKG (The "Annotated Map")

To fix this, the team built a new system called HEG-TKG. Instead of letting the AI guess, they built a massive, structured "map" of medical knowledge before the AI ever starts talking.

The Analogy: The Librarian vs. The Improviser

Old AI (The Improviser): It's like a brilliant actor on stage who makes up a story on the spot. It sounds convincing, but you can't check the script.
HEG-TKG (The Annotated Librarian): Imagine a librarian who has a giant, organized filing cabinet. Every single fact in the cabinet is glued to a specific page number in a real medical journal.
- When you ask a question, the librarian doesn't guess. They pull the exact file, read the page, and hand you the page number.
- The "Time" Feature: This system is special because it doesn't just know what happens; it knows when it happens. It tracks the timeline of a disease (e.g., "Symptom A happens at age 3, Symptom B happens at age 10"). Most other medical maps are static; this one is a moving timeline.

How It Works (The Three-Layer Cake)

The system organizes evidence into three "tiers" of trust, like a quality control checklist:

Gold Tier: Facts confirmed by top medical guidelines and multiple sources. (High confidence).
Silver Tier: Facts found in multiple different studies. (Good confidence).
Bronze Tier: Facts found in just one study. (Needs a double-check).

When the AI generates an answer, it pulls from this "Gold/Silver/Bronze" map. Every claim it makes is tied to a real, verifiable ID number (a PMID) that a human doctor can look up in seconds.

The Results: Trustworthy vs. Trusting

The researchers compared three groups:

Vanilla AI: Just the raw AI guessing.
Guideline-RAG: The AI reading raw text documents (like a PDF) without a map.
HEG-TKG: The AI using the structured, time-aware map.

The Findings:

Accuracy: All three groups were about equally good at giving the right medical advice.
Trust: The "Vanilla" and "Raw Text" groups gave zero verifiable proof.
HEG-TKG: Achieved 100% verifiability. Every single claim could be traced back to a real paper.
The "Fake" Test: The researchers tried to trick the system by feeding it fake medical errors. The system caught 100% of the errors because the "trace" (the citation) showed the error immediately.

Why This Matters for You

Safety: Doctors can't use AI if they can't check the work. This system lets a doctor look at an AI's answer and say, "Okay, I see the source. I can verify this."
Rare Diseases: For rare conditions, a doctor might have never seen a case before. This system acts like a super-powered, verified encyclopedia that tells them exactly what to look for and when to look for it.
Privacy: The system can run on a hospital's own computer (on-premise), meaning patient data never has to leave the building to be processed by a cloud AI.

The Bottom Line

The paper argues that accuracy isn't enough. An AI can be right but still be useless if you can't prove why it's right.

The "Provenance Gap" is the missing link in medical AI. By building a system that forces the AI to show its work using a structured, time-aware map of real medical facts, the researchers have created a tool that doctors can actually trust. It turns the AI from a "magic trick" into a "verified research assistant."

1. Problem Statement: The Provenance Gap

The paper identifies a critical failure in frontier Large Language Models (LLMs) applied to clinical decision support, termed the Provenance Gap. While LLMs can generate clinically accurate differential diagnoses, they frequently fabricate citations (hallucinate PMIDs) or provide citations that resolve to real papers in unrelated fields.

The Issue: Clinicians cannot verify AI-generated claims without traceable evidence. In rare diseases (e.g., Myasthenia Gravis vs. Lambert-Eaton), where non-specialists often make initial diagnoses, the inability to verify sources leads to mistrust and potential patient harm (e.g., incorrect immunotherapy).
Current Limitations:
- Vanilla LLMs: Produce zero verifiable citations in "vanilla" mode.
- Prompt Engineering: Explicitly asking models to cite sources ("citation-prompting") fails; models generate plausible-looking but often irrelevant or non-existent PMIDs (e.g., 78.9% of citations were real papers in unrelated fields).
- Standard RAG: Retrieval-Augmented Generation using raw text chunks (Guideline-RAG) provides source text but lacks structured citation metadata, resulting in zero verifiable citations.
- LLM Judges: Automated LLM evaluators cannot distinguish fabricated citations from real ones without external audit data, leading to false confidence in unverified outputs.

2. Methodology: HEG-TKG System

The authors propose HEG-TKG (Hierarchical Evidence-Grounded Temporal Knowledge Graphs), a system designed to ground clinical reasoning in verified, structured evidence with temporal context.

A. Knowledge Graph Architecture

HEG-TKG employs a two-tier architecture built from 4,512 PubMed records and curated sources:

Tier 1 (Curated Backbone): Extracted from authoritative sources (GeneReviews, OMIM, Orphanet, CDC guidelines). These edges are "protected" and carry the highest confidence.
Tier 2 (Literature Extraction): Multi-LLM extraction pipeline (using Claude Haiku, GPT-4.1-mini, etc.) processes PubMed abstracts to extract subject-predicate-object triplets.
- Consensus Voting: Triplets are deduplicated and assigned quality tiers based on cross-model or cross-document consensus.
- Quality Tiers:
  - GOLD: Cross-tier confirmed (Tier 2 confirms Tier 1) or guideline-backed.
  - SILVER: Multi-model or multi-document consensus.
  - BRONZE: Single model, single source.
Temporal Anchoring: A novel feature where disease milestones are mapped to ISO 8601 durations (e.g., "loss of ambulation at P9Y-P13Y"). This captures the when of disease progression, which static graphs miss.

B. Clinical Inference Pipeline

Retrieval: Given a clinical scenario, the system queries the Neo4j graph using Cypher to retrieve relevant subgraphs (differential features, temporal trajectories, treatment rationales).
Synthesis: An LLM (GPT-4.1 for evaluation, open-source models for deployment) synthesizes the output. Crucially, the prompt includes the structured evidence block with inline PMID tags and quality tiers.
Deployment: The system supports on-premise deployment using open-source models (Qwen, Gemma) behind institutional firewalls, ensuring Patient Health Information (PHI) never leaves the local infrastructure.

C. Evaluation Design

The study utilized a three-arm comparison across 36 clinician-validated scenarios for three rare disease pairs (MG/LEMS, DMD/BMD, CIDP/GBS):

Vanilla: Parametric knowledge only.
Guideline-RAG: Raw text retrieval of the same sources used in HEG-TKG.
HEG-TKG: Structured graph retrieval with citations.

Evaluation included:

Automated Metrics: Evidence Traceability Score (ETS), Clinical Feature Coverage, and the Provenance Gap metric.
LLM Judges: Blind (v1) vs. Citation-Aware (v2) evaluation using PubMed audit reports.
Clinician Evaluation: Independent scoring by neurologists on verifiability, actionability, and safety.
Counterfactual Testing: Injecting 15 clinically incorrect statements to test error resistance and detectability.

3. Key Contributions

Formal Definition of Provenance Gap: Defined as the gap between clinical feature coverage and reliable evidence traceability. Empirically measured across five frontier LLMs.
HEG-TKG Architecture: A temporal knowledge graph with 5,481 nodes, 6,316 edges, and 1,280 temporal anchors, featuring a quality-tiered evidence system (GOLD/SILVER/BRONZE).
LLM Judge Limitation Discovery: Demonstrated that LLM judges cannot assess citation validity without external ground truth (PubMed audit data). In blind tests, they rated fabricated citations as valid; with audit data, their scores for unverified systems collapsed.
Counterfactual Safety: Showed that while LLMs may occasionally defer to injected errors (13% failure rate), the system's traceability allows 100% of errors to be detected via citation audit.
Privacy-Preserving Deployment: Validated that the system can run on-premise with open-source models, keeping patient data within institutional control.

4. Key Results

Citation Verifiability:
- Vanilla/Citation-Prompted LLMs: 0% relevant PMIDs in vanilla mode; best performance (Claude Opus) achieved only 15.3% relevant PMIDs when prompted, with 78.9% being real but irrelevant papers.
- HEG-TKG: Achieved 100% relevant PMIDs. Every output claim is traceable to an average of 181 knowledge graph edges per scenario.
Provenance Gap Reduction:
- HEG-TKG reduced the Provenance Gap by half compared to baselines (0.366 vs. 0.743 for Vanilla) while maintaining comparable clinical feature coverage (71.7% vs. 74.3%).
Clinical Evaluation:
- Verifiability (D1): Clinicians rated HEG-TKG significantly higher than Vanilla ( $\Delta = +1.64$ , $p < 0.001$ , Cohen's $d = 1.81$ ).
- Actionability (D2): HEG-TKG showed a moderate advantage ( $\Delta = +0.86$ ).
- Safety/Completeness: No degradation in safety or completeness scores.
Temporal Reasoning: HEG-TKG produced 577 time-anchored claims (e.g., specific age windows for symptom onset), whereas Vanilla LLMs produced zero.
Error Detection: In counterfactual experiments, 80% of injected errors were resisted by the model's parametric knowledge. The remaining 20% were 100% detectable via the citation trace, allowing a clinician to identify the specific flawed source.

5. Significance and Impact

Trust in Clinical AI: The paper argues that accuracy alone is insufficient for clinical adoption; verifiability is the prerequisite for trust. HEG-TKG provides a mechanism for clinicians to audit AI reasoning in seconds rather than hours.
Beyond "Black Box" RAG: The study proves that simply retrieving more text (Guideline-RAG) does not solve the provenance problem. Structure (graph topology, metadata, quality tiers) is the key to traceability.
Temporal Reasoning: By anchoring evidence to disease trajectories (e.g., "Gowers' sign at age 3-5"), the system addresses a critical gap in existing biomedical knowledge graphs, enabling better longitudinal care planning.
Regulatory Compliance: The system addresses requirements from the EU AI Act and FDA guidance regarding independent verification and transparency. The on-premise deployment model ensures compliance with GDPR and HIPAA.
Methodological Shift: The findings suggest that future clinical AI benchmarks must include external citation verification (e.g., PubMed audits) rather than relying solely on LLM-as-judge evaluations, which are prone to hallucination detection failures.

In conclusion, HEG-TKG demonstrates that bridging the Provenance Gap requires a shift from parametric generation to evidence-grounded, temporally structured reasoning, enabling safe, verifiable, and actionable AI for rare disease diagnosis.