RiTeK: A Dataset for Large Language Models Complex Reasoning over Textual Knowledge Graphs in Medicine

This paper introduces RiTeK, a comprehensive benchmark dataset designed to evaluate and improve Large Language Models' complex reasoning capabilities over medical Textual Knowledge Graphs by addressing current limitations in data scarcity, topological expressiveness, and retrieval performance.

The Gist

Imagine you are trying to solve a very tricky medical mystery. You aren't just asking, "What is a headache?" You are asking, "Which organ, which is responsible for moving blood between a mother and baby, gets messed up when the baby is in distress, and what specific symptoms does that cause?"

To answer this, a computer needs to do two things at once:

1. Navigate a map: It needs to follow a path through a giant web of medical facts (a Knowledge Graph) to find the right connections.
2. Read the fine print: It needs to read the detailed descriptions attached to those facts to make sure it's not picking the wrong thing.

This paper introduces RiTeK, a new tool designed to test if Artificial Intelligence (AI) can actually do this difficult job.

Here is the breakdown of the paper using simple analogies:

1. The Problem: The "Empty Map" and the "Vague Question"

Currently, AI models are great at chatting, but they struggle when asked complex medical questions that require digging through structured data.

• The Old Way: Existing datasets were like giving a detective a map with only two or three streets and asking them to find a hidden treasure. It was too easy and didn't reflect real life.
• The Missing Piece: Most medical maps (Knowledge Graphs) only show the names of things (like "Heart" or "Disease") but lack the story (the text description of what they do). It's like having a phone book with names but no addresses or descriptions.

2. The Solution: Building "RiTeK" (The Ultimate Training Ground)

The authors built a massive new dataset called RiTeK. Think of this as a gym for AI brains, specifically designed to train them on complex medical reasoning.

• The Map (Textual Knowledge Graph): They built two huge maps using real medical data. But unlike old maps, every stop on the map has a "biography" attached to it. So, the AI doesn't just see "Placenta"; it sees "Placenta: An organ that connects the fetus to the uterine wall..."
• The Questions (The Queries): They didn't just write simple questions. They simulated three types of people asking questions:
  • The Patient: "My baby is in distress, what's wrong with the blood flow?"
  • The Doctor: "Which tissue function is compromised in fetal distress?"
  • The Scientist: "Analyze the relationship between X and Y regarding Z."
• The Quality Control: Before releasing this gym, they hired real medical experts (doctors and scientists) to grade the questions. They made sure the questions sounded natural and were medically accurate, not just robotic nonsense.

3. The Stress Test: Putting AI to the Work

Once the gym was built, the authors put 11 different AI retrieval systems (the "athletes") through a rigorous test to see how well they could answer these complex questions.

The Results were surprising (and a bit scary for AI):

• The "Smart" AI (LLMs) got lost: Even the most advanced AI models (like GPT-4) struggled. When asked to follow a complex path through the medical map, they often got confused or made things up (hallucinations).
• The "Search Engine" AI struggled too: Systems designed to search through graphs performed better than pure chatbots, but they still missed the mark on the hardest questions.
• The "Hybrid" Approach won (mostly): The best performers were systems that combined the AI's brain with a structured search method. However, even the winners only got about 30-50% of the answers perfectly right.

4. The Takeaway: We Need Better Tools

The paper concludes that while AI is getting smarter, it still isn't ready to be a fully autonomous medical detective for complex cases.

• The Analogy: Imagine asking a GPS to drive you through a city with no street signs, only vague descriptions of buildings. The GPS might guess the right turn, but it's likely to get you lost.
• The Future: We need to build better "GPS systems" for medical data. These systems need to be able to read the "biographies" of the data points and follow the complex, winding roads of medical relationships simultaneously.

In short: RiTeK is a new, very difficult test that proves current AI is still a bit clumsy when navigating complex medical knowledge. It sets a new standard for what we expect from medical AI in the future: not just knowing facts, but understanding the deep, messy, and detailed connections between them.

Technical Summary

1. Problem Statement

The paper addresses the critical challenge of answering complex, real-world medical questions using Large Language Models (LLMs). While LLMs excel in natural language processing, they struggle with:

• Complex Relational Reasoning: Medical queries often require multi-hop reasoning across structured data (e.g., "Which organ function affected by Fetal Distress circulates maternal blood?").
• Lack of Specialized Datasets: Existing datasets for Textual Knowledge Graphs (TKGs) are limited. They typically feature:
  • Over-simplified reasoning paths (1-2 hops).
  • Lack of diverse topological structures.
  • Absence of rich textual descriptions for nodes, which limits semantic expressiveness.
  • Inability to handle complex constraints and interdependent conditions found in real clinical scenarios.

The core problem is the scarcity of high-quality benchmarks that integrate structured relational data (triples) with unstructured textual descriptions to evaluate LLMs' ability to reason over semi-structured medical knowledge.

2. Methodology

A. Dataset Construction: RiTeK

The authors introduce RiTeK, a large-scale dataset designed for complex reasoning over medical TKGs. It consists of two subsets based on existing medical graphs: RiTeK-PharmKG (pharmaceutical focus) and RiTeK-ADint (non-pharmacological interventions/lifestyle).

Key Construction Steps:

1. Knowledge Graph Enrichment: The authors constructed TKGs by integrating data from PharmKG and ADint with external databases (Ensembl, UMLS, Mondo Disease Ontology) to ensure rich textual descriptions for every entity.
2. Topological Structure Simulation: Unlike previous datasets limited to 2-3 hops, RiTeK utilizes six distinct topological structures (including multi-hop and constrained multi-hop patterns) to mimic complex medical queries.
3. Query Synthesis Pipeline:
   • Relational Template Construction: Medical experts designed 68 relational templates based on the 6 topological structures (e.g., Antibiotic $\to$ causes $\to$ Pathologic Function).
   • Textual Property Extraction: For candidate answers, GPT-4 extracts relevant textual properties from entity documents to form the "ground truth" context.
   • Query Generation: LLMs (GPT-4) synthesize natural-sounding queries by combining relational templates and extracted textual properties. The queries simulate three personas: Medical Scientists, Doctors, and Patients.
   • Filtering & Validation: Additional candidate answers are filtered using multiple LLMs to ensure only valid entities satisfying the textual constraints remain.
4. Expert Evaluation: Four medical experts evaluated 1,000 synthetic queries on Naturalness, Diversity, and Practicality. The dataset achieved high scores (e.g., >99% Acceptable rates for Naturalness).

Dataset Statistics:

• RiTeK-PharmKG: 10,235 queries, 6 topological structures, 68 relational templates.
• RiTeK-ADint: 5,322 queries, 6 topological structures, 58 relational templates.
• Coverage: High node textual coverage (95.61% for PharmKG, 36.73% for ADint) compared to prior benchmarks like STaRK.

B. Experimental Setup

The authors evaluated 11 representative retrieval models across three datasets (RiTeK-PharmKG, RiTeK-ADint, and STaRK-Prime) under three settings:

1. Zero-Shot: Direct prompting.
2. Few-Shot: Providing examples.
3. Supervised Fine-Tuning: Training models on the dataset.

Models Evaluated:

• LLM-only: GPT-4, Chain-of-Thought (CoT), Tree-of-Thought (ToT), Graph-of-Thought (GoT).
• Retrieval-Augmented: Random Walk, Monte Carlo Tree Search (MCTS), Think-on-Graph (TOG), G-retriever, KAR (Knowledge-aware Query Expansion).
• Supervised: G-retriever, GCR (Knowledge-aware query expansion), GNN-RAG.

Metrics: Exact Match (EM) and ROUGE-1 (Precision, Recall, F1).

3. Key Contributions

1. RiTeK Dataset: The first large-scale benchmark specifically for complex reasoning over medical Textual Knowledge Graphs. It uniquely combines diverse topological structures with rich textual node descriptions and complex constraints.
2. Rigorous Construction: A novel pipeline involving medical experts for template design and validation, ensuring the queries reflect real-world clinical complexity (multi-hop, constrained reasoning).
3. Comprehensive Benchmarking: A systematic evaluation of 11 retrieval approaches, revealing significant gaps in current LLM capabilities when handling semi-structured medical data.
4. Insight into Retrieval Limitations: The study highlights that current methods struggle with fine-grained relational understanding and attribute constraints, necessitating new retrieval strategies tailored for medical TKGs.

4. Results and Analysis

Performance Overview

• Baseline Struggles: Even advanced LLMs (GPT-4) and reasoning frameworks (CoT, ToT, GoT) performed poorly on RiTeK in zero-shot settings (e.g., GPT-4 F1 $\approx$ 11% on RiTeK-PharmKG). This indicates that internal LLM knowledge is insufficient for precise relational reasoning without external structured retrieval.
• Retrieval-Augmented Superiority: Models incorporating retrieval (KAR, TOG, G-retriever) significantly outperformed LLM-only baselines.
  • KAR (Knowledge-aware Query Expansion) showed strong performance in zero/few-shot settings by effectively combining textual semantics with KG relations.
  • TOG (Think-on-Graph) excelled in few-shot scenarios, leveraging demonstrations to guide beam search over the graph.
• Supervised Learning:
  • GCR achieved the best overall performance in supervised settings (e.g., 57.28 ROUGE-1 F1 on RiTeK-ADint), demonstrating the value of KG-grounded training.
  • GNN-RAG showed strong path retrieval capabilities but sometimes missed complex, indirect reasoning paths due to reliance on shortest paths.

Key Findings

1. Reasoning vs. Retrieval: Pure reasoning scaffolds (ToT/GoT) fail without access to external structured knowledge. Effective medical QA requires retrieval-augmented generation (RAG) that can navigate complex graph topologies.
2. Impact of Backbone LLMs: The choice of the underlying LLM (e.g., Llama 3.1, Biomixtral) significantly impacts performance. Biomixtral 7b combined with retrieval strategies often yielded the highest recall and F1 scores.
3. Complexity Gap: Current models struggle with queries involving answer attribute constraints (e.g., filtering entities based on specific textual descriptions embedded in the query).
4. Case Study: In a qualitative analysis, baseline models frequently hallucinated or followed incorrect reasoning chains for rare biomedical associations (e.g., confusing Alzheimer's with Schizophrenia), whereas models with precise retrieval (like GCR) maintained higher fidelity to ground-truth paths.

5. Significance

• Advancing Medical AI: RiTeK sets a new standard for evaluating AI in healthcare, moving beyond simple fact retrieval to complex, multi-hop diagnostic reasoning.
• Identifying Research Gaps: The paper clearly identifies that current retrieval systems are ill-equipped for the semi-structured nature of medical data, where textual descriptions and relational constraints are equally critical.
• Future Directions: The findings call for the development of retrieval systems that can better handle complex topological structures, long-tail medical concepts, and multi-modal constraints to improve diagnostic accuracy and treatment planning.

Limitations: The current dataset focuses on single topic entities. Future work should explore multi-entity queries and incorporate additional modalities (e.g., medical images) for a more robust retrieval system.