Technical Summary: EpiGraph

Problem Statement

Epilepsy diagnosis and treatment require evidence-intensive reasoning that integrates heterogeneous clinical knowledge, including biosignal patterns (EEG), genetic mechanisms, pharmacogenomics, and treatment strategies. Current challenges include:

1. Knowledge Fragmentation: Existing biomedical knowledge graphs (KGs) are often disease-agnostic or designed for semantic annotation rather than complex, multi-hop clinical reasoning specific to epilepsy.
2. Reasoning Gaps: Large Language Models (LLMs) struggle with expert-level evidence curation and reasoning in realistic epilepsy settings, particularly when tasks require integrating genetic variants with treatment contraindications or interpreting EEG reports against syndrome-specific knowledge.
3. Evaluation Deficits: Existing benchmarks focus on narrow, contrived tasks that do not reflect the full spectrum of epilepsy clinical practice, such as deep research planning or the integration of EEG data with pharmacogenomic rules.

Methodology

The authors propose EPIGRAPH, a unified framework comprising two core components: EPIKG (a specialized knowledge graph) and EPIBENCH (a multi-task evaluation benchmark).

1. EPIKG: Epilepsy Knowledge Graph

EPIKG is constructed via a structured evidence-to-graph pipeline:

• Schema Definition: The graph organizes entities into five clinical layers: (1) Syndromes, (2) Diagnostics (EEG/biomarkers), (3) Genes, (4) Treatments, and (5) Outcomes. It defines 1,370 cross-layer relation types (e.g., caused_by_gene, treated_with, contraindicated_with).
• Data Sources: The schema is grounded in seven authoritative resources (ILAE 2022, MeSH, OMIM, ChEBI, HPO, AES 2024, UMLS).
• Evidence Extraction: The authors processed over 120,000 PubMed/PMC publications. A two-stage screening process (LLM-assisted filtering followed by domain expert adjudication) yielded 48,166 peer-reviewed papers for extraction.
• Construction: A hybrid extraction pipeline was used:
  • Rule-based: Ontology-derived pattern matching for high-confidence relations.
  • LLM-based: Using MiniMax-Text-01 to extract novel associations from full-text articles.
• Statistics: The resulting graph contains 24,324 unique entities and 32,009 evidence-grounded triplets, with 14,576 cross-layer connections enabling multi-hop reasoning.

2. EPIBENCH: Evaluation Benchmark

EPIBENCH evaluates LLMs on five clinically motivated tasks, utilizing a Graph-RAG (Retrieval-Augmented Generation) approach where a Graph Retriever extracts relevant subgraphs from EPIKG to augment LLM inputs:

• Task 1: Clinical Decision Accuracy (CDA): 1,000 multiple-choice and 5,199 open-ended questions on syndrome diagnosis, EEG interpretation, and treatment selection.
• Task 2: Clinical Report Generation (CRG): Generating evidence-grounded EEG impression reports from raw EEG descriptions and patient history.
• Task 3: Biomarker-Driven Precision Medicine (BPM): Selecting antiseizure medications based on genetic variants and phenotypes (pharmacogenomics).
• Task 4: Treatment Recommendation (TR): Recommending guideline-consistent therapies under patient-specific constraints (e.g., pregnancy, comorbidities).
• Task 5: Deep Research Planning (DRP): Proposing feasible research directions and study designs based on gaps in current epilepsy literature.

3. Experimental Setup

The framework was evaluated on six LLMs (GPT-4o, Claude Sonnet 4, Gemini 2.0 Flash, Llama-3.3-70B, Qwen2.5-72B, Mistral Small 3.1). Performance was compared between standard prompting and Graph-RAG settings. Baselines included MedRAG, DR.KNOWS, and AMG-RAG.

Key Results

Integrating EPIGRAPH consistently improved performance across all tasks and models:

• Overall Gains: Graph-RAG provided an average relative improvement of +11.3% in MCQ accuracy and +34.8% in pharmacogenomic reasoning (Task 3).
• Pharmacogenomics (Task 3): This task showed the largest gains. For instance, Claude Sonnet 4 improved from 66% to 82% accuracy. Open-source models saw dramatic improvements (e.g., Qwen +42%, Llama +36%), suggesting that pharmacogenomic knowledge is largely absent from pre-trained LLM parameters and requires external retrieval.
• Clinical Reasoning Quality: LLM-as-Judge scores for reasoning quality improved by +31.9% on average, significantly outpacing gains in surface-level metrics like ROUGE-L, indicating that Graph-RAG enhances the logical structure of clinical arguments.
• Safety and Guidelines: In Treatment Recommendation (Task 4), Graph-RAG improved Drug Safety Scores by +12.4% and Guideline Concordance by +14.1%, demonstrating better avoidance of contraindicated medications.
• EEG Report Generation (Task 2): On the Harvard EEG dataset, integrating EPIKG improved METEOR scores, with MedGemma-4B showing a +30.8% relative improvement. The generated reports were more precise in describing waveform abnormalities and aligning with neurologist-written impressions.
• Ablation Studies: Sensitivity analysis confirmed that optimal performance requires traversing 3–4 hops in the graph, validating the necessity of multi-hop reasoning for pharmacogenomic tasks. Graph-based retrieval (PPR-PCST) outperformed flat semantic retrieval by up to 8.4 percentage points on precision medicine tasks.

Significance and Claims

The paper claims that EPIGRAPH establishes a new standard for evaluating knowledge-augmented LLMs in clinical neurology. Its significance lies in:

1. First Large-Scale Epilepsy KG: It provides the first open-source, multi-hop relational knowledge graph dedicated to epileptology, bridging the gap between genetic mechanisms, phenotypes, and treatments.
2. Modular Benchmarking: EPIBENCH offers a comprehensive, plug-and-play framework for evaluating LLMs across diverse clinical reasoning tasks, moving beyond simple question answering to include report generation and research planning.
3. Practical Clinical Impact: By grounding LLM outputs in structured, evidence-based knowledge, the framework demonstrates that AI can meaningfully support clinicians in avoiding contraindications, interpreting complex EEG data, and adhering to clinical guidelines.
4. Addressing the "Black Box" of Medical Knowledge: The results suggest that for complex, evidence-intensive domains like epilepsy, retrieving structured knowledge from a curated KG is more effective than relying solely on the parametric knowledge of even the largest LLMs.

The authors position EPIGRAPH not as a final solution, but as a foundational resource to drive future development in retrieval-augmented fine-tuning and memory approaches for clinical AI.