An Agentic System for Rare Disease Diagnosis with Traceable Reasoning

This paper introduces DeepRare, a multi-agent system powered by large language models that integrates diverse clinical data and specialized tools to provide accurate, traceable, and expert-validated rare disease diagnosis, significantly outperforming existing methods across multiple global datasets.

The Gist

Imagine a patient with a mysterious, complex illness. They have been to dozens of doctors, tried every test, and still have no answers. This is the "diagnostic odyssey" that millions of people with rare diseases face, often lasting five years or more.

Enter DeepRare, a new AI system designed to act as a super-powered medical detective. Think of it not as a single doctor, but as a high-tech "Medical Task Force" that never sleeps, never forgets, and has read every medical book ever written.

Here is how DeepRare works, explained through simple analogies:

1. The Brain: The "Chief Detective"

At the center of DeepRare is a powerful AI (like a very smart "Chief Detective"). When a patient walks in with symptoms, this Chief doesn't just guess. It organizes a meeting with a team of specialized assistants.

• The Input: The patient might speak in plain English ("I have a rash and my legs hurt"), use medical code (HPO terms), or have genetic test results (DNA data). The Chief Detective translates all of this into a clear case file.

2. The Team: Specialized "Agent" Assistants

Instead of trying to do everything alone, the Chief Detective sends out different "Agents" to gather clues, just like a detective sending a team to different crime scenes:

• The Librarian: This agent instantly searches through millions of medical journals, guidelines, and websites to find papers that match the patient's symptoms.
• The Case Finder: This agent looks through a massive database of other patients. It asks, "Has anyone else had these exact symptoms? What did they turn out to have?"
• The Genetic Analyst: If the patient has DNA test results, this agent acts like a forensic scientist, scanning the genetic code for tiny errors that might be the cause.
• The Translator: This agent converts messy human descriptions into precise medical terms so the computer understands exactly what is happening.

3. The "Self-Reflection" Loop: The Double-Check

This is DeepRare's secret sauce. After the team gathers clues and the Chief makes a guess (e.g., "It's probably Disease X"), the system doesn't just stop.

• It enters a "Self-Reflection" mode. It acts like a skeptical editor: "Wait, does Disease X really fit all the symptoms? Did we miss a clue? Let's go back and search for more evidence."
• If the evidence doesn't hold up, it discards the guess and tries again. This prevents the AI from "hallucinating" (making things up) and ensures the answer is rock-solid.

4. The Output: A Transparent Report Card

When DeepRare is done, it doesn't just give a list of names. It provides a ranked list of the top 5 most likely diseases, but with a crucial difference: Traceable Reasoning.

• For every guess, it says: "I think it's Disease X because the patient has Symptom A and Symptom B, which matches this specific medical study [Link], and this similar patient case [Link]."
• It's like a student showing their work on a math test. A doctor can click the links and verify the evidence themselves. This builds trust, which is vital in medicine.

Why is this a Big Deal?

• It's Faster: What takes a human team days to research, DeepRare does in minutes.
• It's Smarter: In tests, DeepRare correctly identified the rare disease 57% of the time as the #1 guess, beating the next best AI by a huge margin. In some real-world tests, it even outperformed human experts who had been searching for years.
• It's Honest: It admits when it's unsure and shows its sources. It doesn't pretend to know something it doesn't.

The Bottom Line

DeepRare is like giving every doctor a super-intelligent, tireless research assistant who has memorized the entire library of human medicine. It doesn't replace the doctor; it empowers them to solve the unsolvable mysteries of rare diseases, ending the "diagnostic odyssey" for patients who have been waiting too long for answers.

Technical Summary

1. Problem Statement

Rare diseases affect over 300 million people globally, yet diagnosis is often delayed by an average of five years ("diagnostic odyssey") due to clinical heterogeneity, low prevalence, and limited clinician familiarity. Existing diagnostic tools face four critical challenges:

• Multidisciplinary Complexity: Symptoms span multiple organ systems, requiring broad medical knowledge.
• Data Scarcity: Individual rare diseases have very few cases, making supervised learning prone to overfitting and catastrophic forgetting.
• Dynamic Knowledge: The field evolves rapidly (260–280 new genetic diseases/year), requiring systems that can integrate new knowledge without retraining.
• Lack of Transparency: Clinical deployment requires interpretability; "black box" predictions are insufficient for trust and accountability.

2. Methodology: The DeepRare Agentic System

DeepRare is a multi-agent system powered by Large Language Models (LLMs) designed for differential diagnosis. It employs a three-tier architecture inspired by the Model Context Protocol (MCP), moving beyond simple prompt engineering to a structured, tool-augmented workflow.

A. System Architecture

1. Central Host (The Brain): Powered by an LLM (default: DeepSeek-V3) with a persistent Memory Bank. It orchestrates the workflow, synthesizes evidence, and performs decision-making.
2. Specialized Agent Servers: Modular agents that interact with specific tools and environments:
   • Phenotype Extractor: Converts free-text clinical notes into standardized Human Phenotype Ontology (HPO) terms.
   • Genotype Analyzer: Processes VCF files (Whole Exome Sequencing) using tools like Exomiser for variant prioritization.
   • Knowledge & Case Searchers: Retrieve real-time medical literature, guidelines, and similar patient cases from external databases (PubMed, Orphanet, OMIM, etc.).
   • Disease Normalizer: Maps free-text disease names to standard identifiers (OMIM/Orphanet).
3. External Data Sources: A heterogeneous layer comprising medical literature, curated knowledge bases, and a massive case bank (67,795+ cases).

B. Diagnostic Workflow

The system operates in two iterative stages:

1. Information Collection:
   • The host decomposes the input (Free-text, HPO, VCF).
   • Agents retrieve evidence from web sources, case banks, and bioinformatics tools.
   • Preliminary hypotheses are generated based on phenotype and/or genotype analysis.
2. Self-Reflective Diagnosis:
   • The host performs a self-reflection loop, rigorously validating or refuting hypotheses against retrieved evidence.
   • If hypotheses are ruled out, the system iteratively increases search depth and re-evaluates.
   • Traceable Reasoning: The final output is a ranked list of diagnoses, each accompanied by a transparent reasoning chain that cites specific, verifiable medical sources (URLs, literature, case reports).
   • Post-Processing: A verification step checks the validity of all cited URLs to prevent hallucinated references.

3. Key Contributions

• First Agentic Rare Disease System: Demonstrates that multi-agent orchestration outperforms both traditional bioinformatics tools and standalone LLMs in rare disease diagnosis.
• Traceable Reasoning: Unlike standard LLMs, DeepRare generates diagnostic chains linked to verifiable evidence, achieving 95.4% agreement with experts on the factual accuracy of its references.
• Comprehensive Benchmarking: Introduced a rigorous evaluation framework using 6,563 clinical cases across 8 datasets (including 3,134 unique diseases), spanning public benchmarks (RareBench, MIMIC-IV) and two in-house clinical datasets (Xinhua and Hunan Hospitals) containing original Whole Exome Sequencing (WES) data.
• Human-Level Performance: The system is the first computational model to surpass human expert physicians in specific rare disease diagnostic tasks (Recall@1).

4. Key Results

The system was evaluated across 14 medical specialties and multiple input modalities (HPO only, HPO + Genotype).

• HPO-Based Diagnosis:
  • Achieved 57.18% Recall@1, outperforming the next best method (Reasoning LLM) by 23.79%.
  • Outperformed traditional tools (PhenoBrain, PubCaseFinder) and general/medical LLMs significantly.
• Multi-Modal (HPO + Genotype) Diagnosis:
  • On Xinhua Hospital cases (168 cases with WES), DeepRare achieved 69.1% Recall@1, significantly beating Exomiser (55.9%) and other bioinformatics tools.
  • Integration of genetic data boosted Recall@1 from ~40% (phenotype only) to ~69%.
• Comparison with Human Experts:
  • In a blind study against 5 experienced rare disease physicians, DeepRare achieved 64.4% Recall@1 vs. 54.6% for physicians.
  • At Recall@5, DeepRare scored 78.5% vs. 65.6% for physicians.
• Robustness & Ablation:
  • The agentic workflow improved base LLM performance by ~28–30% (e.g., GPT-4o improved from 26.1% to 54.7% Recall@1).
  • The system demonstrated strong generalization across different underlying LLMs (DeepSeek, GPT-4o, Claude), proving the architecture's value is independent of the specific model.
• Failure Analysis:
  • Primary failure modes were "Reasoning Weighing Errors" (41%) and "Phenotypic Mimicry" (38.5%), rather than factual hallucinations or retrieval failures, indicating the system's core knowledge is robust.

5. Significance

• Clinical Impact: DeepRare reduces the "diagnostic odyssey" by providing rapid, evidence-based differential diagnoses. It acts as a "diagnostic copilot," allowing clinicians to focus on patient care rather than information gathering.
• Democratization of Expertise: By performing at or above the level of specialists, the system can assist non-specialist physicians in resource-limited settings, potentially reducing healthcare disparities.
• Trust and Accountability: The traceable reasoning feature addresses the "black box" problem of AI in medicine, providing clinicians with auditable evidence chains that support clinical decision-making and reduce liability.
• Future Roadmap: The modular, plugin-based architecture allows for seamless expansion into treatment planning, prognosis prediction, and screening for undiagnosed populations.

In conclusion, DeepRare represents a paradigm shift from static diagnostic models to dynamic, agentic systems that combine the reasoning capabilities of LLMs with the precision of specialized medical tools and the reliability of verifiable evidence.