Technical Summary: Rare Disease Mining Agents (RDMA)

Problem Statement

Rare diseases affect approximately 1 in 10 Americans but remain systematically under-documented in clinical records. Current approaches relying on International Classification of Diseases (ICD) codes fail to capture the breadth of rare conditions; over 50% of Orphanet codes lack direct ICD mappings, and only 2.2% of Human Phenotype Ontology (HPO) codes match ICD codes. Consequently, patient populations are often invisible in structured data, delaying diagnosis.

While mining unstructured Electronic Health Record (EHR) notes offers a path to direct mapping to HPO and Orphanet, existing methods face three critical limitations:

1. Generalization Failure: Existing benchmarks (e.g., MIMIC-III attempts) often suffer from poor annotation quality, misinterpreting clinical abbreviations (e.g., confusing "NPH" as a disease rather than an insulin type) or failing to reflect the length and noise of real clinical notes. Models fine-tuned on these datasets often fail to generalize to out-of-distribution data.
2. Lack of Reasoning: Clinically relevant phenotypes are often implicit, requiring the interpretation of laboratory results and clinical reasoning rather than simple entity extraction. Conventional two-stage "extract-and-match" pipelines and dictionary-based methods fail to capture these implied entities.
3. Privacy and Cost Barriers: Deploying large language models (LLMs) for Protected Health Information (PHI) often requires cloud-based solutions subject to extensive institutional review. Local deployment is desirable for privacy but historically requires expensive, large-scale hardware.

Methodology: The RDMA Framework

The authors propose Rare Disease Mining Agents (RDMA), an agentic framework designed to generalize across data characteristics without task-specific training. RDMA coordinates LLM-based reasoning across structured ontology access, implicit phenotype inference, and uncertainty-flagging.

Core Architecture

RDMA processes clinical documents sentence-by-sentence through a multi-step pipeline (Algorithm 1):

1. Semantic Retrieval: For each sentence, the system retrieves top-5 candidates from HPO and Orphanet ontologies using semantic search (FAISS with MedEmbed).
2. Entity Extraction: An LLM extracts potential entities from the sentence and retrieved candidates.
3. Verification and Implication (Agent Tools):
   • Abbreviation Resolution: For rare diseases, the agent expands clinical abbreviations to resolve ambiguity (e.g., distinguishing "NPH" in the context of insulin).
   • Ontology Grounding: The agent verifies if an entity matches an ontology entry. For rare diseases, it further confirms the entity is a disease and not a related biological entity (e.g., a gene or protein).
   • Lab Event Detection: For phenotypes, if an entity is not a direct match, the agent checks if it represents an abnormal lab value (e.g., elevated creatinine) and infers the corresponding phenotype (e.g., hypercreatininemia).
   • Implicit Reasoning: The agent determines if a context implies a phenotype or disease not explicitly stated.
4. Matching: Verified entities are mapped to specific ontological codes (HPO or ORPHA) using the LLM.
5. Dataset Refinement: The system compares new mining results against prior annotations. It flags "uncertain" cases—such as false positives that pass verification or false negatives that are re-verified as true—for human expert review (Algorithm 2).

Deployment Strategy

RDMA is designed to run on a small, 4-bit quantized LLM (specifically Mistral 24B). This approach prioritizes workflow design over model scale, enabling private deployment on standard consumer hardware (e.g., RTX 3090) without cloud-based PHI exposure.

Key Results

The paper evaluates RDMA across multiple benchmarks (BioLark, CSC, RareDis, and a refined MIMIC-III set) comparing it against rule-based methods, fine-tuned encoders, and RAG-based LLM approaches.

• Generalization without Training: RDMA substantially outperforms fine-tuned models (e.g., BioBERT, BioClinicalBERT) and RAG-based baselines across datasets with different characteristics. Notably, fine-tuned models often degrade significantly when moving from training corpora to out-of-distribution benchmarks, whereas RDMA maintains robust performance without any task-specific training.
• Model Size Efficiency: Increasing model size beyond 24B parameters (e.g., to 70B or proprietary models like GPT-5) yields no consistent improvement in extraction performance. RDMA achieves maximal performance with the smaller Mistral 24B model.
• Cost Reduction: By utilizing a quantized 24B model, RDMA reduces inference costs by up to 10x and local hardware costs by up to 17x compared to running larger, non-quantized models (e.g., Llama 3.3 70B) required for comparable performance in other frameworks.
• Annotation Efficiency: In a case study on MIMIC-III, RDMA's uncertainty-flagging mechanism reduced the expert annotation burden by 63% (reducing re-review cases from 333 to 122) while improving agreement quality (Cohen's Kappa increased from 0.46 to 0.81). It also recovered rare disease mentions missed during initial human review.

Significance and Claims

The authors claim that RDMA addresses the critical bottlenecks in clinical information extraction by demonstrating that agentic workflow design is more determining for robust performance than model size.

Key contributions include:

1. A Generalizable Framework: RDMA provides a method that generalizes across diverse data characteristics without the need for expensive, task-specific fine-tuning, overcoming the brittleness of current supervised and RAG-based approaches.
2. Cost-Effective Privacy: The framework proves that high-performance rare disease mining can be achieved on standard consumer hardware using small quantized models, facilitating private, HIPAA-compliant deployment without cloud exposure.
3. Human-AI Collaboration: The uncertainty-flagging mechanism offers a blueprint for scalable human-AI collaboration, significantly reducing the workload for clinical experts while preserving the quality of rare disease documentation.

The paper concludes that while challenges remain in capturing implied phenotypes and handling long-context reasoning, RDMA provides a practical, cost-effective foundation for building rigorous rare disease benchmarks and differential diagnosis frameworks.