Frequency-Stratified Benchmarking Reveals Strong Long-Tail Limitations of Large Language Models in Diagnosis of Monogenic Diseases

Technical Summary: Frequency-Stratified Benchmarking of LLMs in Monogenic Disease Diagnosis

Problem Statement
The knowledge base of monogenic (Mendelian) diseases exhibits a pronounced long-tail distribution: a small number of canonical disease genes are extensively documented in biomedical literature, while a vast number of rare, recently characterized genes are sparsely reported. While Large Language Models (LLMs) derive their biomedical knowledge from these same text corpora, it remains unclear how this uneven knowledge structure impacts their diagnostic reliability. Current evaluation frameworks often rely on overall accuracy metrics, which may obscure systematic failures on low-frequency, evidence-sparse genes. This study addresses the need to determine whether LLMs exhibit frequency-dependent diagnostic bias and whether they can serve as reliable standalone systems for molecular diagnosis across the full spectrum of monogenic diseases.

Methodology
The authors constructed MendelianBench-5K, a novel benchmark designed to evaluate LLM performance through the lens of literature reporting frequency.

• Data Source: The benchmark was derived from the PhenoGemini Atlas, a literature-curated, patient-level phenotype–gene resource containing over 347,000 molecularly diagnosed individuals and 4,739 causative genes.
• Stratification: The authors defined "gene literature frequency" as the proportion of patient records associated with a specific causative gene. They stratified 5,000 molecularly diagnosed cases into five tiers (quintiles, Q1–Q5) based on this frequency, sampling 1,000 cases from each tier to ensure a balanced representation of both common and long-tail genes.
• Annotation: Free-text phenotypic descriptions were converted to Human Phenotype Ontology (HPO) terms using Doc2HPO.
• Evaluation: Nine mainstream LLMs (including GPT-5.5, Gemini-3.1-pro-preview, Claude-opus-4.7, and DeepSeek variants) were evaluated on a zero-shot phenotype-to-gene diagnostic task. Models were prompted to rank the five most likely causal genes given a patient's phenotypic profile.
• Metrics: Performance was measured using Top-1 accuracy, Top-5 accuracy, and Mean Reciprocal Rank (MRR).
• Validation: A subset of 500 cases underwent manual review by clinical physicians to verify the consistency of the benchmark data with source literature, confirming a 97% support rate.

Key Results

1. Limited Overall Performance: Even the best-performing model (GPT-5.5) achieved a Top-1 accuracy of only 18.56% and a Top-5 accuracy of 33.78%. This indicates that, without external retrieval or structured genetic evidence, current LLMs cannot yet function as reliable standalone diagnostic systems for monogenic diseases.
2. Strong Long-Tail Bias: Diagnostic accuracy showed a monotonic, positive correlation with gene literature frequency.
   • Top-1 Accuracy: Increased from 2.24% in the lowest frequency tier (Q1) to 42.48% in the highest frequency tier (Q5)—an approximate 19-fold difference.
   • Top-5 Accuracy: Rose from 6.92% (Q1) to 66.10% (Q5).
   • This trend was consistent across all nine models, with Spearman correlations between frequency tier and accuracy ranging from 0.375 to 0.485.
3. Temporal Decline: Performance degraded significantly for recently published cases. Top-1 accuracy dropped from 28.47% for cases published before 2005 to 7.83% for cases published between 2020 and 2024.
4. Phenotype Input Sensitivity: Model performance improved when moving from single-phenotype inputs to 2–3 phenotypes but plateaued or declined with larger numbers of phenotypes (≥4), suggesting that excessive or uncurated phenotypic input does not necessarily aid ranking and may introduce noise.
5. Systematic Error Patterns:
   • Universal Failures: 58.24% of cases were classified as "Hard" (0/9 models correct), heavily enriched for low-frequency genes (54.98% from Q1–Q2).
   • False Positives: When models erred, they disproportionately predicted well-known, high-frequency "star" genes (e.g., MECP2, FMR1, POLG) rather than the correct rare gene. The top 20 false-positive genes were predominantly from the top 200 most frequent genes in the literature.

Key Contributions

• MendelianBench-5K: The introduction of the first frequency-stratified benchmark for monogenic disease diagnosis, explicitly designed to expose the long-tail limitations of AI models.
• Evidence of Knowledge-Frequency Bias: The study provides empirical evidence that LLM diagnostic errors are not random but are systematically structured by the frequency landscape of biomedical literature. Models tend to "default" to familiar, high-attention genes when uncertain, leading to the under-recognition of rare conditions.
• Evaluation Framework: The authors demonstrate that overall accuracy is insufficient for assessing medical AI in long-tailed domains; frequency-stratified evaluation is necessary to reveal systematic failures on underrepresented cases.

Significance and Claims
The paper claims that current LLMs exhibit pronounced limitations in the molecular diagnosis of monogenic diseases, specifically their inability to generalize to rare, recently characterized, or underrepresented genes. The authors argue that the apparent diagnostic capability of these models is disproportionately concentrated in literature-enriched genes.

Consequently, the study asserts that:

1. Clinical Caution: LLMs should not be deployed as autonomous diagnostic tools for monogenic diseases without integration into expert workflows, genomic variant interpretation, and external evidence retrieval.
2. Benchmarking Standard: Future evaluations of medical LLMs in long-tailed biomedical fields must adopt frequency-stratified benchmarks to avoid overestimating clinical utility on common cases while missing critical failures on rare diseases.
3. Future Directions: The authors suggest that "agentic" systems, which incorporate retrieval, memory, and iterative reflection (e.g., DeepRare), may be required to overcome the limitations of static parametric knowledge and improve performance on long-tail cases.

The authors conclude that while LLMs show promise for well-studied conditions, their performance declines substantially for rare diseases, necessitating a shift toward evidence-driven, clinically accountable diagnostic reasoning rather than reliance on memorized associations.