Technical Summary: RareLens

Problem Statement

Rare diseases present a unique challenge for clinical decision-making due to heterogeneous presentations, sparse evidence, and limited expertise. Patients frequently endure prolonged diagnostic journeys, with over 70% experiencing incorrect or delayed diagnoses. A critical bottleneck occurs at the initial clinical encounter: because rare disease manifestations are nonspecific and overlap with common conditions, clinicians often fail to recognize the possibility of a rare disease, preventing the initiation of targeted investigations.

Existing AI approaches for rare diseases are constrained by two main limitations:

1. Isolated Scope: They predominantly focus on single stages of care (typically diagnosis) rather than supporting the full longitudinal trajectory from risk screening to prognosis.
2. Consensus Bias: They generally rely on a single model or seek consensus among models, implicitly treating disagreement as uncertainty or error to be reduced, rather than recognizing that divergent perspectives may contain complementary information.

Methodology

Dataset: RarelensBench

The authors constructed RarelensBench, a real-world clinical benchmark comprising 157,525 cases (42,836 rare disease cases and 114,689 non-rare cases).

• Coverage: The rare disease cohort spans all 33 Orphanet disease categories and over 7,000 distinct conditions.
• Structure: Cases are organized as longitudinal trajectories with four sequential stages, exposing only information available at each specific point in time to prevent data leakage:
  1. Primary Visit: Demographics, medical history, and physical exam findings (Risk Screening & Hypothesis Generation).
  2. Follow-up Visit: Addition of laboratory and imaging results (Diagnostic Confirmation).
  3. Treatment Visit: Confirmed diagnosis and treatment planning.
  4. Prognosis Visit: Treatment outcomes and long-term trajectory prediction.
• Sources: Data was curated from PubMed-indexed case reports, MIMIC-IV-Ext-BHC, and MIMIC-IV-Ext-CDM, processed through a standardized pipeline for quality control and rare disease identification.

System Architecture: RareLens

RareLens is an end-to-end AI system designed to support the entire rare disease care continuum. Its core innovation is the alignment of divergent reasoning from heterogeneous Large Language Models (LLMs) rather than forcing consensus.

The system comprises four coordinated modules, each mirroring a stage of care:

1. RareAlert (Risk Screening): Estimates patient-level rare disease risk from primary visit data. It utilizes an ensemble of 10 heterogeneous LLMs to generate independent risk scores and reasoning chains. A supervised meta-learner aligns these divergent signals into a calibrated risk score, which is then distilled into a single, locally deployable 32B parameter model (Qwen3-32B).
2. RareDiagnosis (Diagnosis): Operates at primary and follow-up visits. It employs 11 LLMs to generate independent ranked lists of candidate diagnoses and recommended investigations. A learning-to-rank model (XGBoost) aligns these heterogeneous hypotheses, weighting them based on ground-truth supervision to produce a unified, clinically actionable ranking.
3. RareTreatment (Treatment Planning): Generates ranked treatment recommendations after diagnosis. It uses 12 LLMs augmented by a retrieval-augmented generation (RAG) system based on a curated knowledge base of 1,021 rare diseases and real-world case records. A re-ranker model aligns the diverse treatment plans to prioritize clinically appropriate, safe, and complete interventions.
4. RarePrognosis (Prognosis Prediction): Predicts long-term trajectories across three dimensions (overall outcome, functional status, symptom burden). It uses 12 LLMs and a gradient-boosted stacking ensemble to align divergent prognostic predictions.

Alignment Mechanism:
Instead of simple aggregation (mean, median, or majority vote), RareLens treats the outputs of heterogeneous models as "candidate evidence." It learns to reconcile these perspectives by:

• Quantifying inter-model divergence (e.g., reasoning chain distinctness, rank disagreement).
• Training supervised alignment models (meta-learners, rankers, or stacking ensembles) to map the multi-model reasoning signals to a single, calibrated decision.
• Demonstrating that this divergence is an informative signal, particularly for cases where individual models fail.

Key Results

Performance Across Care Stages

RareLens was evaluated against 16 frontier LLMs (including GPT-5, DeepSeek-R1, Claude-3.7-Sonnet, and Gemini-2.5-Pro) and standard ensemble baselines.

• RareAlert (Screening): Achieved an AUC of 0.917, outperforming all 16 evaluated LLMs (the next best, GPT-5, scored 0.905). It demonstrated high sensitivity (0.778) and specificity (0.935). Notably, it recovered 39% of cases misclassified simultaneously by at least ten other models.
• RareDiagnosis: Achieved Top-1 accuracy of 65.5% at the primary visit and 89.8% at follow-up, surpassing the best baseline (GPT-5: 57.1% and 81.0%, respectively). It significantly improved the ranking of correct diagnoses (Mean Reciprocal Rank of 0.710 at primary visit).
• RareTreatment: Achieved a Top-1 accuracy of 89.8%, exceeding GPT-5 (83.0%). It recommended the highest number of appropriate treatments per case (8.7) while minimizing high-risk recommendations (1.8%).
• RarePrognosis: Outperformed all baselines in predicting overall outcome (62.4% accuracy), functional status (63.3%), and symptom burden (67.0%). Under a strict joint criterion (all three dimensions correct), it succeeded in 33.9% of cases vs. 28.3% for the best individual model.

Human-AI Collaboration Study

An external evaluation involving 23 physicians and 1,287 cases compared three conditions: RareLens alone, unaided physicians, and physicians assisted by RareLens.

• End-to-End Success: RareLens alone succeeded in 21.3% of rare disease cases (meeting all criteria from screening to prognosis). Physicians assisted by RareLens succeeded in 10.0%, while unaided physicians succeeded in only 0.8%.
• Collaboration Dynamics: While RareLens assistance significantly improved physician performance (doubling yield in downstream stages), the collaborative group did not reach the performance of the autonomous system. The study notes that physicians often missed the "alert" stage (sensitivity of 0.369 vs. 0.784 for RareLens), which acted as the primary bottleneck.
• Rescue Rates: RareLens successfully "rescued" physician errors (turning incorrect answers correct) at rates significantly higher than it caused regressions (e.g., 49.8% rescue vs. 7.3% regression at screening).

Significance and Claims

The paper claims that divergent reasoning among heterogeneous AI systems is a computable and exploitable source of information, rather than a source of noise to be eliminated.

• Paradigm Shift: The work challenges the prevailing strategy of scaling single models or seeking consensus. Instead, it proposes that diversity in foundation models represents a computational resource that can be explicitly modeled to improve decision-making under high uncertainty.
• End-to-End Care: RareLens demonstrates that AI can support the entire rare disease trajectory, not just isolated diagnostic tasks, by maintaining continuity of information from initial risk recognition through prognosis.
• Clinical Utility: By reframing rare disease identification as a universal screening task at the first encounter, the system addresses the root cause of diagnostic delay. The results suggest that reliable clinical AI may emerge from integrating diverse, imperfect perspectives rather than relying solely on increasingly capable individual models.
• Human-AI Interaction: The study highlights that simply providing model outputs to clinicians does not guarantee optimal outcomes; the integration of AI into workflows must be designed to complement human judgment, particularly in recognizing counterintuitive or rare presentations that challenge existing clinical expectations.

The authors conclude that while RareLens shows promise, its real-world impact requires prospective validation in routine clinical workflows and further investigation into collaboration strategies that truly leverage the complementary strengths of human expertise and machine inference.