Evidence-Grounded Subspecialty Reasoning: Evaluating a Curated Clinical Intelligence Layer on the 2025 Endocrinology Board-Style Examination

1. Problem Statement

While Large Language Models (LLMs) have demonstrated proficiency in general medical licensing exams (e.g., USMLE), they face significant challenges in subspecialty clinical reasoning. Real-world clinical decisions require:

• Integration of rapidly evolving, nuanced guidelines.
• Adherence to strict evidence hierarchies (e.g., society guidelines vs. observational studies).
• Avoidance of hallucinations and factual errors, which pose safety risks in clinical settings.

Current dominant paradigms rely on model scale combined with unconstrained web retrieval (RAG). However, this approach suffers from:

• Heterogeneous source quality: Web retrieval returns ungraded sources without consistent evidence hierarchies.
• Lack of provenance normalization: Difficulty in weighting sources by recency, quality, or clinical relevance.
• Contextual failure: Models may retrieve correct facts but fail to apply them appropriately within complex clinical vignettes.

The authors hypothesize that a domain-specific, curated evidence corpus combined with structured reasoning will outperform general-purpose models augmented with real-time web search for subspecialty tasks.

2. Methodology

Benchmark Dataset

• Source: Endocrine Self-Assessment Program (ESAP) 2025, a 120-question board-style examination developed by the Endocrine Society.
• Content: Covers thyroid, adrenal, pituitary, reproductive systems, diabetes, lipids, and obesity.
• Structure: Clinical vignettes with single-best-answer formats, categorized by reasoning domains (Diagnosis, Pathophysiology, Treatment, Testing, Risk/Prognosis).
• Human Baseline: Derived from aggregate respondent statistics (mean accuracy of 62.3%).

Systems Evaluated

1. Mirror (The Proposed System):
   • Architecture: An evidence-grounded ensemble system.
   • Knowledge Base: A curated, multi-tiered corpus containing:
     • Tier 1: Society guidelines (Endocrine Society, ADA, AACE, KDIGO).
     • Tier 2: High-impact peer-reviewed literature.
     • Tier 3: Landmark randomized controlled trials.
   • Reasoning: Uses a structured stack with specialized components for different clinical archetypes (diagnosis, treatment, etc.) and an arbitration layer to select outputs based on evidence quality.
   • Constraint: Closed-evidence (no external web access).
2. Frontier LLM Baselines:
   • Models: GPT-5, GPT-5.2, and Gemini-3-Pro.
   • Capability: Unrestricted real-time web access to guidelines and literature.
   • Protocol: Allowed to perform unlimited web queries to answer questions.

Evaluation Protocol

• Zero-shot: No fine-tuning on ESAP questions.
• Metrics:
  • Single-answer accuracy: Primary outcome.
  • Top-2 accuracy: Percentage of times the correct answer was in the top two selections (simulating clinical decision support workflows).
  • Citation Verification: Manual audit of 60 random questions to verify source existence, accurate quoting, and logical support.
• Statistical Analysis: McNemar's test for paired proportions; 95% Confidence Intervals (Wilson score).

3. Key Results

Overall Performance

• Mirror: Achieved 87.5% accuracy (105/120).
• Best LLM (GPT-5.2): 74.6% accuracy.
• Human Reference: 62.3% accuracy.
• Significance: Mirror significantly outperformed all LLMs (p < 0.05) and the human reference, despite lacking web access.

Performance by Difficulty

• Hard Questions (Human accuracy < 50%): Mirror achieved 76.7% accuracy, significantly outperforming GPT-5.2 (53.3%) and Gemini-3-Pro (46.7%).
• Medium Questions: Mirror (90.1%) vs. GPT-5.2 (80.3%).
• Easy Questions: All systems performed similarly (>94%).

Top-2 Accuracy

• Mirror: 92.5% (Correct answer in top 2 selections).
• GPT-5.2: 85.25%.
• Implication: Mirror is more reliable as a "narrowing" tool for differential diagnoses.

Error Analysis & Failure Modes

• Mirror Errors: Primarily occurred in complex, multi-domain treatment decisions where prioritization of diagnostic workup vs. treatment was nuanced.
• LLM Errors:
  • GPT-5.2: Frequently failed on specific guideline thresholds (e.g., A1c targets) due to conflicts between parametric training data and current guidelines.
  • Gemini-3-Pro: Struggled with multi-step reasoning requiring the synthesis of multiple evidence sources (e.g., integrating cardiorenal benefits).
• Case Study: In a gestational diabetes case, GPT-5.2 rigidly applied the "insulin is preferred" rule, ignoring patient-specific barriers (needle phobia). Mirror correctly identified Metformin as the feasible alternative based on specific ADA guidance.

Evidence Traceability

• Citation Rate: 74.2% of Mirror's outputs cited at least one society guideline.
• Verification: 100% of sampled citations were verified as accurate (source exists, passage quoted correctly, logic supports answer).
• Guideline Concordance: 86.3% of treatment answers matched society guidelines.

4. Key Contributions

1. Paradigm Shift: Demonstrates that curated evidence + structured reasoning outperforms large scale + unconstrained web retrieval for high-stakes subspecialty tasks.
2. Auditability: Introduces a system where every output is linked to verifiable, tiered evidence, solving the "black box" problem of LLMs in clinical settings.
3. Subspecialty Benchmarking: Provides a rigorous evaluation of AI on the ESAP 2025, showing that AI can exceed the performance of practicing endocrinologists on board-style questions.
4. Clinical Workflow Relevance: Highlights that Top-2 accuracy (92.5%) is a critical metric for clinical decision support, where AI acts as a cognitive aid to narrow options rather than a definitive oracle.

5. Significance and Implications

• Clinical Deployment: The study suggests that for clinical AI, investing in domain-specific evidence infrastructure yields higher returns than simply scaling model parameters.
• Safety & Trust: The ability to provide 100% verified citations addresses the critical barrier of trust and liability in healthcare AI. Clinicians can verify recommendations against primary sources.
• Future Directions: The authors plan to extend this validation to other subspecialties (Cardiology, Nephrology) and conduct prospective studies with real-world clinical vignettes to assess utility beyond examination scores.

Conclusion: The paper argues that for subspecialty medicine, quality of evidence curation and structured reasoning architecture are more critical determinants of performance than model size or real-time web access. Mirror's success indicates a viable path toward deployable, auditable, and highly accurate clinical intelligence systems.