Blinded Expert Evaluation of Large Language Models for Osteoporosis Case Management in Older Adults: Impact of Direct Guideline Integration

Technical Summary: Blinded Expert Evaluation of Large Language Models for Osteoporosis Case Management in Older Adults

Problem Statement
Osteoporosis management in older adults (≥65 years) requires complex, individualized decision-making that integrates fracture risk, pharmacological choices, renal function, polypharmacy, fall risk, and contraindications. While Large Language Models (LLMs) offer potential for clinical decision support, their reliability, safety, and adherence to guidelines in geriatric scenarios remain uncertain. Specifically, there is a lack of evidence regarding whether directly integrating clinical guidelines into LLMs improves their performance compared to baseline models. Furthermore, existing literature often lacks blinded evaluation protocols and focuses on general knowledge rather than complex, case-based geriatric reasoning.

Methodology
This cross-sectional, methodological study employed a blinded, independent expert evaluation protocol to compare four AI approaches across 25 hypothetical osteoporosis case scenarios. The cases were developed by geriatric specialists and categorized into five domains: classic postmenopausal osteoporosis, geriatric challenges (polypharmacy, falls), drug management, secondary osteoporosis/comorbidities, and complications/pitfalls.

The four AI panels evaluated were:

1. ChatGPT 4.5 (Baseline): No external guideline upload.
2. Gemini 3 (Baseline): No external guideline upload.
3. ChatGPT 4.5 + Guideline: Full text of the Turkish Society of Endocrinology and Metabolism (TES) Osteoporosis Guidelines 2025 uploaded.
4. NotebookLM + Guideline: Full text of the TES Guidelines 2025 uploaded.

Evaluation Process:

• Blinding: Two independent geriatric specialists evaluated all 100 responses (25 cases × 4 panels) without knowledge of the generating model.
• Scoring: Responses were rated on a 5-point Likert scale across five dimensions: Scientific Accuracy, Guideline Adherence, Clarity, Holistic Geriatric Approach, and Safety/Contraindication Awareness. Total scores ranged from 5 to 25.
• Hallucination Detection: Defined as factually incorrect, fabricated, or unsupported clinical statements based on the TES Guidelines.
• Statistical Analysis: Inter-rater agreement was assessed using ICC(2,1) and weighted Cohen's kappa. Panel comparisons utilized Friedman's test with Bonferroni-corrected Wilcoxon signed-rank tests for pairwise comparisons.

Key Results

• Overall Performance: The ChatGPT 4.5 base model achieved the highest mean total score (22.76/25), significantly outperforming Gemini 3 (21.80), NotebookLM + Guideline (21.56), and ChatGPT 4.5 + Guideline (21.42).
• Impact of Guideline Integration: Contrary to expectations, directly uploading guideline documents did not improve performance. In fact, the ChatGPT 4.5 base model scored significantly higher on "Scientific Accuracy" and "Guideline Adherence" than the same model with guidelines uploaded.
• Domain-Specific Findings:
  • Significant differences between panels were found only in Scientific Accuracy ($p=0.0006$) and Guideline Adherence ($p<0.0001$).
  • No significant differences were observed in Clarity, Holistic Geriatric Approach, or Safety/Contraindication Awareness.
  • "Safety/Contraindication Awareness" was the lowest-rated criterion across all base models.
• Hallucinations: Hallucinations were detected in only one case (4%) within the Gemini 3 base model; no hallucinations were found in the other three panels.
• Inter-rater Reliability: Agreement was excellent for Scientific Accuracy in the ChatGPT 4.5 base model ($\kappa=0.834$) but varied from poor to low for subjective criteria like "Holistic Geriatric Approach" and "Safety Awareness" across all panels.

Key Contributions

1. Blinded Evaluation Protocol: The study addresses a methodological gap by implementing a blinded, independent expert assessment, reducing brand bias often present in AI comparisons.
2. Guideline Integration Efficacy: It provides empirical evidence that unstructured, direct document uploading of guidelines does not enhance LLM performance in geriatric osteoporosis management and may even degrade guideline adherence scores compared to baseline models.
3. Safety Limitations: The research highlights that while LLMs may excel in knowledge generation, they share significant limitations in context-sensitive clinical reasoning, specifically regarding safety, contraindication awareness, and holistic geriatric approaches.

Significance and Claims
The authors conclude that while LLMs can support decision-making in geriatric osteoporosis, they cannot replace expert supervision. The study demonstrates that:

• Current base models (specifically ChatGPT 4.5) may outperform versions augmented with unstructured guideline uploads.
• Safety and contraindication recognition remain critical weaknesses requiring human oversight.
• The lack of benefit from direct guideline uploading suggests that unstructured presentation is inadequate for integration; future systems likely require structured Retrieval Augmented Generation (RAG), knowledge graphs, or specific geriatric assessment modules to be effective.
• Clinical supervision is necessary to ensure safety, particularly in vulnerable older populations with multimorbidity and polypharmacy.

The paper explicitly states that these findings are based on models as of February 2026 and a single national guideline, and it calls for future studies to evaluate physician-supervised systems in real-world clinical settings.