Blinded Multi-Rater Comparative Evaluation of a Large Language Model and Clinician-Authored Responses in CGM-Informed Diabetes Counseling

1. Problem Statement

Continuous Glucose Monitoring (CGM) is a cornerstone of modern diabetes management, providing real-time data on glycemic patterns. However, interpreting this data remains a significant bottleneck in clinical practice due to:

• Time Intensity: Explaining complex CGM patterns (Time in Range, variability, hypoglycemia risk) to patients is time-consuming for clinicians.
• Patient Overwhelm: Patients often struggle to translate raw data and visualizations into actionable lifestyle or behavioral changes.
• Emotional Burden: Diabetes management involves significant psychological stress, requiring empathetic communication that is difficult to scale.
• Limitations of Current AI: While Large Language Models (LLMs) show promise, they often suffer from hallucinations, lack of grounding in specific patient data, and a failure to adhere strictly to clinical guidelines. Existing evaluations rarely compare AI outputs directly against clinician-authored responses in structured, patient-facing counseling scenarios.

2. Methodology

The study employed a blinded, multi-rater, comparative design to evaluate a Retrieval-Augmented Generation (RAG) based Conversational Agent (CA) against senior clinicians.

A. System Architecture

• Model: The CA utilized GPT-5.1 (accessed via OpenAI API) without parameter fine-tuning.
• Retrieval-Augmented Generation (RAG): To ensure clinical accuracy and safety, the system retrieved relevant information from a curated corpus containing:
  • International diabetes guidelines (ADA, etc.).
  • CGM interpretation standards (TIR, TBR, TAR definitions).
  • Psychosocial resources and patient education materials.
  • Medical dialogue datasets (MedDialog) for conversational style.
• Input Processing:
  • Structured Data: Raw CGM time-series data (from public datasets) were pre-processed in Python to calculate consensus metrics (Mean Glucose, SD, CV, TIR, TBR, TAR).
  • Multimodal Input: The system received the calculated metrics, a synthetic patient vignette (clinical profile, lifestyle, concerns), and visual CGM plots (AGP and continuous traces).
• Safety Guardrails: The system prompt explicitly instructed the CA to avoid prescriptive medical advice (e.g., dose adjustments) and to encourage consultation with the care team for risky patterns.

B. Study Design & Data

• Cases: 12 de-identified CGM cases were constructed using data from the ShanghaiT1DM/T2DM and OhioT1DM datasets. Each case included a synthetic vignette and visual materials.
• Questions: A bank of 24 questions per case (totaling 288 unique case-question pairs) covered six domains: glucose interpretation, food/exercise impact, medication guidance, emotional concerns, long-term goals, and technical issues.
• Participants: 6 senior UK diabetes clinicians (10+ years experience) served as both authors (Phase 1) and raters (Phase 2).
• Procedure:
  1. Phase 1: Clinicians wrote responses to 2 assigned cases each. The CA generated parallel responses for the same cases.
  2. Phase 2: The 6 clinicians blindly rated 3 responses per question (1 CA, 2 other clinicians) on a 5-point Likert scale.
• Evaluation Metrics:
  • Quality Dimensions: Clinical Accuracy, Guideline Adherence, Actionability, Personalization, Communication Clarity, and Empathy.
  • Safety: 3-level flag (No concern, Minor concern, Major concern).
  • Source Identification: Raters guessed if the response was Human or AI.

C. Statistical Analysis

• Primary Analysis: Linear mixed-effects models with random intercepts for case and rater to compare mean scores between CA and Clinician.
• Reliability: Inter-rater reliability assessed via Intraclass Correlation Coefficients (ICC).
• Sensitivity: Wilcoxon signed-rank tests for paired differences and analysis of response length (word count) as a covariate.

3. Key Results

The study analyzed 288 unique responses (144 CA, 144 Clinician) resulting in 864 ratings.

A. Quality Comparison

The CA significantly outperformed clinicians across all metrics:

• Overall Quality: CA mean score 4.37 vs. Clinician 3.58 (Mean difference: 0.782, 95% CI 0.692–0.872, P < .001).
• Dimension Breakdown:
  • Empathy: Largest gap (CA 4.37 vs. Clinician 3.30; Diff: 1.062).
  • Actionability: Second largest gap (CA 4.42 vs. Clinician 3.43; Diff: 0.992).
  • Clinical Accuracy & Guideline Adherence: CA also scored significantly higher, though the margin was smaller than for empathy.
• Consistency: The CA's performance was consistent across all 6 question domains, including psychosocial topics (fear of hypoglycemia, frustration).

B. Safety and Reliability

• Safety Flags: No significant difference in safety concerns. Both groups had a low rate of "Major Concerns" (0.7% for both).
• Inter-rater Reliability: Single-rater reliability was modest (ICC ~0.27), but improved to moderate when averaging 3 raters (ICC ~0.53).
• Source Identification: Raters correctly identified the source 80.7% of the time (significantly above chance), suggesting stylistic differences remain detectable.

C. Response Length

• CA responses were significantly longer (Mean 211 words) than clinician responses (73 words).
• Crucially, statistical adjustment for word count showed that length did not explain the quality difference. In fact, for clinicians, longer responses were sometimes negatively associated with empathy ratings.

4. Key Contributions

1. First Blinded Comparative Evaluation: This is among the first studies to directly compare RAG-grounded LLM outputs against clinician-authored responses in a structured CGM counseling context using blinded expert raters.
2. Superiority in "Soft" Skills: Contrary to the assumption that AI lacks empathy, the study found the CA outperformed clinicians significantly in empathy and actionability, likely due to the system's strict adherence to supportive language guidelines and structured output.
3. Validation of RAG in Clinical Contexts: The study demonstrates that grounding LLMs in curated, guideline-based corpora can produce responses that are not only safe but clinically superior in standardized explanatory tasks.
4. Defined Scope of Utility: The research clearly delineates the AI's role as an adjunct for education and explanation rather than a tool for autonomous therapeutic decision-making.

5. Significance and Implications

• Clinical Workflow Optimization: The findings suggest that RAG-based CAs can effectively handle standardized, time-intensive tasks (CGM pattern explanation, patient education, pre-consultation prep), potentially freeing up clinician time for complex decision-making and high-risk cases.
• Scalable Patient Support: The system offers a scalable solution for providing consistent, empathetic, and guideline-compliant feedback to the growing population of CGM users, addressing the "emotional burden" of diabetes management.
• Safety First: The study reinforces that while AI can outperform humans in structured communication, it must remain a supportive tool under clinician oversight, particularly for medication adjustments and individualized treatment plans.
• Future Directions: The authors call for prospective studies in real-world interactive workflows to assess long-term impacts on patient outcomes, consultation efficiency, and the system's ability to handle dynamic, multi-turn conversations.

Conclusion: The study concludes that retrieval-grounded LLMs are viable, high-quality adjunct tools for routine CGM review and patient education, capable of generating responses that specialists rate as more empathetic and actionable than those of human clinicians in controlled scenarios, provided they are used within strict safety boundaries.