Assessing the Quality, Safety, and Readability of Large Language Model Responses from ChatGPT and Gemini for Knee Osteoarthritis Patient Education

Technical Summary: Assessing the Quality, Safety, and Readability of Large Language Model Responses for Knee Osteoarthritis Patient Education

Problem Statement
Knee osteoarthritis (OA) is a leading cause of chronic pain and disability, requiring complex, long-term self-management involving exercise, weight control, pharmacology, and potential surgical intervention. While clinical guidelines emphasize patient education, time-constrained consultations often leave gaps that patients fill via online searches. Large Language Models (LLMs) like ChatGPT and Gemini are increasingly used for this purpose, yet their suitability for disease-specific, guideline-dense conditions like knee OA remains unverified. Existing evaluations often focus on general medicine or single quality dimensions, frequently neglecting the integrative medicine queries (e.g., Unani treatment) common in South Asian clinical settings and failing to assess multiple quality domains simultaneously. This study addresses the gap in evidence regarding the accuracy, safety, and readability of AI-generated patient education for knee OA.

Methodology
The study employed a comparative cross-sectional design to evaluate responses from ChatGPT (GPT-4o) and Gemini (Gemini 1.5 Pro) accessed in May 2025.

• Question Development: A bank of 36 patient-relevant questions was curated from seven major sources (including AAOS, ACR, NHS, and CDC) and supplemented with clinician-derived queries specific to South Asian contexts, such as Unani treatment, floor-level prayer, and Indian-style toilet usage. Questions were categorized into ten domains: general understanding, symptoms, causes, lifestyle, exercise, pharmacology, surgery, prognosis, integrative medicine, and resources.
• Response Generation: Each question was entered individually into both platforms in fresh sessions without additional prompting or system instructions to simulate a typical patient interaction.
• Quality Evaluation: Five clinicians (all with >10 years of musculoskeletal experience) independently rated responses on a 5-point Likert scale across five domains: accuracy, completeness, clarity, safety, and trustworthiness. Raters were blinded to the platform identity.
• Safety Criteria: Safety was defined by four specific criteria: suggesting delayed consultation, encouraging unsupervised medication changes, recommending therapies without evidence, or causing disproportionate anxiety.
• Readability Analysis: Ten validated readability indices (including Flesch Reading Ease, Flesch-Kincaid Grade Level, and Fry Readability Formula) were applied to all responses.
• Statistical Analysis: Non-parametric Mann–Whitney U tests were used for group comparisons. A Bonferroni correction was applied for 15 simultaneous comparisons (5 quality domains + 10 readability indices), setting the significance threshold at p < 0.0033. Inter-rater reliability was assessed using single-measure [ICC(A,1)] and average-measure [ICC(A,k)] intraclass correlation coefficients.

Key Results

• Quality Domains: ChatGPT demonstrated statistically significant superiority over Gemini in accuracy (19.69 ± 1.69 vs. 17.83 ± 2.14; p = 0.0001, r = 0.45) and completeness (18.08 ± 1.76 vs. 16.19 ± 2.52; p = 0.0004, r = 0.41). These differences survived Bonferroni correction and represented medium-to-large effect sizes.
• Safety and Trust: No significant differences were found between the platforms regarding clarity, safety, or trustworthiness. However, the study notes that inter-rater reliability for these specific domains was poor, suggesting the null results may reflect measurement limitations rather than true equivalence.
• Readability: ChatGPT responses had significantly higher scores on the Fry Readability Formula (11.90 vs. 10.70; p = 0.003), indicating greater textual complexity. The Coleman–Liau index also favored Gemini but did not survive correction. Crucially, both platforms produced text at an 11th-to-13th-grade reading level, well above the recommended 6th-to-8th-grade level for patient education.
• Inter-Rater Reliability: Single-measure agreement (ICC(A,1)) was poor across all domains (0.002–0.208). Average-measure reliability (ICC(A,k)) improved, reaching moderate levels for completeness (0.567) and approaching moderate levels for accuracy (0.421), supporting the robustness of the aggregate findings despite individual rater variance.

Significance and Claims
The paper claims that for knee osteoarthritis patient education, ChatGPT currently offers higher clinician-rated accuracy and completeness than Gemini, while Gemini holds a modest readability advantage. The study emphasizes that neither platform produces content at an appropriate reading level for the general patient population without modification.

The authors assert that their findings are specific to the current model versions and the defined evaluation framework. They caution that the lack of significant differences in safety and clarity should not be interpreted as proof of equivalence due to the low inter-rater reliability in those domains. The inclusion of Unani medicine queries is highlighted as a distinct contribution, addressing the cultural specificity of South Asian patient populations often ignored in Western-centric AI evaluations.

Limitations and Scope
The study explicitly acknowledges several constraints:

1. Reliability: The poor single-meater reliability suggests that individual clinician ratings vary significantly; findings are robust only at the aggregate level.
2. Temporal Specificity: The results are tied to the specific model versions (GPT-4o and Gemini 1.5 Pro) as of May 2025; continuous model updates may alter performance.
3. Patient Outcomes: The study evaluates content quality as judged by experts, not actual patient comprehension, recall, or adherence.
4. Generalizability: The evaluation was limited to English-language responses, two platforms, and a specific set of 36 questions.

Conclusion
The paper concludes that while ChatGPT outperforms Gemini in accuracy and completeness for knee OA education, both models require review by qualified health professionals before use in patient-facing materials. The authors recommend that future research focus on prospective patient-based validation and the development of standardized calibration protocols for rater reliability.