Quality and Readability of Artificial Intelligence Chatbot Responses to Patient Questions About Exocrine Pancreatic Insufficiency: A Comparative Study

Technical Summary: Quality and Readability of AI Chatbot Responses to Patient Questions About Exocrine Pancreatic Insufficiency

Problem Statement
Exocrine Pancreatic Insufficiency (EPI) is a clinical syndrome characterized by impaired digestion and absorption, requiring complex management involving pancreatic enzyme replacement therapy (PERT), dietary adjustments, and nutritional monitoring. Effective management relies heavily on patient understanding, yet patients often face nonspecific symptoms and complex diagnostic pathways. While generative artificial intelligence (AI) and large language models (LLMs) like ChatGPT, Copilot, Gemini, and Perplexity are increasingly used for health information, there is a lack of standardized, multimodel comparisons regarding their ability to provide traceable, well-structured, and comprehensible information specifically for EPI. Previous research suggests that while AI can generate high-quality text, it often fails to meet recommended readability levels for patient education (typically a sixth-grade reading level) and lacks source transparency.

Methodology
The study employed a web-based cross-sectional comparative design conducted in July 2026.

• Question Generation: Researchers identified 20 core patient questions regarding EPI by synthesizing data from Google Trends (global search interest over 5 years), Medical Subject Headings (MeSH), clinical guidelines (including post-ERCP pancreatitis and PERT management), and patient community forums (Inspire and HealthUnlocked).
• Models Evaluated: Four widely used AI chatbots were tested: ChatGPT 5.6, Microsoft Copilot, Google Gemini 3.6 Flash, and Perplexity.
• Data Collection: Each of the 20 questions was submitted once to each model using standardized, default settings without role assignment, word limits, or citation requests. This yielded a total of 80 responses.
• Quality Assessment: Responses were evaluated using four instruments:
  • DISCERN: Assessed treatment information quality, evidence sources, and uncertainty (16–75 scale).
  • EQIP: Assessed completeness, structure, and presentation of patient education materials (0–100 scale).
  • Global Quality Score (GQS): Assessed overall quality, organization, and usefulness (1–5 scale).
  • JAMA Benchmarks: Assessed authorship, attribution, disclosure, and currency (0–4 scale).
• Readability Assessment: Six metrics were calculated using Python and the textstat library: Automated Readability Index (ARI), Gunning Fog Index (GFI), Flesch-Kincaid Grade Level (FKGL), Coleman-Liau Index (CL), Simple Measure of Gobbledygook (SMOG), and Flesch Reading Ease Score (FRES). The target threshold was a sixth-grade reading level (FRES ≥ 80; other indices < 6).
• Statistical Analysis: Quality scores were compared using the Kruskal-Wallis test with effect size ($\epsilon^2$). Readability measures were compared against thresholds using Wilcoxon signed-rank tests.

Key Results

• Information Quality: No statistically significant differences were found among the four models for DISCERN ($P=0.187$), EQIP ($P=0.529$), JAMA ($P=0.392$), or GQS ($P=0.467$) scores.
  • DISCERN: Median scores ranged from 35.00 (Perplexity) to 38.50 (Copilot), falling within the "poor" to "boundary of poor/fair" range, indicating limited treatment information quality.
  • EQIP: Median scores ranged from 52.50 to 55.00, indicating "good" quality with minor problems regarding structure and presentation.
  • GQS: All models achieved a median score of 4.00, suggesting favorable organization and usefulness.
  • JAMA: All models scored a median of 0.00, indicating a near-total absence of authorship, source attribution, disclosure, and update dates.
• Readability: None of the models met the recommended sixth-grade reading level.
  • Median values for all grade-level indices (ARI, GFI, FKGL, CL, SMOG) were substantially higher than 6 (ranging roughly from 13 to 19).
  • Median FRES scores were all below 80 (ranging from 13.04 for Gemini to 30.76 for Perplexity), indicating text that is "very difficult" to read.
  • Model Differences: Perplexity and Copilot generally produced more readable responses (lower complexity scores) compared to Gemini, which exhibited the highest linguistic complexity. However, even the "easiest" responses failed to meet the sixth-grade threshold.

Key Contributions

• Multimodel Comparison: This study provides one of the first systematic comparisons of four leading AI chatbots specifically regarding EPI, a condition with high patient education needs but limited prior AI evaluation.
• Dimensional Separation: The research highlights a critical disconnect between "surface usability" (high GQS scores) and "evidence traceability" (low JAMA scores). It demonstrates that AI responses can be well-organized and fluent while simultaneously lacking verifiable sources and failing to meet basic readability standards.
• Readability Gap: The study quantifies the significant gap between AI-generated medical content and the literacy levels recommended for patient education materials, showing that even models perceived as more accessible (Perplexity, Copilot) still produce text at a college-level complexity.

Significance and Claims
The authors claim that while AI chatbots generate well-structured and broadly usable information about EPI, they currently possess significant limitations in treatment information quality and source transparency. The study concludes that these responses are best used as supplementary information only after professional verification.

The paper asserts that:

1. Current Limitations: The combination of high structural quality (GQS) and low source transparency (JAMA) creates a risk where patients may trust fluent responses without verifying their validity or currency.
2. Readability Barrier: The linguistic complexity of the responses substantially exceeds the recommended level for general patient education, potentially hindering comprehension regardless of the model used.
3. Recommendations for Developers: To improve clinical utility, developers should integrate authoritative citations, indicate the currency of guidelines, generate plain-language content, and adapt responses to patient health literacy levels.
4. Clinical Implications: Healthcare professionals should be aware that patients may use these tools for self-management (e.g., adjusting PERT dosage) and should proactively verify information, correct misconceptions, and provide reviewed materials at appropriate reading levels.

The authors remain modest regarding their findings, noting that the study represents a snapshot of specific model versions and does not validate factual medical accuracy or guideline concordance, emphasizing that quality and readability scores alone cannot establish suitability for independent treatment guidance.