Large Language Models for Pulmonary Embolism Health Education: A Four- Model Comparison of Reliability, Quality, and Readability

Technical Summary: Large Language Models for Pulmonary Embolism Health Education

Problem Statement
Pulmonary embolism (PE) is a life-threatening condition requiring timely, reliable, and understandable public information. While Large Language Models (LLMs) are increasingly utilized for health queries, their performance in generating PE-specific patient education has not been systematically compared across multiple models. Existing literature highlights concerns regarding "hallucinations," overconfidence, lack of source transparency, and the use of language that exceeds recommended health literacy levels. There is a critical need to evaluate whether mainstream LLMs can simultaneously provide reliable content, transparent sourcing, and text accessible to the general public (specifically at a sixth-grade reading level).

Methodology
This cross-sectional study compared four publicly accessible web-based LLMs: ChatGPT (GPT‑5.5), Gemini (3.5 Flash), Microsoft Copilot, and Perplexity AI. The study adhered to the CHART (Chatbot Assessment Reporting Tool) reporting guideline.

• Data Collection: Thirty-one high-frequency PE-related queries were selected from worldwide Google Trends data (July 17, 2021 – July 17, 2026) after mapping to Medical Subject Headings (MeSH) and screening for relevance. These queries covered symptoms, risk factors, diagnosis, treatment, prognosis, and special populations.
• Experimental Design: Each query was submitted once to each of the four models in a zero-shot, single-turn session using default settings and cleared browser histories to minimize personalization. This yielded 124 total responses.
• Evaluation Metrics:
  • Reliability and Quality: Assessed using DISCERN (16 items, 16–75 scale), EQIP (20 items, 0–100%), JAMA Benchmark Criteria (4 items for source transparency), and the Global Quality Scale (GQS, 1–5).
  • Readability: Assessed using six indices: Automated Readability Index (ARI), Flesch Reading Ease Score (FRES), Flesch-Kincaid Grade Level (FKGL), Gunning Fog Index (GFI), Coleman-Liau Index (CLI), and Simple Measure of Gobbledygook (SMOG).
• Statistical Analysis: Two independent clinical researchers scored all responses. Differences between models were analyzed using Kruskal-Wallis tests with epsilon-squared effect sizes, followed by Dunn-Bonferroni post hoc comparisons. Readability was compared against the sixth-grade benchmark using Wilcoxon signed-rank tests.

Key Results

• Response Completion: All four models answered every query without refusals or blank outputs.
• Quality and Reliability:
  • DISCERN: Copilot and Perplexity achieved median scores of 41 ("fair" quality), significantly outperforming ChatGPT (35) and Gemini (33), both of which fell into the "poor" range ($P < 0.001$). No model reached the "good" threshold (51+).
  • EQIP: All models generally scored in the "good with minor issues" range (55–75%), with Copilot (75) and Perplexity (70) scoring higher than ChatGPT (60) and Gemini (55).
  • JAMA Benchmark: Copilot and Perplexity demonstrated significantly better source transparency (median 1.0) compared to ChatGPT and Gemini (median 0.0), driven by the display of citations ($P < 0.001$).
  • GQS: There was no significant difference in overall quality ratings among the models (median 4.0 for all, $P = 0.652$).
• Readability:
  • Failure to Meet Standards: No model met the recommended sixth-grade reading level on any of the six readability indices.
  • Grade Levels: Median grade-level indices ranged from approximately 9 to 16. ChatGPT produced the most readable text (median FKGL ~9, FRES ~47.9), while Copilot produced the most complex text (median FKGL ~11, FRES ~34.8).
  • Variability: Significant variation in readability and quality existed within each model depending on the specific query (e.g., guidelines and pregnancy topics were more technical).

Key Contributions

1. Comparative Benchmarking: This study provides the first multi-model comparison of LLMs specifically for PE education, utilizing a standardized set of real-world search queries.
2. Differentiation of Fluency vs. Reliability: The findings demonstrate that high Global Quality Scale (GQS) scores (fluency and organization) do not correlate with source transparency (JAMA) or structural reliability (DISCERN). Models that displayed citations (Copilot, Perplexity) scored higher on transparency but did not necessarily produce more readable text.
3. Readability Gap: The study quantifies the persistent gap between LLM output and health literacy standards, showing that even the "best" performing models generate text significantly above the sixth-grade level recommended by the American Medical Association.
4. Query-Level Variability: The analysis highlights that model performance is not uniform; a model may provide a high-quality answer to one query while failing on another, particularly regarding technical complexity and risk communication.

Significance and Claims
The paper concludes that while mainstream LLMs can generate fluent and broadly structured PE health education, no current model combines reliable content, transparent sourcing, and suitable readability.

• Supplemental Role: The authors assert that LLMs should serve only as a supplement to, not a replacement for, professional medical judgment. They are suitable for low-risk explanatory tasks (e.g., general definitions, lifestyle guidance) but are unsafe for independent diagnosis, risk stratification, or treatment selection.
• Governance and Design: The study calls for model developers to integrate authoritative evidence retrieval, offer health-literacy-adapted language versions, and implement human review cycles.
• Clinical Safety: The authors emphasize that in high-risk conditions like PE, confident prose can obscure weak evidence transparency. Therefore, clinical institutions using AI for patient materials must adopt a "machine generation – professional review – patient testing" workflow to ensure safety, accuracy, and clarity.

The study acknowledges limitations, including the dynamic nature of LLMs (results are time-specific), the use of English-only queries, and the reliance on readability formulas rather than direct patient comprehension testing. Consequently, the findings are presented as an exploratory snapshot of model performance rather than a definitive universal ranking.