📄 medicine

Can Large Language Models Provide High-Quality Information for Patients with Multidrug-Resistant Organism Infections: A Multidimensional Comparative Analysis of DeepSeek-V3, ChatGPT-5.2, Gemini-3, and Grok-4

This study evaluates four large language models on their ability to generate patient education content for multidrug-resistant organism infections, finding that while Grok-4 achieved the highest scores in accuracy and safety, no model met acceptable standalone standards, necessitating mandatory expert review and human oversight before clinical use.

Original authors: Yongxin Ma, Lei Li, Yige Shi, Zebing Ma, Mengyao Liu, Yingying Wang, Huayu Fan, Xiangyang Cao, Rui Chen

Published 2026-07-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yongxin Ma, Lei Li, Yige Shi, Zebing Ma, Mengyao Liu, Yingying Wang, Huayu Fan, Xiangyang Cao, Rui Chen

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a patient in a hospital, and you've been told you have a "superbug" infection (a multidrug-resistant organism, or MDRO) that is hard to treat. You are scared, confused, and have a million questions: How did I get this? How do I keep my family safe? What does this mean for my future?

In the past, you would have to wait for a busy nurse to explain everything. But today, many people turn to AI chatbots (Large Language Models) for answers. This paper asks a critical question: If you ask four of the smartest AI chatbots about these superbugs, will they give you safe, accurate, and easy-to-understand advice?

The researchers treated this like a taste test, but instead of ice cream, they were testing "health advice." Here is what they found, explained simply.

The Contestants

The study pitted four popular AI models against each other:

DeepSeek-V3
ChatGPT-5.2
Gemini-3
Grok-4

They asked these AIs ten specific questions that real patients and families actually ask, such as "How do I protect my family?" and "What is the prognosis?"

The Judges

To grade the answers, the researchers didn't just use a computer. They hired a panel of seven human experts (doctors, nurses, and infection control specialists). Think of them as strict food critics. They rated the AI answers on five things:

Accuracy: Is the medical fact correct?
Comprehensiveness: Did it cover everything important?
Safety: Is the advice dangerous?
Humanistic Care: Is it kind and empathetic?
Practicality: Can a regular person actually do what the AI suggests?

The Results: Who Won?

1. The "Smartest" but "Scariest" (Grok-4)

The Good: Grok-4 got the highest scores for accuracy, safety, and covering all the details. It was the most "knowledgeable" of the group.
The Bad: Its tone was a bit cold and negative. It sounded like a strict teacher warning you about doom. While it was right, it didn't make the patient feel very hopeful.

2. The "Most Polite" but "Least Accurate" (DeepSeek-V3)

The Good: This AI was the most cheerful and encouraging. It sounded like a supportive friend.
The Bad: While it was kind, it wasn't the top scorer for safety or accuracy compared to Grok-4.

3. The "Too Complicated" (ChatGPT-5.2)

The Problem: This model gave the lowest scores overall. It wrote in such complex, academic language that it was like reading a university textbook.
The Analogy: If the other AIs were explaining a recipe, ChatGPT-5.2 was explaining the chemistry of the flour. It was so hard to read that a regular person might get lost and give up.

4. The "Middle Ground" (Gemini-3)

It fell somewhere in the middle, but like the others, it struggled with being both simple and kind.

The Big Problem: The "Reading Level" Gap

The researchers checked how hard the text was to read.

The Rule: Health advice should be written at a 6th-grade reading level (like a middle schooler) so everyone can understand it.
The Reality: None of the AI models met this standard. Even the "easiest" one was written at a level that required a high school or college education to fully grasp.
The Metaphor: Imagine trying to explain how to change a tire to a child using advanced engineering terms. Even if the instructions are technically correct, the child won't be able to fix the tire. That is what happened here. The AIs were too smart for their own good.

The "Human Touch" Missing

The experts noticed that all the AIs were a bit robotic. They gave facts but lacked empathy.

Patients with superbugs often feel isolated, ashamed, or scared.
The AIs didn't do a good job of saying, "It's okay, we are here to help you," or "You are doing a great job." They just listed rules. This is dangerous because if a patient feels scared or judged, they might not listen to the safety rules.

The Final Verdict: Can We Trust Them?

The short answer is: Not yet, not on their own.

The paper concludes that while these AI models are powerful tools, they are not ready to replace doctors or nurses for patient education.

The Risk: If a patient reads a complex, slightly inaccurate, or overly scary answer from an AI, they might make bad decisions, panic, or stop following safety rules.
The Solution: The authors suggest a "Three-Step Sandwich" approach:
1. Draft: Let the AI write the first draft (it's fast and knows a lot).
2. Review: A human expert (doctor/nurse) must check and fix the facts and safety warnings.
3. Translate: A human must rewrite the text to make it simple (6th-grade level) and kind (empathetic).

Summary

Think of these AI models as very knowledgeable but slightly clumsy interns. They have read every medical book in the library, but they don't know how to talk to a scared patient, they write in confusing language, and they sometimes miss the emotional nuance.

Until we can teach them to be simpler, kinder, and more careful, they should only be used as assistants to help doctors, never as the final voice of truth for patients.

Technical Summary: A Multidimensional Comparative Analysis of LLMs for MDRO Patient Education

Problem Statement
The rise of multidrug-resistant organism (MDRO) infections poses a critical threat to public health and medical safety, necessitating rigorous infection prevention and control (IPC) measures that rely heavily on patient and family compliance. However, a significant cognitive gap exists in current health education regarding MDROs, exacerbated by the complexity of professional terminology, the fast-paced clinical environment, and limited human resources. While Large Language Models (LLMs) offer a potential solution for scalable patient education, their reliability in high-risk clinical communication remains uncertain due to risks of "hallucinations," inaccurate medical advice, and inappropriate emotional tones. This study addresses the core concern of whether current leading LLMs can provide high-quality, safe, and comprehensible information for patients with MDRO infections.

Methodology
The study employed a quantitative, comparative design to evaluate four leading LLMs: DeepSeek-V3, ChatGPT-5.2, Gemini-3, and Grok-4.

Data Generation: Ten core patient education questions were formulated based on semi-structured interviews with MDRO patients/families and expert consultation. These covered basic concepts, transmission, prevention, treatment, and caregiver support. The models generated responses using a standardized prompt to ensure structural consistency.
Content Quality Evaluation: Seven independent experts (2 physicians, 1 nurse, 4 IPC professionals) with 10–25 years of experience evaluated the model outputs using a 5-point Likert scale across five domains: Accuracy, Comprehensiveness, Safety, Humanistic Care, and Practicality. Reliability was confirmed via Intraclass Correlation Coefficient (ICC).
Readability Analysis: Text complexity was assessed using three metrics: Flesch-Kincaid Grade Level (FKGL), Gunning Fog Index (GFI), and Flesch Reading Ease Score (FRES). These were compared against the American Medical Association's (AMA) recommended sixth-grade reading level for patient materials.
Sentiment Analysis: Lexicographical sentiment analysis was performed using the sentimentr package in R to determine the emotional tone (positive, neutral, negative) of the generated texts.
Statistical Analysis: The Friedman test was used to assess overall differences among models, followed by Wilcoxon signed-rank tests with Bonferroni correction for pairwise comparisons.

Key Results

Content Quality: Significant differences were observed across all five evaluation domains ( $p < 0.001$ ). Grok-4 demonstrated the highest overall performance, achieving top scores in Accuracy (4.13 ± 0.64), Comprehensiveness (4.31 ± 0.62), Safety (4.16 ± 0.58), Humanistic Care (3.51 ± 0.82), and Practicality (4.09 ± 0.84). ChatGPT-5.2 received the lowest ratings across several domains, particularly in safety and humanistic care.
Readability: All models produced text that exceeded the recommended sixth-grade reading level, indicating high complexity. ChatGPT-5.2 had the highest reading level (FKGL 15.72), making it the most difficult to understand. Gemini-3 and DeepSeek-V3 were relatively more readable but still surpassed the target benchmark.
Sentiment: DeepSeek-V3 exhibited a positive emotional tendency (0.064), while ChatGPT-5.2 remained neutral (0.013). Conversely, Grok-4 (−0.051) and Gemini-3 (−0.087) displayed negative tendencies, likely due to a focus on disease severity and risk.
Humanistic Care: A critical finding was the generally low scores for humanistic care across all models. Even the top-performing models lacked sufficient emotional buffering and patient-centered communication strategies, which are vital for reducing the stigma and anxiety associated with isolation.

Key Contributions

Empirical Benchmarking: The study provides the first multidimensional comparative analysis of specific LLM versions (including DeepSeek-V3 and Grok-4) specifically tailored to the high-stakes context of MDRO patient education.
Identification of Gaps: It highlights a critical disconnect between the technical accuracy of LLMs and their suitability for patient communication, specifically noting deficiencies in readability (too complex) and humanistic tone (insufficient empathy).
Risk Assessment: The research underscores the potential risks of unregulated LLM use in IPC, where inaccurate or overly negative information could lead to non-compliance, self-medication errors, or increased patient anxiety.

Significance and Claims
The paper concludes that while LLMs possess a viable foundation for generating structured educational content, no single model currently meets acceptable standards for independent use in the high-risk environment of MDRO patient education. The authors assert that LLMs should be employed solely as tools for assisted drafting, not as substitutes for professional decision-making.

The study proposes a three-tiered application framework for the safe integration of LLMs:

Initial Draft Generation: Using models to create structured text efficiently.
Professional Proofreading: Mandatory expert review to ensure accuracy, policy alignment, and safety.
Language and Emotional Optimization: Human intervention to simplify language (lowering readability scores) and inject appropriate empathy.

The authors emphasize that without mandatory expert review and contextual adaptation, the deployment of LLMs in this domain risks compromising scientific rigor, patient safety, and the trust essential for effective infection control. Future research is directed toward optimizing readability, refining emotional communication mechanisms, and developing scenario-specific model selection strategies.