Safety, Accuracy, Empathy, Information Quality, and Readability of Large Language Models in Answering Caregiver Questions About Pediatric Burn First Aid and Inpatient Care: A Cross-Sectional Comparative Study
This cross-sectional comparative study evaluated four large language models on safety, accuracy, empathy, information quality, and readability regarding pediatric burn care, finding that while they generally provided safe and accurate responses, significant variations existed in empathy, source transparency, and readability that necessitate improvements for effective caregiver guidance.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
When a young child suffers a burn, the first few minutes are critical. The way a parent or caregiver responds in that moment of panic can determine how deep the injury goes, how much pain the child feels, and how well the skin heals later. Standard medical advice is clear: stop the burning process, cool the wound with running tap water, cover it loosely, and seek professional help for serious injuries. Yet, in the heat of the moment, a worried family member might turn to a digital assistant for immediate guidance. These artificial intelligence tools, known as large language models, have become common sources of health information. They can chat, explain complex topics in simple terms, and offer reassurance. But when a child's health is at stake, the difference between a helpful suggestion and a dangerous mistake can be life-altering. The question is not just whether these machines can answer correctly, but whether they can do so safely, with the right amount of compassion, and in language that a stressed parent can actually understand.
A team of researchers at Beijing Children's Hospital set out to test exactly this. They wanted to see how four popular artificial intelligence chatbots performed when asked questions that a parent might have about treating a child's burn. The study focused on two main phases of care: the immediate first aid needed right after the injury, and the ongoing nursing care required while the child is in the hospital. The researchers did not ask the machines to diagnose specific patients or treat real injuries. Instead, they created a set of thirty-six questions that reflected real-world concerns, such as how to cool a burn, what to do if clothing sticks to the skin, how to manage pain, and when to rush to the emergency room. They fed these same questions into four different chatbots: ChatGPT, Gemini, DeepSeek, and Doubao. Each machine generated a response, resulting in a total of one hundred and forty-four answers to evaluate.
To judge these answers, the researchers assembled a panel of five experts, including specialists in pediatric burns, surgery, and nursing. These experts had decades of experience and were trained to look for specific things. They checked if the advice was safe, meaning it did not recommend harmful home remedies or delay necessary medical care. They measured accuracy by comparing the answers to established medical guidelines. They also rated how empathetic the responses were, looking for language that acknowledged a parent's fear and offered support without blame. Finally, they analyzed how easy the text was to read, ensuring the information was not buried in confusing medical jargon. The experts worked independently and did not know which chatbot had written which answer, ensuring their judgments were fair and unbiased.
The results showed that these artificial intelligence tools are generally quite capable, but they are not perfect. Most of the time, the chatbots provided safe and accurate information. For instance, ChatGPT gave safe advice in thirty-three out of thirty-six cases, while the other models ranged from twenty-nine to thirty-one safe responses out of thirty-six. However, the study found that no single model was flawless. In several instances, the answers lacked crucial warnings about when to seek urgent medical help, or they failed to emphasize enough that certain home treatments, like using ice, could actually make the injury worse. While the differences in factual accuracy between the four models were statistically small, the differences in how they communicated were much more noticeable. One model, Doubao, received lower scores for empathy, meaning its responses felt less supportive to a distressed parent compared to the others.
The study also highlighted a significant barrier in how these machines speak to people. The researchers measured the reading level of the text, which is like checking the grade level a student needs to understand the material. They found that the answers from some models were written at a level that would be difficult for many adults to read quickly, especially when they are already stressed. The models that performed best in terms of readability produced text that was simpler and more direct, closer to the level recommended for patient education materials. In contrast, other models used more complex sentence structures and vocabulary that might confuse a caregiver in a crisis. The researchers concluded that while these tools are becoming reliable sources of information, they still struggle with the human side of the equation. They can provide facts, but they sometimes miss the mark on clarity and emotional support.
Ultimately, the study suggests that these digital assistants are useful tools, but they should not replace professional medical judgment. The best performance came from models that combined factual correctness with clear, plain language and a tone that respected the user's anxiety. The researchers noted that for these tools to be truly helpful in pediatric care, they need to be designed to prioritize concise, action-oriented guidance. They must be able to clearly state when a situation is an emergency and explain the evidence behind their advice without hiding it in technical terms. Until these systems can consistently deliver information that is safe, accurate, and easy to understand under pressure, parents and caregivers should treat them as a starting point for information, not the final word on a child's health.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.