Large language models differ in patient education quality across the robot-assisted radical prostatectomy care pathway

When a man is diagnosed with prostate cancer, the path forward often involves a major decision: whether to remove the entire prostate gland using a robot-assisted surgical technique. This procedure, known as robot-assisted radical prostatectomy, is a standard treatment for localized disease, but it comes with a complex web of information a patient must understand. They need to know how the surgery works, what recovery looks like, how it might affect their ability to control their bladder or maintain sexual function, and how doctors will watch for the cancer returning. In the past, patients relied on printed brochures or a single conversation with their surgeon to navigate these questions. Today, many turn to artificial intelligence chatbots for instant answers. These large language models can generate text that looks and sounds like human advice, but the critical question remains: can they be trusted to explain such a life-altering medical journey accurately and clearly?

A team of researchers from the Affiliated Hospital of North Sichuan Medical College set out to test this very question. They focused on three popular artificial intelligence systems: ChatGPT, DeepSeek, and Doubao. The team did not ask the machines to perform surgery or diagnose a specific person; instead, they treated the models as information providers. They compiled a list of twenty-five distinct questions that a patient might ask at different stages of the care process, from the initial screening for cancer to the long-term follow-up after surgery. These questions covered everything from interpreting test results to understanding the risks of nerve damage. The researchers then fed each of these twenty-five questions into all three models, generating a total of seventy-five separate responses. To ensure a fair test, the questions were asked individually in separate conversations so that the models could not learn from previous answers in the same chat session.

Once the responses were collected, the researchers needed a way to judge their quality. They recruited three medical experts who acted as blind reviewers, meaning they did not know which artificial intelligence had written which answer. These experts evaluated the text using several established tools designed to measure the quality of patient education. They looked at whether the information was complete, whether it offered practical advice a patient could actually use, and whether it was written in a way that was easy to understand. They also checked for specific errors, such as made-up facts or outdated medical recommendations, and calculated how difficult the text would be for an average person to read. The goal was to see if one machine consistently outperformed the others or if they all struggled with the same issues.

The results revealed that while all three models produced generally safe and accurate information, they had distinct personalities and strengths. One model, ChatGPT, consistently provided the most actionable advice. When a patient asked what to do next, this model was more likely to give clear, step-by-step instructions that a person could follow. It also scored higher on measures of how well the information was structured to support a patient's decision-making. Another model, DeepSeek, produced text that was noticeably easier to read. Its sentences were slightly less complex, and its vocabulary was somewhat more accessible than the others. However, even the "easier" text from DeepSeek was still written at a level that would be difficult for many adults to read comfortably, exceeding the reading level typically recommended for public health materials. The third model, Doubao, performed similarly to the others in terms of accuracy but struggled more with readability and occasionally left out key details.

Despite these differences, the study found that no single model was perfect, and none should be used as a replacement for a doctor's advice. The researchers discovered that while the machines rarely made up facts or gave dangerously outdated advice, they did sometimes omit important information, particularly in the responses from DeepSeek and Doubao. Furthermore, the human experts who graded the answers did not always agree with each other. This lack of agreement suggests that judging the quality of medical advice is a difficult task, even for trained professionals, and that small differences in scores between the models might not be as meaningful as they appear. The study also highlighted a persistent challenge: all three models generated text that was too complex for the average person. The medical terms and sentence structures required a high school education or better to fully grasp, which creates a barrier for many patients who need this information most.

The researchers concluded that these artificial intelligence tools show promise as helpers for doctors, perhaps serving as a first draft for educational materials that a urologist can then review and refine. They can quickly generate explanations that cover a wide range of topics, but they cannot yet replace the nuanced, personalized conversation that happens between a surgeon and a patient. A robot can list the risks of surgery, but it cannot look a patient in the eye, understand their specific fears, or weigh the unique balance of their medical history and personal values. For now, the best use of these powerful machines is as a support tool for clinicians, not as an independent source of medical guidance. The study serves as a reminder that while technology can make information more available, the human element of care remains essential for ensuring that information is not just accurate, but also truly understood.