Comparison of ChatGPT and Gemini in Providing Postoperative Information After Tonsillectomy: An Expert-Based Assessment
An expert-based assessment comparing ChatGPT and Gemini for postoperative tonsillectomy education in Turkish found no statistically significant difference in overall quality or safety between the two models, underscoring the necessity for clinician review before using AI-generated patient information.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Every year, millions of people undergo a tonsillectomy, a common surgery to remove the tonsils at the back of the throat. While the procedure is routine, the recovery is often difficult. Patients wake up with significant pain, trouble swallowing, and a constant worry about bleeding or dehydration. Because most of this recovery happens at home, families rely on clear instructions to know what is normal and what requires a doctor's help. In recent years, many have turned to artificial intelligence tools to get these answers instantly. These computer programs, known as large language models, can hold conversations and generate text that sounds very human. They promise to be available twenty-four hours a day to explain medical terms or offer advice. However, because these programs generate answers based on patterns in data rather than a deep understanding of medicine, there is a risk they could give advice that is confusing, outdated, or even dangerous. The question facing doctors and patients is simple: can we trust these machines to guide a family through a difficult recovery, or do they need a human expert to check their work?
To answer this, a team of researchers in Turkey set up a careful test involving two of the most advanced artificial intelligence systems available: ChatGPT and Gemini. They did not ask the computers random questions. Instead, they started with real people. The researchers collected questions from forty patients who had recently had their tonsils removed. These patients wrote down exactly what they wanted to know about their recovery, covering topics like pain management, diet, and when to seek emergency help. A panel of ten expert doctors then refined these into fifteen clear questions in Turkish. On a single day, the researchers asked both ChatGPT and Gemini to answer these same fifteen questions. The computers generated their responses, and the researchers saved them exactly as they appeared, without changing a single word.
Next, the researchers brought in sixteen different ear, nose, and throat specialists to act as judges. These doctors did not know which computer program wrote which answer. They simply saw two sets of responses, labeled only as "Response A" and "Response B." Each doctor read the answers to all fifteen questions and rated them on four specific things: how scientifically accurate the information was, whether the advice was appropriate for a real patient, how easy it was to understand, and most importantly, whether the answer was safe or if it contained anything that could mislead a patient into harm. The doctors gave a score from one to five for each category, with five being the best possible rating. In total, the team gathered nearly two thousand individual ratings to compare the two systems side by side.
The results showed that both artificial intelligence programs performed quite well overall. They provided information that was generally accurate and clinically appropriate, earning high marks for getting the medical facts right. When the researchers looked at the total scores, ChatGPT received a slightly higher average rating than Gemini, but the difference was so small that it could easily have happened by chance. In fact, when the experts looked at the specific categories, the two programs were nearly identical in their ability to provide correct medical facts. Neither system proved to be clearly better than the other in a way that the data could confirm with certainty. The study found no evidence that one model was superior to the other for this specific task.
However, a closer look revealed a more important story about safety. Both programs received their lowest scores in the category of patient safety. This means that while the answers were often factually correct, the experts still spotted potential risks, such as missing warnings about bleeding or giving vague advice on medication. Even though ChatGPT scored slightly higher than Gemini in safety, the difference was not significant enough to declare it the safer choice. The experts also noted that the doctors did not always agree with each other when rating safety, suggesting that spotting a hidden danger in a paragraph of text is difficult even for professionals. This lack of perfect agreement highlights that safety is a complex judgment, not just a simple check of facts.
The study concludes that while these artificial intelligence tools are impressive and can generate useful information, they are not yet ready to replace a doctor's advice. The fact that both systems received lower safety scores suggests that a human expert must always review the information before it is given to a patient. The research does not show that one program is the clear winner, nor does it prove that they are perfectly safe to use on their own. Instead, it suggests that these tools work best as helpers that provide a draft of information, which a medical professional then checks and corrects. Until the technology improves to the point where it can consistently identify and avoid all potential risks without human help, the safest path for patients remains a conversation with their own doctor.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.