Comparative evaluation of large language models for guideline-based oral cancer management: a standardized assessment of GPT-4o, DeepSeek-R1, and Grok-3

Cancer of the mouth is a serious condition that requires precise, coordinated care. When a patient is diagnosed, doctors must decide on a treatment path that often involves surgery, radiation, or medication, depending on exactly where the tumor is and how far it has spread. These decisions are guided by strict medical rules known as clinical guidelines, which are written by experts to ensure every patient receives the best possible care based on the latest science. Recently, a new type of computer program called a large language model has emerged. These programs can read vast amounts of text and write fluent answers to questions, leading many to wonder if they could help doctors summarize complex rules or suggest treatment plans. However, just because a computer speaks clearly does not mean its medical advice is correct. In the high-stakes world of cancer care, a confident but wrong answer could be dangerous, making it essential to test whether these digital tools can truly understand and apply medical guidelines.

A team of researchers from Jilin University in China set out to test this idea by putting three of the most advanced large language models to work on oral cancer cases. They chose three specific models: GPT-4o, DeepSeek-R1, and Grok-3. To see how well these programs performed, the researchers created a standardized test based on the official National Comprehensive Cancer Network guidelines for head and neck cancers. The test had two main parts. First, the models were asked to act as expert surgeons and explain the rules for treating mouth cancer to a person with no medical training. They had to cover the causes, how the disease is found, and the steps for treatment and recovery. Second, the models were given standardized oral cancer scenarios and clinical vignettes and asked to provide a full management plan, including what the diagnosis should be, how to stage the disease, and exactly what surgery or radiation would be appropriate.

To judge the answers, three experienced oral and maxillofacial surgeons reviewed the outputs. They scored the responses on a scale of zero to ten for the summary task and on a scale of zero to one hundred for the case management task, looking at factors like accuracy, clarity, and whether the treatment plan followed the official rules. The results showed that the models performed differently, and no single program was perfect at everything. In the task of summarizing the guidelines for a general audience, the DeepSeek-R1 model received the highest overall score of 8.8 out of 10, followed by GPT-4o with 8.3 and Grok-3 with 7.8. DeepSeek-R1 also scored highest in specific areas like clarity and logical flow.

When the models moved on to the more difficult task of managing specific patient cases, the scores were higher overall, but the differences between the models became more distinct. DeepSeek-R1 again achieved the highest comprehensive score of 87.44 out of 100. It performed particularly well in suggesting the right treatment plans, planning the extent of surgery, and assessing the prognosis, or likely outcome, for the patient. Grok-3 came in second with a score of 77.77, and GPT-4o received a score of 68.61. Interestingly, while DeepSeek-R1 led in most categories, Grok-3 actually received the highest score for suggesting the correct radiation dose. The researchers also noted that the text similarity scores, which measure how closely the computer's words matched a reference answer, did not always align with the medical accuracy judged by the human experts. This suggests that a computer can write words that look very similar to the right answer but still get the medical details wrong.

Despite the high scores achieved by the best-performing model, the study found significant flaws that prevent these tools from being used as independent decision-makers. All three models sometimes confused cancer of the mouth with cancer of the throat, a critical error because the two diseases are treated differently. They also occasionally invented specific details, such as exact surgical margins or antibiotic schedules, that were not supported by the patient's case or the official guidelines. The researchers observed that while the models could organize information well, they lacked the deep, contextual understanding required to tailor a plan to a specific patient's unique situation. The study concludes that these artificial intelligence tools should be viewed as assistants that help doctors draft plans or summarize information, rather than as autonomous doctors who can make the final call. The authors emphasize that human experts must always verify the output, and they call for larger studies to further refine how these technologies can safely support cancer care.