Metric-dependent evaluation of a retrieval-augmented large language model workflow for oral squamous cell carcinoma treatment planning: a retrospective study
This retrospective study demonstrates that the ranking of four RAG-enabled LLMs for oral squamous cell carcinoma treatment planning varies significantly depending on the evaluation metric used, highlighting the limitations of relying on single aggregate scores to determine clinical model performance.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine a doctor sitting down to plan treatment for a patient with oral cancer. This is not a simple task where there is only one correct answer. The decision depends on the exact location of the tumor, how far it has spread, the patient's overall health, and their personal preferences. Even when doctors follow the best medical guidelines, they might choose different but equally valid paths for the same patient. One doctor might recommend surgery first, while another might suggest radiation. Both plans could be medically sound. This complexity makes it very difficult to test whether a computer program can help doctors make these decisions. If a computer suggests a plan that looks different from the one a human doctor wrote down years ago, is the computer wrong? Or is it just offering a different, equally good solution?
This question sits at the heart of a recent study looking at artificial intelligence in medicine. Researchers wanted to see if large language models—computer programs trained on vast amounts of text to understand and generate human language—could help plan treatment for oral squamous cell carcinoma, a common type of mouth cancer. These models are often paired with a system called retrieval-augmented generation. Think of this as giving the computer a library of medical books and guidelines to consult before it writes its answer, rather than letting it rely only on what it remembers from its training. The goal was to see if these AI systems could produce treatment plans that matched the quality of those created by experienced human doctors.
The researchers gathered 100 real cases of oral cancer from a hospital in China. These cases covered patients ranging in age from 29 to 89, with an average age of nearly 60. Most of the patients were men. For each case, the team fed the patient's medical details into four different versions of an AI system. Each system was equipped with the same library of medical guidelines and research papers. The AI then generated a treatment plan for each patient. The researchers then compared these AI-generated plans against the actual treatment plans that human doctors had created for these same patients in the past.
To make this comparison, the team used several different methods. They asked one AI system to act as a judge and score the plans. They used computer programs to measure how similar the AI's words were to the human doctor's words. They also looked at how closely the AI's ideas matched the doctor's ideas using a mathematical measure of meaning. Finally, they had five experienced head-and-neck cancer specialists review the plans blindly, rating them on safety, logic, and usefulness.
The results were surprising and revealed a crucial lesson about how we test medical AI. When the researchers looked at the scores, the "best" AI changed depending on which method they used to measure it. One AI system, called Qwen, received the highest score when judged by another AI. Another system, Doubao, looked most similar to the human plans when measured by how closely their meanings aligned. A third system, Claude, had the most similar wording to the human plans. However, when the human doctors reviewed the plans, they gave a different ranking entirely, favoring two of the other systems.
This finding shows that there is no single, perfect way to say which AI is the best. A program might look great if you count how many words it shares with a human plan, but that doesn't mean it made the right medical decisions. Conversely, a program might use completely different words but still offer a safe and effective treatment plan. The study found that the computer programs were capable of generating plausible treatment plans, but the researchers could not prove that any single one was superior to the others or that they were ready to be used on their own in a hospital.
The researchers were careful to note what their study did not prove. Because they only looked at past records, they could not tell if the AI would actually improve patient outcomes or prevent mistakes in a real-time setting. They also could not separate the effect of the AI's "library" from the AI's own intelligence, so they could not say for sure if the extra medical books made the difference. Furthermore, the study did not test if the AI could handle the full complexity of a real clinic, where doctors and patients interact dynamically.
Ultimately, the study suggests that evaluating medical AI is far more complicated than simply giving it a test score. The way you measure the AI changes which one appears to win. For these tools to be safe and useful in the future, they will need to be tested in more rigorous ways, with human doctors working alongside them to ensure that the plans are not just textually similar to past records, but genuinely safe and helpful for patients. Until then, the most important finding is that we must be very careful about how we decide which computer program is ready to help save lives.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.