Quality and Consistency of Four Large Language Models in Answering AAOS Guideline-Based Questions on Knee Osteoarthritis and Total Knee Arthroplasty: A Blinded Multi-Rater Benchmark Study in Chinese
This blinded multi-rater benchmark study evaluated four large language models on AAOS guideline-based questions regarding knee osteoarthritis and total knee arthroplasty, finding that while GPT demonstrated the highest overall quality and consistency, the performance ranking varied by evaluation domain and was sensitive to the statistical methods used.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
When a person faces a painful, stiff knee, the path to relief is often paved with complex medical advice. Doctors rely on established guidelines to decide whether a patient should try physical therapy, take medication, or undergo surgery to replace the joint. These rules are built on years of research and are designed to keep patients safe while offering the best chance for recovery. In recent years, people have increasingly turned to artificial intelligence to make sense of this information, asking computer programs to explain treatment options, risks, and recovery times in plain language. However, just because a computer speaks fluently does not mean it is telling the truth or staying consistent. If a machine gives a different answer every time you ask the same question, or if it confidently suggests a treatment that experts know is dangerous, the tool becomes a source of confusion rather than help.
A team of researchers in China set out to test how well four different artificial intelligence systems handle these critical medical questions. They focused on knee osteoarthritis and total knee replacement, two of the most common conditions treated by orthopedic surgeons. The team did not simply ask the computers for opinions; they created a rigorous test based on official guidelines from the American Academy of Orthopaedic Surgeons. They wrote twenty-four specific questions covering everything from non-surgical treatments like weight loss and exercise to surgical details about how to manage risks during an operation. To ensure a fair test, they asked each of the four AI models to answer every question three separate times, treating each attempt as a fresh conversation with no memory of the previous one. This approach allowed them to see not only if the answers were correct, but also if the machines could be trusted to give the same reliable advice again and again.
The answers generated by the computers were then handed to eight human experts who specialize in knee disorders and joint surgery. These doctors and specialists read the responses without knowing which computer wrote them. They scored each answer on a scale of one to five based on four key qualities: how relevant the answer was to the question, how medically accurate it was, how easy it was to understand, and whether it was safe. Safety was a major concern, meaning the experts looked to see if the computer avoided giving dangerous advice or failed to mention important risks. The experts also gave a separate score for consistency, rating whether the three answers from the same computer for the same question were essentially the same story or if they contradicted each other.
The results showed that the four artificial intelligence systems performed differently, and no single machine was the clear winner in every category. One model, known as GPT, produced the highest overall scores. It was particularly strong at providing accurate medical information, staying on topic, and including necessary safety warnings. Another model, DeepSeek, scored slightly lower on overall quality but was rated as the most understandable by the experts, meaning its language was the clearest and easiest to follow. The other two models, ERNIE and Kimi, received lower scores across the board, particularly in accuracy and safety. The study found that the gap between the top performer and the second-best was small and depended on how the data was analyzed, suggesting that while one machine might have a slight edge, the difference is not a massive, unshakeable fact.
Perhaps just as important as the quality of the answers was the consistency of the machines. The researchers found that the top two models were much better at giving the same answer when asked the same question three times in a row. The leading model was the most consistent, followed closely by the second-best model. The other two systems were more likely to change their minds or alter their advice between attempts. This matters because a patient asking for advice today might get a different answer tomorrow if they use a less stable system, which could lead to confusion or poor decision-making. The experts noted that while the computers were generally good at explaining the guidelines, they were not perfect, and their performance varied depending on whether the question was about non-surgical care or the details of surgery.
The study concludes that while these artificial intelligence tools show promise for helping patients understand their knee conditions, they are not yet ready to replace a doctor's judgment. The best-performing models offered high-quality, safe, and clear information, but the differences between them were subtle and specific to certain strengths. One machine might be better at facts, while another is better at clear language. The researchers emphasized that these findings reflect how the computers performed on a written test and do not prove that using them will improve a patient's actual health or recovery. For now, the most reliable approach is to view these tools as helpful assistants that can provide a starting point for conversation, but whose advice must always be checked by a medical professional before making any decisions about treatment.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.