Comparative Performance of Large Language Models in Answering Guideline-Based Questions on Juvenile Idiopathic Arthritis: A Blinded Expert Evaluation

In the world of childhood medicine, there is a condition called juvenile idiopathic arthritis. It is a group of diseases where a child's joints become swollen and painful for no clear reason, and it can affect how they grow and move. Because the condition comes in many different forms, doctors must be very careful to identify exactly which type a child has and choose the right treatment. The rules for making these decisions are written down in detailed guides created by medical experts. These guides tell doctors when to start medicines, how to watch for side effects, and how to handle complex situations like infections or eye inflammation. For years, these guides have been the trusted map for doctors, but a new kind of tool has appeared on the horizon: large language models. These are computer programs trained on vast amounts of text that can write answers to questions in fluent, human-like language. They are becoming popular for finding medical information, but no one knew if they could be trusted to follow the strict rules of pediatric arthritis care or if they might invent facts that sound right but are actually wrong.

A team of researchers in Turkey decided to put these computer programs to a serious test. They wanted to see if three of the most advanced models available could answer questions about juvenile idiopathic arthritis as well as a human expert would. To do this, they did not ask the computers to write stories or chat about feelings. Instead, they gave them twenty-one specific questions based on the official medical guides. Some questions asked for simple facts, like how to tell different types of the disease apart. Others asked about treatment plans, such as what medicine to give a child who is not getting better. The most difficult questions were hypothetical stories about patients, asking the computers to figure out the best next step for a specific child with a complex history. The researchers asked three different computer models to answer every single question. To make sure the test was fair, they used neutral labels to hide the names of the computers and mixed up the order in which the answers were shown.

Five expert doctors, specialists in children's joint diseases, then read these answers. They did not know which computer wrote which answer. They scored each response on how accurate it was compared to the official guides, how useful it would be in a real doctor's office, and how good the overall answer was. The results showed a clear difference between the machines. One model, which the researchers called GPT-5.6 Sol, consistently gave the best answers. It scored the highest for accuracy and for being useful to a doctor. The other two models, Gemini 3.1 Pro and Claude Opus 5, performed well but fell short of the leader. In some areas, like diagnosing the disease, one of the weaker models did better than the other, but in other areas, like treatment plans, the order flipped. This proved that no single computer was perfect at everything, and their performance changed depending on the type of question being asked.

The study also looked at how the answers were written. One of the models produced very long responses, writing thousands of words across the twenty-one questions. The researchers expected that longer answers might be better, but the doctors did not agree. The model that wrote the most words did not get the highest scores. In fact, the best-scoring model was often more direct. The doctors also noticed that the language used by all the computers was quite difficult to read, written at a level that would be hard for a typical adult to understand without a medical background. This suggests that while these tools can generate fluent text, they do not always know how to simplify complex medical rules for a general audience.

The most important finding of this research is that these computer programs cannot replace a human doctor. While they can retrieve information and organize it into a coherent answer, they sometimes blur the line between what is a strict rule and what is just a common feature of the disease. They might suggest a treatment option that is possible but not the best one, or they might mix up the details of how a disease is classified. The researchers concluded that these tools should only be used under the supervision of a specialist who can check the facts. The study showed that even the best computer model needs a human expert to verify its work, especially when the health of a child is at stake. The technology is powerful, but in the delicate field of pediatric rheumatology, it remains a tool to assist, not a decision-maker on its own.