Comparative Reliability of Gemini and NotebookLM on AAPD Guideline-Based Pediatric Dentistry Multiple-Choice Questions: Accuracy, Consistency, and Error-Pattern Analysis

In the world of modern dentistry, especially when treating children, doctors rely on detailed rulebooks to make the right choices. These rulebooks, known as clinical guidelines, act as a map for navigating complex situations like deep tooth decay or infections in baby teeth. They tell practitioners exactly which treatments work best, how long they last, and what to watch out for. However, turning these written rules into quick, reliable answers during a busy day or a classroom lesson is not always simple. Recently, a new type of computer program, called a large language model, has emerged as a tool that can read these texts and answer questions instantly. These programs are like vast libraries that can speak back to you, but they sometimes make mistakes or change their minds if you ask the same question twice. To make them more trustworthy, developers have created a method where the computer is forced to look at a specific document before answering, ensuring it sticks to the facts rather than guessing. The question facing researchers is whether this extra step of checking the document actually makes the computer more reliable, or if it simply makes it look more confident while hiding new kinds of errors.

A team of researchers from Xiangya Stomatological Hospital in China set out to test this idea using a specific set of questions based on the 2024 American Academy of Pediatric Dentistry guidelines for treating baby teeth. They did not ask the computer to treat real patients; instead, they created a simulation where the computer had to answer ninety multiple-choice questions, similar to those found in medical exams. Each question presented a scenario about a child's tooth and offered four possible answers, with only one being the correct choice according to the official guidelines. The researchers wanted to see how two different ways of using the computer performed. The first method was a standard chat, where the computer answered based on everything it had learned from the internet and its training data. The second method used a tool called NotebookLM, which was given the actual guideline document to read before answering, forcing it to ground its response in that specific text. To ensure the results were not just a lucky guess, the researchers asked three different computer accounts to answer every single question, generating hundreds of responses to look for patterns.

The results showed that both methods were quite good at getting the right answer, but they behaved in very different ways. The standard chat version got the majority of the answers correct about ninety-two percent of the time. However, it was inconsistent. When the three different accounts answered the same question, they often disagreed with each other. About one out of every six times, the accounts would give different answers, suggesting that the computer was uncertain or drifting between options. In contrast, the version that read the document first got the right answer nearly ninety-nine percent of the time. More importantly, it was perfectly consistent. Every single time the three accounts answered a question, they gave the exact same response. This suggests that giving the computer the document to read made it much more stable and less likely to change its mind, even though it did not necessarily make it perfect.

However, the researchers found that this stability came with a hidden catch. While the standard chat made mistakes that varied from one attempt to the next, the document-reading version made a mistake that was the same every single time. For one specific question comparing chemomechanical caries removal versus rotary-bur caries removal, all three accounts of the document-reading tool gave the same wrong answer. This is a significant difference in risk. When a computer gives different wrong answers, a user might notice the confusion and realize something is wrong. But when a computer gives the same wrong answer every time, and does so while citing a document, it looks very authoritative and reliable. The error is consistent and confident, making it much harder for a student or a doctor to spot that the information is incorrect. The study found that these errors tended to happen on questions that required comparing numbers, like success rates or the effectiveness of different tools, rather than simple facts.

The study concludes that simply asking a computer to read a document does not guarantee it will understand the information perfectly. Instead, the main benefit of this method is that it stops the computer from being unpredictable. It creates a system that is steady and repeatable, which is valuable for education and training. Yet, this steadiness means that if the system does get something wrong, that error will be repeated consistently, potentially misleading users who trust the computer too much. The researchers suggest that we should not just look at how often a computer gets the right answer, but also look at how it behaves when it is wrong. In the future, tools used for medical decisions need to be evaluated not just on their accuracy, but on whether their mistakes are obvious or hidden, and whether they can be trusted to stay the same when asked again. This approach helps ensure that as we bring these powerful tools into classrooms and clinics, we understand both their strengths and the specific ways they might fail.