Performance of Large Language Models as Assistive Tools for Clinical Pharmacists in the Stomatological Hospital: A Multi-dimensional Comparative Study

In the specialized world of dental hospitals, clinical pharmacists serve as the guardians of medication safety. Their job is to ensure that every drug prescribed for oral diseases is the right choice, the right dose, and safe for the patient's specific health history. This work involves reviewing prescriptions, checking for dangerous drug interactions, and adjusting treatments for patients with complex conditions like kidney disease or liver issues. However, this role is becoming increasingly difficult. The volume of medical information grows daily, guidelines change constantly, and pharmacists often face heavy workloads that leave little time for deep research. They must make quick, accurate decisions while managing patients who may be taking multiple medications for both oral and systemic health problems.

To help manage this pressure, many hospitals are looking toward artificial intelligence, specifically large language models. These are computer programs trained on vast amounts of text that can understand and generate human language. They are designed to retrieve information, analyze complex scenarios, and offer answers to questions that usually require years of professional study. The question facing the medical community is whether these digital tools can truly act as reliable assistants to human experts, or if they are merely sophisticated chatbots that might offer confident but incorrect advice. A new study from researchers in China set out to test this directly, moving beyond general questions to see how well these models perform in the specific, high-stakes environment of a dental hospital.

The researchers selected three of the most advanced large language models available at the time: ChatGPT-5.5 Instant, Gemini-3.5 Flash-Lite, and DeepSeek-V3. To test them fairly, the team created a set of thirty realistic questions that clinical pharmacists actually encounter every day. These questions covered six critical areas: how much of a drug to give, how drugs interact with one another, which patients should never take certain medicines, which drugs cannot be mixed in the same IV bag, how a drug should be administered, and how to adjust doses for patients with failing kidneys or livers. The questions ranged from straightforward facts to complex scenarios requiring deep medical reasoning.

Five senior clinical pharmacists then reviewed the answers provided by each model. They did not know which model generated which answer, ensuring their judgment was unbiased. They scored the responses based on how accurate, complete, and useful they were for a real-world pharmacy setting. The results showed a clear hierarchy in performance. ChatGPT-5.5 Instant produced the highest quality answers, earning a median score that placed it in the "good" to "excellent" range. DeepSeek-V3 followed in second place, while Gemini-3.5 Flash-Lite scored the lowest of the three. The differences were statistically significant, meaning the gap in performance was real and not due to chance.

Interestingly, the difficulty of the questions did not change the ranking. Whether the question was simple or complex, ChatGPT consistently outperformed the others. The study found no evidence that the models got worse as the questions became harder; they maintained their relative positions regardless of the challenge. This suggests that the models' performance is driven by their inherent ability to reason and access knowledge, rather than just their ability to recall simple facts. However, the study also highlighted a significant limitation common to all three models: their readability. The answers they generated were written in a style that was very difficult for the average person to understand, often requiring a college-level education to follow. This is a crucial point for clinical use, as pharmacists need to explain these concepts to patients, and the models currently struggle to simplify their language without losing accuracy.

Another critical finding concerned the sources of information. The study evaluated the quality of information sources provided by the models and found no statistically significant difference in reliability scores among the three. However, the authors emphasized that broader literature in dentistry and related medical fields consistently reports that large language models frequently generate fabricated references, including invented book titles, journal names, and authors. While this specific study assessed source quality, the pervasive issue of hallucinated citations in the wider field means that none of the models are currently safe to use as a standalone source of truth without verification. They cannot replace official drug databases or the judgment of a licensed pharmacist.

The researchers concluded that these artificial intelligence tools show great promise as assistants for clinical pharmacists in dental hospitals. They can handle a wide variety of drug information tasks with a level of quality that is acceptable for professional support. However, they must be viewed as tools to aid human decision-making, not as independent decision-makers. The human pharmacist remains essential to verify the information, check for the latest guidelines, and ensure patient safety. As these models continue to evolve, they may become even more capable, but for now, they serve best as a powerful second pair of eyes for the experts who keep patients safe.