📄 medicine

Large Language Models as Decision-Support Tools for Laser Dentistry: A Blinded, Expert-Rated Comparison

In a blinded, expert-rated comparison of five AI platforms using 24 synthetic laser-dentistry vignettes, a medical-domain retrieval-augmented generation (RAG) system and a leading general-purpose large language model significantly outperformed other tools in accuracy, safety, and completeness, while a general-purpose RAG platform performed the worst.

Original authors: Yaniv Mayer, Georgi Tomov, Eran Gabay, Sharonit Sahar Helft

Published 2026-08-26

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yaniv Mayer, Georgi Tomov, Eran Gabay, Sharonit Sahar Helft

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Dentists have long used lasers as precise tools to treat gum disease, clean root canals, and whiten teeth. Unlike a drill that grinds away, a laser beam can target specific tissues, such as the soft pink gums or the hard white enamel, by tuning its light to interact with water or blood. However, choosing the right laser is a delicate task. If the light is too strong or the wrong color, it can burn the tooth's nerve or damage the eye. If it is too weak, the treatment fails. Because there is no single, universal rulebook for every situation, dentists must carefully calculate the power, the speed of the pulses, and the cooling methods for each patient. This complexity has led some to wonder if artificial intelligence could act as a guide, offering instant advice on how to set these machines safely and effectively.

To test this idea, researchers at the Technion and New Bulgarian University set up a controlled experiment involving five different artificial intelligence platforms. They did not ask the computers to diagnose real patients, but rather to solve twenty-four carefully crafted, fictional dental cases. These scenarios covered three main areas of laser dentistry: gum and surgical work, root canal treatments, and restorative procedures like crowns or fillings. For each case, the team asked the same six-part question: Is a laser the right tool? Which type of laser should be used? What are the exact settings for power and timing? What is the step-by-step plan? What safety measures are needed? And what is the expected result? The five platforms tested included three general-purpose chatbots that are widely known, one system designed specifically to search medical journals, and another that searches the general internet for answers.

The results of this comparison were clear and revealed significant differences in performance. The researchers enlisted three expert dentists, each specializing in one of the three areas, to grade the answers without knowing which computer generated them. The experts scored the responses on a scale of one to five, looking for accuracy, safety, and completeness. The findings showed that not all artificial intelligence is created equal when it comes to medical advice. The system that searched through peer-reviewed medical literature, called OpenEvidence, and a top-tier general chatbot named Claude, provided the best guidance. They consistently offered the most accurate settings and the safest protocols. In contrast, the system that searched the open internet, Perplexity, performed the worst. It produced the highest number of answers that contained dangerous errors or missing critical steps. In fact, more than half of the responses from the internet-searching tool contained at least one element that an expert would consider poor or potentially harmful, whereas the medical literature system never produced a single unsafe answer.

The study also highlighted that the quality of the advice depended heavily on the domain of dentistry. The differences between the best and worst platforms were most dramatic in root canal and restorative cases, where the settings must be precise to avoid burning the tooth's nerve. In gum disease cases, where there is often more flexibility in how a treatment is performed, the gap between the platforms was smaller. Interestingly, the length of the answer did not guarantee quality. One of the best systems, Claude, gave concise, direct answers that were highly rated, while another platform gave very long responses that were often filled with errors. The researchers found that the system relying on medical journals was able to ground its answers in established science, avoiding the "hallucinations" or made-up facts that can plague other systems. While the general chatbots were often good at sounding confident, they were more likely to suggest incorrect laser settings or miss safety warnings.

This experiment suggests that while artificial intelligence can be a powerful tool for dentists, it cannot yet be trusted to work alone. The study demonstrated that a system built on verified medical knowledge outperformed those scanning the open web, but even the best systems made occasional mistakes. The researchers concluded that these tools should serve as assistants to help specialists make decisions, not as replacements for them. Before a dentist uses a laser based on a computer's suggestion, a human expert must verify the settings against current medical evidence. The study serves as a reminder that in the high-stakes world of dentistry, where a wrong setting can cause real physical harm, the source of the information matters more than the speed at which it is delivered.

Technical Summary: Large Language Models as Decision-Support Tools for Laser Dentistry

Problem Statement
Laser dentistry is an established adjunct across periodontology, endodontics, and restorative dentistry, yet its application is hindered by a heterogeneous evidence base and a lack of standardized protocols. Clinical decision-making requires precise selection of laser wavelengths, operating parameters (power, fluence, pulse rate), and safety measures to avoid irreversible tissue damage, such as intrapulpal temperature rises. While Large Language Models (LLMs) have emerged as potential decision-support tools in medicine, there is a critical gap in their evaluation for laser dentistry. A primary safety concern is "hallucination"—the confident generation of fabricated clinical data. Furthermore, it remains unclear whether Retrieval-Augmented Generation (RAG) platforms, which ground outputs in external knowledge bases, offer superior safety and accuracy compared to general-purpose LLMs in this parameter-precise domain.

Methodology
This prospective, in-silico, blinded benchmarking study evaluated five conversational AI platforms across three clinical domains: Periodontal/Surgical, Endodontic, and Restorative/Prosthetic.

Platforms Evaluated: Three general-purpose LLMs (ChatGPT 5.5, Claude Opus 4.8, Gemini 3.5 Thinking) and two RAG platforms (OpenEvidence, a medical-domain RAG; and Perplexity, a general-purpose RAG).
Dataset: 24 synthetic clinical vignettes (8 per domain) were constructed to align with consensus statements and systematic reviews. Each vignette included demographics, history, and clinical findings.
Prompting: A standardized six-part master prompt was used, requiring the AI to assess indications, recommend specific laser types/wavelengths, define evidence-based operating parameters, outline clinical protocols, specify safety precautions, and predict outcomes.
Evaluation: Three board-certified specialists (a periodontist, an endodontist, and a restorative specialist) independently scored the responses within their respective domains. Scoring was blinded, with responses randomized under codes.
Metrics: Responses were rated on a 5-point Likert scale for four parameters: Accuracy (correctness of laser choice and parameters), Safety (thermal, pulpal, ocular, and infection control), Freedom from Hallucinations (absence of fabricated data), and Completeness (usability as a clinical protocol). A composite score was calculated as the mean of these four.
Statistical Analysis: Matched within-vignette scores were analyzed using Friedman tests, Kendall's coefficient of concordance (W) for effect size, and Holm-corrected Conover post-hoc tests.

Key Contributions

First Comparative Framework: This study provides the first evaluation of LLMs and RAG platforms specifically for laser dentistry decision support, spanning the three principal clinical domains.
Differentiation of RAG Architectures: It distinguishes between medical-domain RAG (curated peer-reviewed literature) and general-purpose RAG (open-web search), testing the hypothesis that the source of retrieval matters more than the retrieval mechanism itself.
Safety-Centric Benchmarking: The study introduces a rigorous safety rubric that penalizes protocols risking irreversible harm (e.g., lack of cooling or ocular protection) and identifies "dangerous" responses (score ≤2) versus "excellent" ones (score 5).
Blinded Specialist Rating: By utilizing domain-matched specialists who scored only within their expertise, the study minimizes rater bias while maintaining high clinical relevance.

Results

Overall Performance: Significant differences were found among platforms for the composite score (Friedman $\chi^2$ = 58.5, $p < 0.001$ , $W = 0.61$ ), as well as for Accuracy, Safety, and Completeness. No significant difference was found for Freedom from Hallucinations ( $p = 0.053$ ), though all platforms scored relatively high on this metric.
Ranking:
- Top Performers: OpenEvidence (mean 4.67) and Claude Opus 4.8 (mean 4.44) ranked highest and were statistically indistinguishable from each other. Both significantly outperformed the other three platforms.
- Middle Tier: ChatGPT 5.5 (3.54) and Gemini 3.5 Thinking (3.45) showed intermediate performance.
- Lowest Performer: Perplexity (3.03) ranked lowest.
Safety and Risk: The proportion of responses containing at least one "dangerous" parameter (score ≤2) varied drastically: 0% for OpenEvidence, 8.3% for Claude, 20.8% for ChatGPT, 37.5% for Gemini, and 54.2% for Perplexity.
Domain Variability: Platform discrimination was highest in Endodontic ( $W = 0.87$ ) and Restorative/Prosthetic ( $W = 0.86$ ) domains, where precise parameters are critical, and lowest in Periodontal/Surgical ( $W = 0.36$ ), where multiple defensible approaches may exist.
Response Length: While OpenEvidence produced the longest responses (mean 2,183 words), length was not the sole driver of quality; Claude achieved the second-highest score with concise answers (458 words), while Perplexity ranked lowest despite moderate length.

Significance and Claims
The paper claims that for laser dentistry decision support, a medical-domain RAG platform and a leading general-purpose LLM provide the most accurate, safe, and complete guidance, whereas a general-purpose RAG platform performs poorly. The study rejects the null hypothesis that platforms perform equivalently, except regarding the rate of hallucinations.

The authors emphasize that the source of retrieved evidence is the critical differentiator. OpenEvidence's success suggests that grounding outputs in curated, peer-reviewed biomedical literature reduces fabrication and improves alignment with evidence-based parameters. In contrast, general-purpose RAG (Perplexity) failed to provide safe guidance in over half of the cases, likely due to the heterogeneous quality of open-web sources.

The study concludes that while these tools show promise, the wide spread in potentially dangerous responses argues against unsupervised use. The comparative ranking is the robust finding of this matched design, indicating that these tools should assist rather than replace specialist judgment, with every recommendation requiring verification against current evidence before clinical application. The authors note limitations, including the in-silico nature of the study, the use of a single rater per domain, and the potential for slight optimization bias since one of the evaluated models (Claude) assisted in drafting the vignettes.

Technical Summary: Large Language Models as Decision-Support Tools for Laser Dentistry

More like this