📄 medicine

Can Large Language Models Assist Surgeons in Managing Implant-Related Complications? A Guideline-Informed Comparative Evaluation

This study evaluates four large language models on guideline-informed implant-complication scenarios, finding that while they differ in performance and safety—particularly in complex cases with Gemini 3.1 outperforming DeepSeek V3.2—they cannot yet replace expert clinical judgment.

Original authors: Emrah Bilen¹, Zeynep Soylu², Ali Bulut, Muhammed Kahkeci

Published 2026-08-10

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Emrah Bilen¹, Zeynep Soylu², Ali Bulut, Muhammed Kahkeci

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a chef trying to cook a perfect meal, but instead of a recipe book, you have a super-smart robot that has read every cookbook, food blog, and restaurant review in the world. This robot, known as a Large Language Model (LLM), can chat with you, answer questions, and even suggest complex dishes. In the world of medicine, these AI robots are becoming popular helpers for doctors, offering quick advice on everything from common colds to tricky surgeries. But here's the catch: just because a robot can talk confidently doesn't mean it knows the truth. Sometimes, these AI systems make things up, a phenomenon scientists call "hallucinating," or they give advice that sounds good but could actually hurt a patient. This is especially scary in dentistry, where placing a dental implant is like building a tiny, permanent foundation in a person's jawbone. If the foundation is wrong, or if a complication like a nerve injury or severe bleeding happens, the results can be painful and permanent. So, the big question isn't just "Can the AI talk?" but "Can the AI be trusted to save the day when things go wrong?"

This study decided to put four of the most famous AI robots to the test: Gemini 3.1, ChatGPT 5.4, Claude 4.6, and DeepSeek V3.2. The researchers didn't just ask them simple questions like "What is a dental implant?" Instead, they created 24 tricky, made-up stories (called clinical vignettes) about dental implants going wrong. These stories ranged from "intermediate" trouble (like a small infection) to "expert-level" disasters (like a nerve being crushed or a sinus membrane tearing). The AI robots had to act like experienced surgeons, diagnosing the problem and suggesting a treatment plan based on real-world medical guidelines. Then, two real-life expert dentists, who didn't know which AI wrote which answer, graded the robots on a scale of 1 to 5. They looked at how accurate the diagnosis was, how safe the advice was, and whether the plan made sense. They also played "spot the danger," checking if the AI gave any advice that could seriously harm a patient or made up fake medical facts.

The results were a bit like a race where the runners had very different stamina levels. The study found that the robots were not all created equal. Gemini 3.1 was the clear winner, consistently scoring the highest marks for being accurate, safe, and clear. It was like the student who actually studied the textbook and knew exactly what to do. On the other end of the spectrum, DeepSeek V3.2 struggled the most. It gave lower scores overall and, more worryingly, was the most likely to "hallucinate" or give dangerous advice. In fact, DeepSeek made up fake medical facts or guidelines in nearly one out of every three answers (29.2%), and it suggested harmful treatments in about 17% of the cases. The other two robots, ChatGPT 5.4 and Claude 4.6, landed somewhere in the middle, doing okay but not as consistently as the winner.

The researchers also discovered that the difficulty of the problem mattered a lot. When the scenarios were simple or "intermediate," all the robots performed roughly the same, like a group of friends all knowing how to tie their shoes. But when the problems got "difficult" and required complex thinking, the gap between the robots widened significantly. Gemini 3.1 stayed strong and smart, while DeepSeek V3.2 started to stumble. Interestingly, even in the hardest "expert-level" scenarios, the differences weren't statistically huge, likely because the problems were just too tough for any of them, or because there weren't enough of those specific hard cases to prove a difference.

The most important takeaway from this paper is a big "No" to the idea that AI can replace a human surgeon. The study suggests that while these tools might be helpful for checking facts or learning, they are not ready to make life-or-death decisions on their own. The "safety burden" was too high, especially for the weaker models. The authors conclude that in the high-stakes world of dental implants, an AI can be a useful assistant, like a co-pilot, but it can never be the pilot. The human expert needs to be in the seat, double-checking everything, because when it comes to patient safety, a confident robot that makes things up is a risk no one can afford.

Technical Summary: Guideline-Informed Comparative Evaluation of LLMs in Implant-Related Complications

Problem Statement

While surgical techniques and digital workflows in implantology have advanced, managing implant-related complications (e.g., nerve injuries, membrane perforations, severe bleeding) remains clinically demanding. Clinicians, particularly general practitioners lacking immediate specialist support, increasingly utilize Large Language Models (LLMs) for point-of-care guidance. However, the reliability of LLMs in high-stakes clinical decision-making is uncertain. Existing literature has primarily evaluated LLMs in patient education or theoretical knowledge synthesis, often failing to address:

Safety-Critical Failures: The distinction between general response quality and specific safety events (e.g., hallucinations, harmful recommendations).
Complexity Stratification: How model performance varies across intermediate, difficult, and expert-level clinical scenarios.
Guideline Concordance: The ability of models to adhere to consensus-based management frameworks from major organizations (EAO, ITI, AAOMS) in complex, multi-variable situations.

The study addresses the gap in evidence regarding whether LLMs can serve as reliable supportive tools or if they pose significant patient safety risks in expert-level implant complication management.

Methodology

This study employed a comparative, cross-sectional in-silico evaluation design.

Clinical Scenarios: Twenty-four clinical vignettes were developed, stratified by difficulty (6 intermediate, 12 difficult, 6 expert) and categorized into six complication types (sinus augmentation, intraoperative, early/late postoperative, hard-tissue augmentation, peri-implant soft-tissue). These scenarios were constructed to align with consensus documents from the European Association for Osseointegration (EAO), International Team for Implantology (ITI), and American Association of Oral and Maxillofacial Surgeons (AAOMS).
Models Evaluated: Four state-of-the-art LLMs were tested:
- Gemini 3.1 Pro
- ChatGPT 5.4
- Claude 4.6
- DeepSeek V3.2
- Protocol: Models were accessed via web interfaces with default settings, advanced reasoning modes enabled, and no external plugins. Each scenario was submitted once per model using a standardized pre-prompt.
Evaluation Framework: Responses were anonymized and randomized, then independently evaluated by two blinded experts (an oral and maxillofacial surgeon and a periodontologist).
- Likert Domains (1–5 scale): Diagnostic accuracy, management appropriateness, safety, completeness/actionability, and clarity/coherence.
- Binary Safety Outcomes: Presence of major harmful recommendations, missed red flags, and hallucinated/fabricated content.
- Reliability: Inter-rater and intra-rater reliability were assessed using Cohen's kappa, with disagreements resolved by a third expert.
Statistical Analysis: Friedman tests with Kendall's W were used for overall model comparisons, followed by post-hoc Wilcoxon signed-rank tests with Holm correction. Cochran's Q tests were used for binary safety outcomes.

Key Results

Overall Performance: Significant differences were observed among models across all five evaluation domains ( $p < 0.05$ $p < 0.05$ ). Gemini 3.1 consistently achieved the highest scores, while DeepSeek V3.2 scored the lowest.
- Post-hoc analysis revealed that Gemini 3.1 significantly outperformed DeepSeek V3.2 in all domains (e.g., overall score: 4.30 vs. 3.58; $p=0.003$ ).
- No other pairwise comparisons remained statistically significant after Holm correction.
Impact of Scenario Difficulty:
- Intermediate Scenarios: No significant differences were found among models.
- Expert Scenarios: No significant differences were found (likely due to sample size limitations or universal difficulty in high-stakes reasoning).
- Difficult Scenarios: Significant differences emerged ( $p=0.014$ ). Gemini 3.1 significantly outperformed DeepSeek V3.2 (4.25 vs. 3.60; $p=0.023$ ), suggesting model variability is most pronounced in complex, multi-variable cases.
Safety Outcomes:
- Major Harmful Recommendations: Significant differences were found ( $p=0.041$ ). DeepSeek V3.2 had the highest burden (4/24 cases, 16.7%), while Gemini 3.1 and ChatGPT 5.4 had zero instances.
- Hallucinations/Fabrications: Significant differences were found ( $p=0.035$ ). DeepSeek V3.2 exhibited the highest rate (7/24 cases, 29.2%), followed by Claude 4.6 (3/24 cases, 12.5%), Gemini 3.1 (2/24 cases, 8.3%), and ChatGPT 5.4 (1/24 cases, 4.2%).
- Missed Red Flags: No significant differences were observed among models.

Key Contributions

Safety-Separated Evaluation: The study introduces a methodological framework that separates overall response quality from specific patient safety events (harmful recommendations and hallucinations), arguing that a high average score does not preclude the generation of clinically critical errors.
Difficulty-Stratified Benchmarking: By categorizing scenarios into intermediate, difficult, and expert levels, the study demonstrates that LLM performance disparities are not uniform; they are most evident in "difficult" scenarios requiring the integration of multiple clinical variables.
Guideline-Informed Standard: The evaluation utilized a reference standard based on EAO, ITI, and AAOMS consensus principles rather than simple knowledge retrieval, assessing the models' ability to apply guidelines to specific, complex clinical contexts.
Empirical Safety Data: The study provides quantitative data on the frequency of "hallucinated" or "fabricated" guideline claims and harmful recommendations across four specific model versions, highlighting DeepSeek V3.2's higher risk profile in this specific domain.

Significance and Claims

The authors conclude that while LLMs show promise as supportive tools for clinical education and decision-checking, they cannot replace expert clinical judgment in the management of implant-related complications.

Context Sensitivity: The performance of LLMs is highly context-dependent. They perform comparably on straightforward (intermediate) cases but diverge significantly in complex (difficult) scenarios, where the risk of error increases.
Safety Profile Variability: The study emphasizes that safety profiles vary drastically between models. The high incidence of harmful recommendations and hallucinations in DeepSeek V3.2 compared to the near-zero incidence in Gemini 3.1 and ChatGPT 5.4 suggests that model selection is critical for patient safety.
Role of AI: The findings support the view of LLMs as "supportive tools" that require expert supervision and validation, rather than autonomous decision-making systems. The authors assert that the integration of these models into clinical workflows must be accompanied by rigorous evaluation and a clear understanding of their limitations in handling safety-critical, multi-variable clinical reasoning.