Technical Summary: GlobalDentBench

Problem Statement

While Large Language Models (LLMs) have demonstrated proficiency in standardized medical licensing examinations, their reasoning robustness and safety in real-world clinical scenarios remain critically underexplored, particularly in dentistry. Existing dental evaluations are largely restricted to multiple-choice questions (MCQs) that measure factual recall rather than the inferential reasoning required for routine clinical practice. Furthermore, current benchmarks are fragmented, often limited to local licensing contexts or narrow subfields, and lack a globally representative framework capable of assessing models under authentic clinical conditions involving diagnostic ambiguity, incomplete information, and iterative reasoning. This gap poses significant risks, as errors in dental diagnosis can lead to irreversible patient harm, yet there is no standardized, multi-level framework to evaluate whether current LLMs are ready for clinical deployment.

Methodology

The authors introduce GlobalDentBench, a multinational benchmark designed to evaluate LLM clinical reasoning in dentistry through a rigorous, expert-calibrated pipeline.

1. Dataset Construction

GlobalDentBench comprises 8,978 expert-validated questions spanning 88 countries and regions and 14 dental specialties. The dataset is constructed from three distinct source types to ensure alignment with authentic practice:

• Multiple-Choice Questions (MCQs): 3,679 items sourced from official national dental qualification examinations (e.g., US, UK, Canada, Australia).
• Short-Answer Questions (SAQs): 3,709 items derived from officially recognized, peer-reviewed dental textbooks.
• Case-Based Questions (CBQs): 1,590 items extracted from peer-reviewed clinical case reports (2020–2025) to evaluate scenario-driven reasoning.

2. Taxonomy and Reasoning Levels

The benchmark categorizes items using a three-tiered reasoning framework:

• L1 (Knowledge Recall): Assessing foundational knowledge without complex case reasoning.
• L2 (Routine Reasoning): Assessing reasoning based on core clinical presentations and typical diagnostic logic.
• L3 (Individualized Reasoning): Assessing reasoning that requires integrating patient-specific constraints, non-standard considerations, and individualized factors.

3. Automated Pipeline with Expert Calibration

To handle the heterogeneity of source materials, the authors developed an automated agent pipeline consisting of three stages:

• Document Normalization: Converting diverse formats (PDF, images, Word) into a unified markdown representation.
• Type-Aware Construction: Extracting and structuring questions, options, and reference answers/key points specific to MCQ, SAQ, and CBQ formats, utilizing a self-correction loop to verify fidelity to source texts.
• Unified Tagging: Automatically assigning dental discipline and reasoning level labels.

Crucially, this automated process is validated by a human-in-the-loop framework involving six senior dentists. Experts audited 32.89% of the CBQ corpus and verified the pipeline's output, achieving agreement rates of 99.98% for MCQs/SAQs and 96.78% for CBQs.

4. Evaluation Protocol

The study evaluated 12 frontier LLMs (6 proprietary, 6 open-source).

• MCQs: Scored by exact-match accuracy.
• SAQs and CBQs: Evaluated using an automated judge model (Gemini-3-Flash-Preview), selected for its high agreement (98.15%) with expert grading.
• Safety Analysis: CBQ responses were classified into three risk categories: S0 (Safe), S1 (Unsafe, reversible harm), and S2 (Unsafe, irreversible harm).

Key Results

1. Performance Degradation with Complexity

A sharp, stepwise decline in performance was observed as reasoning complexity increased:

• By Question Type: Accuracy dropped from 81.34% on MCQs to 64.53% on SAQs and plummeted to 22.34% on CBQs.
• By Reasoning Level: Performance declined from 74.01% at L1 to 55.64% at L2 and 35.71% at L3. No model exceeded 50% on individualized reasoning (L3).

2. Model Comparisons

• Top Performers: Gemini-3.1-Pro-Preview achieved the highest overall macro-average score (63.27%). Among open-source models, GLM-5 led with 56.43%.
• Cost-Performance: Higher inference costs did not consistently correlate with better performance. Gemini-3-Flash-Preview offered the most favorable trade-off (61.59% score at $3.72/1k queries), while Grok-4 incurred the highest cost ($38.14/1k) with a lower score (57.77%).

3. Disciplinary Heterogeneity

Performance varied significantly across the 14 dental disciplines:

• Highest: Oral Mucosal Diseases (OMD, 63.66%), Pulp and Periapical Diseases (PPD, 59.62%).
• Lowest: Orthodontics (Ortho, 50.59%), Pediatric Dentistry (PD, 51.08%), and Conventional Prosthodontics (CP, 51.92%).
• The authors note that domains with standardized terminology (e.g., OMD) yielded higher scores, while those requiring spatial analysis and longitudinal planning (e.g., Orthodontics) showed weaker performance.

4. Safety and Risk Analysis

Risk analysis of 1,590 case-based questions revealed alarming safety concerns:

• Overall Unsafe Rate: 31.01% of LLM-generated clinical recommendations were classified as unsafe (S1 or S2).
• Severity: 4.51% of responses posed risks of irreversible patient harm (S2).
• Specialty-Specific Risks: Orthodontics had the highest overall unsafe rate (44.30%), while Systemic Health, Pharmacology, and Safety (SHPS) showed the highest concentration of severe (S2) errors (14.15% average).
• Model Variation: Unsafe rates ranged from 15.97% to 45.85% across models, with GPT-5.4, Gemini-3.1-Pro-Preview, and Gemini-3-Flash-Preview showing the lowest risk profiles.

Significance and Claims

The paper claims that GlobalDentBench establishes a scalable, trustworthy foundation for evaluating medical AI, filling a critical void in multi-regional, full-spectrum assessment tools for dentistry.

• Limitations of Current LLMs: The findings expose fundamental limitations in current models, indicating they function primarily as supervised aids for knowledge retrieval rather than autonomous clinical agents. Their inability to handle individualized reasoning (L3) and the high rate of unsafe recommendations in complex cases preclude their autonomous deployment in high-stakes healthcare settings.
• Methodological Contribution: The benchmark provides a template for constructing evaluations in other medical specialties, demonstrating that automated construction combined with expert calibration can yield high-fidelity datasets.
• Clinical Implication: The authors assert that the integration of LLMs into clinical workflows must be strictly confined to human-supervised roles. The urgent need for rigorous validation and "human-in-the-loop" oversight is underscored by the data, particularly given the risks of irreversible harm in specific specialties.

In conclusion, GlobalDentBench shifts the evaluation paradigm from simple factual recall to clinically grounded, multi-step reasoning, revealing that while frontier models are proficient in standardized testing, they remain unreliable for the nuanced, individualized decision-making required in real-world dental practice.