Evaluating the Usefulness, Quality, Reliability, and Readability of Large Language Model Responses About Pediatric Laser Dentistry: A Parent-Oriented Study

Technical Summary: Evaluating LLM Responses on Pediatric Laser Dentistry

Problem Statement
Parents increasingly rely on Large Language Model (LLM)-based chatbots for health information, yet the reliability of these systems in the specific domain of pediatric laser dentistry remains unverified. While LLMs offer accessible information, their outputs in healthcare contexts require validation regarding factual accuracy, clinical appropriateness, and readability. Specifically, information regarding dental lasers must balance potential benefits with limitations, avoiding the presentation of lasers as universally superior or painless. Previous studies have highlighted variability in LLM performance based on model, topic, and evaluation method, but evidence regarding chatbot-generated information specifically for parent-oriented questions on pediatric laser dentistry is limited. This study addresses the need to evaluate whether current advanced LLMs can serve as reliable adjuncts for parental education in this niche clinical area.

Methodology
The study employed a quantitative, comparative design to evaluate four paid, advanced LLM platforms: ChatGPT-5.5 Thinking, Google Gemini 3.1 Pro, Claude Opus 4.7, and DeepSeek-V4.

• Data Collection: A set of 20 content-validated, open-ended, parent-oriented questions was developed (S-CVI/Ave = 0.86) covering indications, limitations, safety, and comparisons with conventional procedures. These questions were submitted once daily for seven consecutive days (April 30–May 6, 2026) to each model, generating a total of 560 responses. To ensure independence, questions were entered into new, incognito sessions without using memory features or follow-up prompts.
• Evaluation Metrics: Two blinded pediatric dentistry specialists independently assessed the responses using three primary domains:
  1. Usefulness: A modified 4-point scale (1 = very useful to 4 = not useful).
  2. Global Quality Scale (GQS): A 5-point scale assessing overall flow and comprehensiveness.
  3. CLEAR Score: A 25-point tool (Completeness, Lack of false information, Evidence-supported, Appropriateness, Relevance) assessing content quality and reliability.
• Readability and Length: Responses were analyzed for word/character count and readability using Flesch Reading Ease (FRE), Flesch–Kincaid Grade Level (FKGL), and SMOG Index.
• Statistical Analysis: Data were analyzed using Friedman tests for omnibus comparisons among the four models, with Bonferroni-corrected Wilcoxon signed-rank tests for pairwise comparisons. Kendall's W was calculated as the effect size. Inter-rater reliability was assessed via intraclass correlation coefficients (ICC).

Key Results

• Expert Ratings: Significant differences were observed across all models for usefulness, GQS, and CLEAR scores (p < 0.001).
  • ChatGPT-5.5 Thinking achieved the most favorable results, with a median usefulness score of 1.0 ("very useful"), a GQS of 5.0, and a CLEAR score of 22.0 ("very good content"). It significantly outperformed all other models.
  • DeepSeek-V4 produced the lowest expert-rated quality (median CLEAR score of 15.0, "average content") and usefulness (median 2.0), despite generating the longest responses.
  • Claude Opus 4.7 and Google Gemini 3.1 Pro demonstrated intermediate performance, with no statistically significant difference between them in usefulness or GQS after correction, though they differed in CLEAR scores.
• Readability and Length:
  • DeepSeek generated the longest responses (median ~608 words) and achieved the highest readability scores (highest FRE, lowest FKGL and SMOG), indicating the easiest reading level.
  • ChatGPT generated the shortest responses.
  • Crucially, all models required reading levels (FKGL and SMOG) significantly higher than the levels typically recommended for patient education materials.
• Temporal Variability: While day-to-day differences in expert-rated outcomes were statistically significant, the effect sizes (Kendall's W ≈ 0.06) were negligible, indicating practically stable short-term performance across the seven-day collection period.
• Reliability: Inter-rater reliability between the two evaluators was excellent (ICC > 0.97 for all metrics).

Key Contributions

• This is the first study to systematically evaluate and compare four advanced, paid LLMs specifically on parent-oriented questions regarding pediatric laser dentistry.
• The study utilizes a matched-block design over seven days to assess short-term temporal variability, a factor often overlooked in single-instance evaluations.
• It introduces a critical dissociation between linguistic accessibility and content quality: the model producing the most readable and verbose responses (DeepSeek) received the lowest expert ratings for factual accuracy and clinical appropriateness.
• The research provides a validated, blinded assessment framework (Usefulness, GQS, CLEAR) applied to a specific, high-stakes pediatric dental subdomain.

Significance and Claims
The paper claims that while current LLMs may support parental education, they should not replace professional pediatric dental consultation. The study rejects the null hypothesis, demonstrating that significant performance disparities exist between models. The authors emphasize that ChatGPT-5.5 Thinking currently offers the most reliable and high-quality information for this specific topic, whereas DeepSeek's verbosity does not equate to clinical accuracy.

The authors conclude that despite the potential of LLMs as educational tools, the variability in content quality and the consistently high reading levels required to understand the responses suggest caution. Parents using these tools must be aware that the information may be incomplete, overly complex, or potentially misleading, particularly regarding the limitations and safety considerations of laser use in children. The study underscores the necessity for topic-specific evaluations before LLMs can be considered reliable adjuncts in pediatric dental care.