Large Language Models in Healthcare Simulation Education: A Bibliometric Analysis with AI-Assisted Screening

Technical Summary: Large Language Models in Healthcare Simulation Education

Problem Statement
The rapid emergence of Large Language Models (LLMs) following the public release of ChatGPT in late 2022 has triggered an unprecedented expansion in healthcare education and simulation-based training, particularly regarding non-technical skills (NTS). However, the volume of literature has grown at a pace that renders traditional narrative reviews and systematic synthesis impractical. Consequently, there is no comprehensive, quantitative map of this research landscape. Critical questions remain unanswered regarding the distribution of research across NTS domains (e.g., communication vs. teamwork), the dominance of specific proprietary models versus open-source alternatives, the prevalence of different simulation modalities, and the extent of specialty-specific engagement, particularly in fields like urology with established simulation infrastructures. Furthermore, the sheer scale of the initial literature (over 100,000 records) presents a methodological bottleneck for human-led screening, necessitating scalable, reproducible approaches to evidence synthesis.

Methodology
This study employed a bibliometric analysis design adhering to PRISMA 2020 guidelines and the PRISMA-trAIce checklist for transparent AI reporting. The methodology was defined by two core principles: analytical rigour and full reproducibility using exclusively open-access data sources.

• Data Sources and Search Strategy: Seven open-access databases (OpenAlex, PubMed, Europe PMC, Crossref, Semantic Scholar, CORE, DOAJ) were queried for English-language publications between January 2020 and March 2026. The search yielded 100,277 initial records.
• Screening Pipeline: A hybrid screening approach was utilized to manage the volume:
  1. Deterministic Keyword Funnel: A sequential three-stage filter reduced the corpus from 86,635 unique records to 5,725, and subsequently to 830 candidates. Filters required the presence of LLM/AI terminology, healthcare/education context, and simulation-specific terms. A second-pass filter enforced stricter criteria, including specific named models and medical keywords in titles.
  2. AI-Assisted Verification: The 830 candidate papers were screened by 83 independent instances of the Claude Sonnet 4.6 AI agent. Each agent applied pre-specified inclusion/exclusion criteria to title and abstract data, returning binary decisions, confidence ratings, and tier classifications.
  3. Validation: A lead researcher independently reviewed 180 papers (22% of the candidate pool). Inter-rater reliability between the AI agents and the human reviewer yielded a Cohen's kappa of 0.86 pre-reconciliation and 1.0 post-reconciliation, exceeding the mean human-human kappa of 0.82 reported in existing literature.
• Analysis: The final corpus of 551 papers underwent bibliometric analysis including publication trend analysis (CAGR), citation metrics (h-index, g-index), author productivity (Lotka's law), journal concentration (Bradford's law), and geographic distribution. Thematic mapping was conducted via keyword co-occurrence networks and community detection (Louvain algorithm). NTS domains were classified using regular-expression pattern matching across nine specific competencies.

Key Contributions

• First Comprehensive Map: This study provides the first bibliometric analysis specifically mapping the intersection of LLMs, healthcare simulation, and non-technical skills training.
• Validated AI Screening Pipeline: The authors demonstrate a replicable, open-access pipeline for large-scale evidence synthesis using multiple independent AI agents. The methodology achieved high reliability (Cohen's kappa > 0.86) and offers a scalable alternative to traditional multi-reviewer screening for fast-evolving fields.
• Granular Stratification: The corpus was stratified into three tiers: Tier 1 (LLMs in healthcare education broadly), Tier 2 (LLMs in healthcare simulation), and Tier 3 (LLMs in healthcare simulation with NTS), allowing for nuanced analysis of research depth.
• Open Science: The entire pipeline, including code, data, and AI prompts, is publicly available on GitHub, enabling replication and longitudinal updates without institutional database subscriptions.

Results

• Growth Trajectory: The field exhibits explosive growth with a Compound Annual Growth Rate (CAGR) of 109%. Publication volume is almost entirely post-ChatGPT (99% published from 2023 onward), with 2025 being the peak year (52% of total output).
• Model Landscape: The research ecosystem is heavily concentrated on a single proprietary model. ChatGPT appears in 46.5% of papers. Other proprietary models (GPT-4, Gemini, Claude) account for an additional ~15%. Open-source models are virtually absent, appearing in only 0.7% of papers (4 papers mentioning Llama).
• Simulation Modalities: Virtual patient chatbots are the dominant modality (19% of the corpus), reflecting the natural alignment between conversational AI and patient interaction. Scenario-based simulation (13%) and VR/XR (9.4%) follow. Mannequin-based and high-fidelity simulation are markedly underrepresented (1.1%).
• Non-Technical Skills (NTS) Distribution: Research is heavily skewed toward individual competencies. Communication (26.3%) and Decision-making (24.5%) dominate. Team-based and crisis-oriented skills are critically underrepresented: Teamwork (5.8%), Leadership (3.4%), Situational Awareness (0.9%), and Crisis Resource Management (CRM) are barely studied.
• Geographic and Authorship: Contributions come from 58 countries, with the US (20.2%) and China (18.8%) leading. Africa (1.1%) and South America (3.9%) are significantly underrepresented. Author productivity follows a steep Lotka's law distribution (exponent 3.7), with 88.7% of authors contributing only a single paper.
• Urology Sub-Analysis: Only 6 papers (1.1%) addressed the intersection of LLMs, simulation, and urology. None examined LLM integration within established boot camp training formats, despite the specialty's robust simulation infrastructure.

Significance and Claims
The paper claims that the field of LLMs in healthcare simulation is in an "inflection" phase, characterized by rapid volume growth but significant structural imbalances.

• Research Gaps: The authors identify critical blind spots where research lags behind clinical necessity. Specifically, the lack of focus on team-based skills (teamwork, leadership, CRM) suggests that current LLM applications may be reinforcing individual competencies while neglecting the collaborative dynamics essential for patient safety.
• Model Dependency: The overwhelming reliance on a single proprietary model (ChatGPT) raises concerns regarding reproducibility, accessibility for low-resource institutions, and the generalizability of findings across different AI architectures.
• Specialty Neglect: The near-absence of urology-specific research in simulation contexts highlights a missed opportunity to leverage established training frameworks (e.g., boot camps) for AI augmentation.
• Methodological Feasibility: The study asserts that AI-assisted screening using independent agents is a feasible, reliable, and scalable method for mapping fast-evolving research fields, offering a pathway to "living bibliometric analyses" that can update in near-real-time.

The authors conclude that while the breadth of research is remarkable, the field lacks the thematic balance, specialty-specific depth, and rigorous evaluation (e.g., randomized controlled trials) required to confidently guide the integration of LLMs into simulation-based NTS training. They call for future research to address the identified gaps in team-based skills, open-source model evaluation, and specialty-specific applications.