Trends in AI and Human-AI Interaction in Clinical Trials -- A Hybrid Human-AI Exploration

Technical Summary: Trends in AI and Human-AI Interaction in Clinical Trials – A Hybrid Human-AI Exploration

Problem Statement
The proliferation of Artificial Intelligence (AI) in clinical research has led to a significant volume of registered trials on platforms like ClinicalTrials.gov. However, robustly identifying and comparing these trials is hindered by inconsistent reporting practices. Registry records often vary substantially in how they describe AI methods, define the role of human users, specify input/output data, and characterize the clinical context. While reporting guidelines such as SPIRIT-AI and CONSORT-AI exist to improve transparency, evidence suggests compliance remains incomplete, with important features of AI interventions frequently underreported. Furthermore, frameworks for understanding human-AI interaction in real-world clinical settings remain underdeveloped, and the definition of "AI" has evolved from symbolic systems to include machine learning, deep learning, and large language models (LLMs). This study addresses the challenge of characterizing temporal trends in AI terminology and geographical distribution, while exploring the feasibility of a hybrid human-AI workflow to screen and categorize trial records regarding human-AI interaction.

Methodology
The study utilized a hybrid screening approach combining frontier generative AI with human review.

• Data Source: The authors queried the ClinicalTrials.gov repository using a comprehensive search string comprising 16 AI-related terms (e.g., "artificial intelligence," "machine learning," "GPT," "chatbot," "neural network"). The search was restricted to the "Intervention" fields but retained observational studies.
• Search Scope: The search covered trials with first submission dates on or before April 1, 2026, retrieving 5,828 records (approximately 1% of the total registry).
• Hybrid Workflow:
  • AI Classifier: A frontier generative AI model (GPT-5.5) was employed to screen and categorize the records. The model was prompted to classify trials into predefined categories regarding AI use and human-AI interaction, providing a confidence level (high, medium, low) and a short explanation. The process consumed over 10 million input tokens and 3.2 million output tokens.
  • Human Classifier: A representative random sample of 100 trials was independently classified by two human experts with master's-level or higher expertise in clinical datasets. A third human arbitrator resolved discrepancies. Humans were permitted to abstain if confidence was low.
• Classification Schema: Trials were categorized based on:
  • AI Usage: "No use of AI" vs. "Use of AI."
  • Interaction Type: "No human-AI interaction," "Patient-AI," "Caregiver-AI," "Health professional-AI," "Other human-AI," and "Hybrid-AI" (multi-actor).

Key Results

• Temporal and Terminological Trends: The analysis revealed a marked increase in AI-related trials over the last decade. While early mentions focused on "expert systems," there has been a sustained rise in "artificial intelligence" and "AI" since 2016. Terms such as "machine learning," "deep learning," "large language model" (LLM), and "chatbot" have seen substantial growth since 2017–2018, with "ChatGPT" and "GPT" rising sharply post-2022. Conversely, terms like "neural network" and "natural language processing" have shown a leveling off.
• Geographical Distribution: China and the United States account for the largest volumes of AI-related trials, with China showing a sharp increase since 2018, overtaking the US in recent years. Both nations possess approximately four times the number of AI trials compared to the next highest country (Italy). Notable recent increases were also observed in Italy, France, Spain, the UK, and Turkey.
• Hybrid Screening Performance:
  • AI Usage Identification: Human and AI classifiers showed strong agreement in identifying trials that did not substantively use AI (13 out of 14 agreements in the sample). The AI model reported "high" confidence for most of these, with only two instances of reduced confidence corresponding to human uncertainty.
  • Interaction Classification: Agreement was lower when classifying human-AI interaction. Of the 100 trials, there were 30 total classification disagreements. While 23 of these concerned whether interaction existed at all, the majority of disagreements occurred between "Health professional-AI interaction" and "No human-AI interaction."
  • Sources of Disagreement: Ambiguity in trial reporting was the primary cause of classification difficulty. Specific challenges included conflicting terminology (e.g., "Deep learning" listed but only "automated software" described) and AI components embedded within broader interventions (e.g., sound control in VR) where the substantive use of AI was unclear.

Significance and Claims
The paper claims that a hybrid human-AI approach is a potentially viable alternative to time-intensive manual screening for systematic reviews of clinical trials. The results suggest that frontier large language models (specifically GPT-5.5) can effectively identify the presence or absence of AI in trial records with high confidence. However, the authors modestly conclude that classifying specific modes of human-AI interaction remains challenging due to variable conformance to reporting standards and the lack of explicit requirements to categorize interaction types in current guidelines (CONSORT-AI/SPIRIT-AI).

The study highlights that while AI can assist in the initial identification and broad categorization of trials, clearer trial reporting and more precise definitions of "interaction"—particularly regarding health professional involvement—are necessary to improve the reliability of automated or hybrid screening. The authors also note the environmental and financial costs of such computational approaches, acknowledging the carbon footprint associated with the token usage required for the analysis. Future work is proposed to refine interaction definitions and explore programmatic NLP approaches to address classification disagreements.