Implementation of Human-in-the-Loop ChatGPT-based Patient Screening Across Multiple Diverse Clinical Trials

This study demonstrates that a human-in-the-loop, LLM-assisted prescreening workflow achieved high accuracy and low cost ($0.12 per patient) across 26 diverse clinical trials by effectively leveraging coordinator feedback to automate and refine patient eligibility screening.

The Gist

Imagine a massive library containing millions of medical records, and inside that library, there are hundreds of different "clubs" (clinical trials) looking for new members. Each club has a very specific, complicated list of rules for who can join.

The Problem:
In the past, finding the right members for these clubs was like hiring a team of librarians to read every single book in the library, page by page, to see if it fit the rules. It was slow, expensive, and many potential members were missed because the librarians got tired or overwhelmed.

The Solution:
The doctors in this study built a super-smart digital assistant (an AI powered by Large Language Models, or LLMs) to help the librarians. Think of this AI as a high-speed, tireless scout that can read thousands of medical records in seconds.

How It Works: The "Smart Scout" System

1. The Scout Reads the Rules: The AI is taught the specific rules for each clinical trial (e.g., "Must have had a specific type of surgery," "Cannot have taken a certain drug in the last year").
2. The Scout Scans the Library: Instead of humans reading every note, the AI scans the electronic health records of thousands of patients. It looks for clues in doctor's notes, lab results, and imaging reports.
3. The "Maybe" List: The AI doesn't just say "Yes" or "No." It acts like a traffic cop, sorting patients into lanes:
   • Green Lane: "Almost certainly fits the rules."
   • Yellow Lane: "Looks like they might fit, but I'm not 100% sure."
   • Red Lane: "Definitely doesn't fit."
4. The Human Check (The "Human-in-the-Loop"): This is the most important part. The AI doesn't make the final decision. It hands the "Green" and "Yellow" lists to real human research coordinators. The humans act as the final judges. They quickly review the AI's suggestions to confirm if the patient is truly eligible.

Why This is a Game-Changer

• Speed and Scale: In just one year, this system screened 39,000 patients across 26 different trials. A human team would have taken years to do this manually.
• High Accuracy: The AI was incredibly good at its job. It correctly identified whether a patient met a specific rule about 94% of the time. Even better, it rarely missed a patient who should have been included (98% sensitivity), ensuring no one was accidentally left out.
• Learning from Mistakes: The system isn't static. If a human coordinator says, "Hey, the AI got this wrong because it didn't understand this specific medical term," the AI learns from that feedback. It updates its own rulebook to do better next time. It's like a student who studies their test errors to get an A on the next one.
• Cheap: The cost to run this digital scout was about 12 cents per patient. That's cheaper than a cup of coffee!

The Result

By using this "Smart Scout" system, the hospital didn't just save time and money; they found more eligible patients for life-saving research. The AI handled the heavy lifting of reading the fine print, allowing the human experts to focus on what they do best: making the final, compassionate decisions about patient care.

In short: They built a tireless, super-fast robot assistant that reads medical charts, flags the best candidates for clinical trials, and learns from human feedback, all while costing almost nothing. It's a perfect team-up between human wisdom and machine speed.

Technical Summary

1. Problem Statement

Clinical trial enrollment is a major bottleneck in medical research. Manual patient eligibility screening is resource-intensive, time-consuming, and prone to human error, often leading to insufficient accrual, delayed research, and inflated costs. While Natural Language Processing (NLP) has been used for screening, earlier approaches required extensive task-specific customization and lacked generalizability across diverse trial designs. Large Language Models (LLMs) offer broader applicability but have primarily been tested in controlled, single-trial settings or on synthetic data. There is a lack of evidence regarding the feasibility, performance, and cost-effectiveness of deploying LLMs for prospective, multi-trial screening in a real-world academic medical center, particularly when integrated with human-in-the-loop (HITL) learning strategies.

2. Methodology

The authors implemented a Retrieval-Augmented Generation (RAG) pipeline integrated with a human-in-the-loop workflow at UT Southwestern Medical Center.

• Workflow Architecture:

  • Data Ingestion: The system extracts data nightly from the Epic EHR (via Caboodle/Clarity), including structured data (labs, pathology) and unstructured free-text notes (progress notes, imaging reports). Scanned external PDFs were excluded per policy.
  • Tiered Evaluation: To optimize efficiency, criteria were organized into tiers.
    • Tier 1 (Gatekeeper): A single primary criterion is evaluated first. If the AI labels it as "met" or "likely met," the system proceeds to Tier 2. If Tier 1 is "not met," "unlikely," or "uncertain," Tier 2 is skipped.
    • Tier 2: Remaining criteria are evaluated only if Tier 1 passes.
  • RAG Pipeline: For each criterion, the system retrieves the top 30 most relevant text nodes (using a hierarchical node parser: document $\to$ paragraph $\to$ sentence) from a vector database.
  • LLM Inference: The LLM (initially GPT-4o, later GPT-4o-mini and GPT-4.1-mini) receives the criterion text, protocol verbatim, and retrieved context. It outputs a status label (met, likely met, likely not met, not met, uncertain, no documentation found), a 5-point certainty score, and a rationale.
  • Human-in-the-Loop (HITL) & Feedback:
    • Triage: Coordinators review a sorted list of patients. Review thresholds were dynamic (e.g., mandatory review for patients meeting $\ge$50% of criteria; prioritized for $\ge$80%).
    • Feedback Loop: Coordinators provided rationale for discordant cases. This feedback triggered an automated refinement bot that updated the "tips" (prompt instructions) for specific criteria.
    • Continuous Learning: The system updated prompts based on coordinator feedback and literature reviews (via a PubMed agent) without re-scoring historical data, ensuring prospective improvement.

• Scope: The study covered 26 active trials (21 oncology, 5 non-oncology) involving 112 distinct eligibility criteria over a 12-month period (Oct 2024–Sep 2025).

3. Key Contributions

1. Scalable Multi-Trial Deployment: Demonstrated the first prospective deployment of an LLM-based screening workflow across a diverse portfolio of 26 concurrent trials, moving beyond single-trial or synthetic evaluations.
2. Human-in-the-Loop Learning Mechanism: Developed a semi-automated system where coordinator feedback directly refines the AI's prompt logic (tips) in real-time, creating a self-improving pipeline.
3. Tiered Efficiency Strategy: Introduced a "gatekeeper" criterion approach (Tier 1) to prevent unnecessary computation and human review for clearly ineligible patients, significantly optimizing resource usage.
4. Cost-Effectiveness Analysis: Provided a granular breakdown of compute costs, demonstrating that high-accuracy screening can be achieved at a negligible cost per patient.
5. Real-World Performance Metrics: Reported rigorous criterion-level performance metrics (Accuracy, Sensitivity, Specificity, PPV, NPV, F1) derived from human-verified ground truth in a live clinical environment.

4. Results

• Volume: The system prescreened 39,182 patients. Of these, 914 patients (high likelihood of eligibility) underwent coordinator review, covering 5,096 criterion evaluations.
• Performance Metrics (Weighted Averages):
  • Accuracy: 0.94 (95% CI: 0.92–0.96)
  • Sensitivity: 0.98 (95% CI: 0.97–0.99) – Highly effective at identifying potential candidates.
  • Specificity: 0.81 (95% CI: 0.71–0.88)
  • Positive Predictive Value (PPV): 0.95
  • Negative Predictive Value (NPV): 0.93
  • F1 Score: 0.97
• Impact of AI Confidence:
  • Patients with AI-labeled criteria $\ge$80% met were reviewed at a much higher rate (93.7%) compared to those with 60–80% (55.1%).
  • The "prescreen success" rate was significantly higher for the $\ge$80% group (43.5%) compared to the 60–80% group (19.1%), validating the AI's ability to triage high-yield candidates.
• Iterative Improvement: 27 criteria prompts were automatically updated based on feedback. Figure 1 illustrates a specific case where performance on a "prior therapy" criterion improved from poor to near-ceiling levels after the system learned to recognize non-standard trial drugs via prompt refinement.
• Cost: The total Azure compute cost was $1,283.48 for 10,596 evaluated patients, averaging $0.12 per patient.

5. Significance and Conclusion

This study validates that LLM-assisted, human-in-the-loop workflows are a feasible, scalable, and cost-effective solution for clinical trial screening.

• Operational Efficiency: The system acts as a powerful filter, enriching the coordinator's workload with high-probability candidates while filtering out the vast majority of ineligible patients, thereby reducing manual effort.
• Adaptability: The HITL mechanism allows the system to adapt to specific trial nuances and documentation patterns over time, addressing the "black box" limitation of static NLP models.
• Limitations & Future Work: The study notes that the tiered design may lead to false negatives if the Tier 1 gate is incorrectly labeled. Additionally, the system currently relies heavily on text retrieval, which can struggle with time-sensitive structured data (e.g., latest lab values) compared to direct database queries. Future work should focus on integrating agentic controllers to dynamically choose between structured data lookups and unstructured text retrieval, and on measuring downstream impacts on actual enrollment and accrual rates.

In summary, the authors provide a pragmatic template for integrating LLMs into routine clinical operations, achieving >94% agreement with human reviewers at a fraction of the cost of manual screening.