EmulatRx: Empowering Clinical Trial Design with Agentic Intelligence and Real World Data

This paper introduces EmulatRx, an agentic framework that leverages large-scale real-world data and multi-role agent collaboration to autonomously refine clinical trial protocols and generate actionable insights, thereby accelerating the design process for both acute and chronic diseases.

The Gist

Imagine you are a chef trying to create the perfect recipe for a new dish. In the medical world, this "dish" is a Clinical Trial—a study designed to test if a new medicine works.

Traditionally, designing this recipe is a slow, expensive, and exhausting process. It requires a team of experts: a head chef (the Clinician) who knows what flavors work, a sous-chef (the Statistician) who calculates the exact math to ensure the dish isn't poisonous, and a pantry manager (the Informatician) who has to dig through thousands of dusty, messy notebooks (Real-World Data) to find the right ingredients. They have to meet in person, argue over details, rewrite the recipe, and start over. This can take months or even years.

EmulatRx is like a super-smart, robotic kitchen crew that does all of this work for you in a matter of minutes.

Here is how it works, broken down into simple parts:

1. The Problem: The "Messy Pantry"

Hospitals have massive amounts of data (Electronic Health Records) about real patients. It's like a giant pantry filled with millions of jars of ingredients. But the labels are messy, some jars are empty, and the ingredients are mixed up.

• The old way: Humans spend years trying to find the right jars, clean the labels, and figure out if they can make a specific dish.
• The new way: EmulatRx is a team of AI robots that can instantly read the messy labels, clean the jars, and tell you exactly what you have.

2. The Team: A Digital Kitchen Crew

EmulatRx isn't just one robot; it's a team of five specialized AI agents that talk to each other, just like a real kitchen staff:

• The Supervisor (The Head Chef): The boss. It doesn't cook; it manages the team, makes sure everyone is doing their job, and decides when the recipe is finally ready.
• The Trialist (The Recipe Researcher): This agent goes to the library (and the internet) to find existing recipes (clinical trials) that are similar to what you want to cook. It reads them and summarizes the key steps.
• The Informatician (The Pantry Manager): This agent takes the recipe and looks into the messy pantry (the hospital data). It figures out: "Do we actually have these ingredients?" If a recipe says "fresh organic tomatoes" but the pantry only has "canned tomatoes," this agent asks the Chef if that's okay or if they need to find a substitute.
• The Clinician (The Taste-Tester): This agent is the medical expert. It checks if the recipe makes sense medically. "Wait, if we mix these two drugs, will it hurt the patient?" It uses its knowledge of medical books to ensure the plan is safe.
• The Statistician (The Math Wizard): This agent runs the numbers. It simulates the cooking process thousands of times to predict: "If we cook this for 1,000 people, will 90% get better? Will 5% get sick?" It ensures the results are mathematically solid.

3. How They Work Together (The "Meeting")

In the old days, these experts would meet once a week. In EmulatRx, they have instant meetings.

• The Pantry Manager says, "I can't find enough patients with this specific heart condition."
• The Head Chef immediately asks the Taste-Tester, "Can we relax the rule slightly?"
• The Taste-Tester checks the medical books and says, "Yes, if we include patients with slightly higher blood pressure, it's still safe."
• The Pantry Manager updates the list, and the Math Wizard re-runs the numbers.
• This happens in seconds, not weeks.

4. The Magic Ingredient: "Learning from Feedback"

The system is designed to get smarter. If a human expert looks at the robot's recipe and says, "That doesn't look right," the system learns from that feedback. It's like a cooking apprentice who gets better every time the master chef corrects their knife skills. Over time, the AI team becomes incredibly good at designing these trials.

5. The Result: A "Simulation" Before the Real Thing

Before spending millions of dollars and years of time on a real trial with real patients, EmulatRx runs a simulation.

• It uses real data from thousands of past patients to pretend the trial has already happened.
• It tells the researchers: "Based on real-world data, this new drug looks like it will work great for young patients but might not work for older ones," or "This trial design is too strict; we won't find enough people to test it."
• This saves money, saves time, and most importantly, keeps patients safe by catching bad ideas before they ever reach a real person.

In Summary

EmulatRx is a digital orchestra conducting the complex symphony of clinical trial design. Instead of a human conductor struggling to keep 50 musicians in sync, this AI conductor ensures every section (the doctors, the data experts, the mathematicians) plays perfectly together, instantly turning a chaotic pile of medical data into a clear, safe, and efficient plan for saving lives.

It turns a process that used to take years into one that takes minutes, allowing doctors to focus less on paperwork and more on curing diseases.

Technical Summary

1. Problem Statement

Clinical Trial Design (CTD) is a resource-intensive, time-consuming process requiring deep domain expertise. While Real-World Data (RWD), such as Electronic Health Records (EHRs), offers immense potential to inform trial design through Target Trial Emulation (TTE), current implementation faces significant hurdles:

• Mapping Complexity: Translating natural language trial protocols (eligibility criteria, treatments, outcomes) into computable queries for structured EHR data (e.g., OMOP CDM) is difficult and error-prone.
• Data Gaps: RWD often lacks specific biomarkers or unstructured details found in protocols, requiring expert judgment to determine surrogates or exclusions.
• Bias and Confounding: Observational data introduces selection bias and confounding, necessitating sophisticated causal inference methods to estimate "true" treatment effects.
• Iterative Bottleneck: Extracting Real-World Evidence (RWE) currently requires extensive, manual, iterative conversations between data scientists and domain experts to refine protocols, which is slow and inefficient.

2. Methodology: The EmulatRx Framework

EmulatRx is a multi-agent system (MAS) designed to autonomously execute the end-to-end CTD workflow. It leverages Large Language Models (LLMs) augmented with tools, structured memory, and Reinforcement Learning from Human Feedback (RLHF).

A. Architecture and Agents

The system comprises five specialized agents, orchestrated by a central Supervisor:

1. Supervisor: The central decision-maker that orchestrates the workflow, coordinates interactions, and determines iteration vs. termination.
2. Trialist: Retrieves and standardizes trial information from registries (ClinicalTrials.gov) and literature. It constructs a Clinical Trial Knowledge Graph to map protocols to structured concepts.
3. Informatician: Bridges trial protocols with RWD. It translates eligibility criteria into executable SQL queries (using the OMOP CDM), handles unstructured clinical notes via NLP (NLI models), and constructs analytics-ready cohorts.
4. Clinician: Provides domain expertise. It validates trial designs, suggests covariates, interprets statistical results, and uses Retrieval-Augmented Generation (RAG) over biomedical literature to ground decisions.
5. Statistician: Performs causal inference and statistical analysis. It selects balancing methods (e.g., Propensity Score Matching, IPTW), handles immortal time bias (via clone-censor-weighting), conducts survival analysis, and performs subgroup/eligibility optimization.

B. Key Technical Components

• Knowledge Graph & Standardization: Uses a Neo4j-backed graph to store trial components (eligibility, treatments, outcomes) mapped to standard vocabularies (OMOP, UMLS, RxNorm, LOINC).
• Tool-Augmented Reasoning: Agents utilize specific tools:
  • Trial Retriever: Queries ClinicalTrials.gov.
  • RAG Module: Grounds clinical reasoning in PubMed/literature.
  • Trial Simulator: Executes statistical libraries (Cox PH, Random Survival Forests) for confounder adjustment and effect estimation.
• Iterative Refinement Loop: Agents engage in "meetings" (structured conversations) to resolve issues (e.g., data sparsity, covariate imbalance). If the Informatician finds missing data, the Clinician suggests surrogates; if the Statistician finds bias, the system iterates on the protocol.
• Advanced Capabilities:
  • RLHF: Uses PPO and DPO to align agent outputs with expert preferences.
  • Eligibility Criteria (EC) Optimization: Uses Shapley values to quantify how specific inclusion/exclusion rules impact treatment effect estimates.
  • Adaptive Sample Size: Calculates required sample sizes using the Schoenfeld formula based on real-world event rates.
  • Safety Modeling: Automated adverse event (AE) identification and modeling.

3. Key Contributions

1. First End-to-End Agentic CTD System: Unlike previous tools that handle isolated subtasks, EmulatRx automates the entire pipeline from protocol retrieval to final causal inference report.
2. Dynamic Multi-Agent Collaboration: Introduces a flexible, non-linear workflow where agents can dynamically consult one another (e.g., Statistician asking Clinician for covariate adjustments) rather than following a rigid pipeline.
3. Integration of Unstructured Data: Successfully combines structured EHR queries with NLP-based analysis of unstructured clinical notes to improve cohort identification.
4. Methodological Rigor: Incorporates advanced causal inference techniques (Doubly Robust Estimation, Immortal Time Bias correction) directly into the agent workflow.
5. RLHF Integration: Demonstrates a framework for continuously improving agent performance through expert feedback loops.

4. Results

The system was evaluated on 20 clinical trials across two datasets:

• MIMIC-IV: Acute conditions (Septic shock, Acute Heart Failure, AKI).
• INSIGHT Network: Chronic conditions (Alzheimer's, Parkinson's).

Performance Metrics:

• Trialist (Query & Parsing): Using GPT-4o, the system achieved 100% precision and recall in retrieving relevant trials from the knowledge graph. For entity parsing, GPT-4o achieved 98.9% recall and 96.7% precision, significantly outperforming other models (Phi-4, DeepSeek-R1, Gemma-3) in handling complex criteria and omitted concepts.
• Informatician (SQL Generation): GPT-4o demonstrated the highest accuracy in generating executable SQL, with error rates significantly lower than local models, especially for complex eligibility criteria.
• Statistician (Causal Inference): On synthetic data with known ground truth, the Statistician agent accurately recapitulated Hazard Ratios (HR) and Average Treatment Effects (ATE) across different LLM backends. It successfully identified null effects in aggregate data and uncovered significant subgroup effects (heterogeneity) upon stratification.
• Clinician (Reasoning): Expert evaluation (5-point Likert scale) showed GPT-4o achieved the highest scores (avg 4.88) for readability, correctness, and usefulness, significantly outperforming local models.
• Efficiency: EmulatRx reduced the trial design process from days/weeks (manual) to ~5.75 minutes (median runtime with GPT-4o).
• Case Studies: Successfully emulated trials for Nesiritide (Heart Failure), Renal Replacement Therapy (AKI), and Hydrocortisone (Septic Shock), generating reports with statistically significant findings (e.g., HR 0.59 for Nesiritide) and identifying safety signals.

5. Significance and Impact

• Acceleration of Research: EmulatRx drastically reduces the time and cost associated with designing clinical trials and generating hypotheses from RWD.
• Democratization of Expertise: By encoding domain expertise into agents, it lowers the barrier for researchers to perform complex target trial emulations without needing a full team of specialists for every step.
• Improved Trial Feasibility: The system's ability to simulate trials on RWD before launching a real RCT helps identify feasibility issues, optimize sample sizes, and refine eligibility criteria, potentially increasing the success rate of future randomized trials.
• Scalability: The modular, agent-based architecture allows for easy integration of new data sources, medical domains, or statistical methods, making it a scalable solution for the future of precision medicine and drug development.

In conclusion, EmulatRx represents a paradigm shift from manual, expert-driven trial design to an autonomous, intelligent, and collaborative agentic workflow, effectively bridging the gap between real-world data and rigorous clinical evidence generation.