AERO: An AI Agent for Adaptive Eligibility Refinement and Optimization of Clinical Trial Criteria in Real-World Trial Emulation

The paper introduces AERO, an AI agent framework that optimizes clinical trial eligibility criteria for real-world data emulation by leveraging large language models to systematically classify and refine criteria, thereby improving the generalizability and accuracy of treatment effect estimates as demonstrated in a WARCEF trial emulation.

The Gist

Imagine you are trying to recreate a famous, perfectly controlled cooking competition (a Randomized Controlled Trial, or RCT) using a giant, messy, real-world pantry full of ingredients from thousands of different households (your Electronic Health Records).

In the original competition, the judges had a very strict list of rules: "Only use eggs from chickens under 2 years old," "No salt if the cook has a specific allergy," and "The cook must be able to stand for 4 hours without a break." These rules ensured the competition was fair and the results were clear.

However, when you try to find these exact ingredients in the real-world pantry, you hit a wall. You can't tell the age of the chicken just by looking at the egg. You don't have a record of every cook's allergy history. And you certainly can't know if a cook could stand for 4 hours if they never actually had to. If you try to apply the original rules exactly as written, you might end up throwing away 90% of your pantry, leaving you with almost no cooks to study. Or worse, you might accidentally keep only the "perfect" cooks, making your results look different from the real world.

Enter AERO: The Smart Sous-Chef

The paper introduces AERO (AI Agent for Adaptive Eligibility Refinement and Optimization). Think of AERO as a highly intelligent, well-read sous-chef who helps you translate those strict competition rules into something workable for your messy real-world pantry, without losing the spirit of the original contest.

Here is how AERO works, using simple metaphors:

1. The "Four-Box" Sorting System

Instead of blindly trying to follow every rule, AERO looks at each rule and asks, "What is this rule really for?" It sorts every rule into one of four boxes:

• Box 1: The "Must-Haves" (Strict Inclusion): These are the core rules that define who the contest is for. Example: "The cook must be making soup." AERO keeps these as hard filters. If you aren't making soup, you're out.
• Box 2: The "Safety Warnings" (Strict Exclusion): These are rules about danger. Example: "No one with a severe nut allergy can enter." AERO keeps these too, because safety is non-negotiable and usually easy to spot in the records.
• Box 3: The "Background Noise" (Confounders): These are rules that describe the cook but don't necessarily disqualify them. Example: "The cook must have used a specific brand of salt in the past." In the real world, this might just be a factor that makes the soup taste different, not a reason to kick the cook out. AERO says, "Don't throw them out! Just write this down and adjust for it later when we taste the soup." This keeps more people in the study.
• Box 4: The "Impossible Tasks" (Drop/Operational): These are rules that make no sense in a real-world pantry. Example: "The cook must be able to follow a 4-hour protocol without a break." You can't check this in a database. AERO says, "We can't measure this, so let's drop this rule entirely so we don't accidentally exclude good cooks."

2. The "Knowledge Librarian"

AERO isn't just guessing. It acts like a librarian who pulls out three different books before making a decision:

• A Medical Encyclopedia (UpToDate) to understand the disease.
• A Smart AI Assistant (Claude) to interpret the context.
• A Drug Safety Manual (ToolUniverse) to check for dangerous interactions.

By combining the original trial rules with this extra knowledge, AERO decides which rules to keep, which to tweak, and which to toss.

3. The Test Drive: The WARCEF Trial

To see if AERO works, the researchers used it to recreate the WARCEF trial.

• The Original Trial: Compared Warfarin (a blood thinner) vs. Aspirin for heart failure patients. The result? No difference. The two drugs worked about the same.
• The Problem: If you tried to find these patients in real-world hospital records using the original strict rules, you'd likely get a tiny, weird group of patients that didn't look like real people.
• The AERO Solution: AERO re-sorted the rules. It kept the heart failure diagnosis (Must-Have) and the safety exclusions (Safety Warning). But it moved things like "recent pacemaker" or "specific medication history" into the "Background Noise" box, meaning they kept those patients but adjusted the math later.

The Result:
When they ran the study with AERO's optimized rules, they got a result of HR = 1.56 (which is a statistical way of saying "no significant difference"). This matched the original trial's conclusion (HR = 1.01, "no difference").

The "Ablation" Lesson (The "What If" Experiment)
The paper also ran a cool experiment to prove why AERO's sorting matters. They took one specific rule: "No patients on a specific blood thinner (LMWH)."

• Scenario A (Strict Rule): They threw everyone on that blood thinner out of the study. Suddenly, the results changed! It looked like one drug was better than the other. Why? Because by throwing those people out, they accidentally removed the sickest patients, skewing the group.
• Scenario B (AERO's Way): They kept those patients but treated the blood thinner as "Background Noise" to adjust for later. The result went back to "No difference," matching the original truth.

The Big Takeaway

The paper claims that how you decide who gets into a study changes the results.

If you try to copy-paste a strict lab trial into the messy real world, you might break the experiment. AERO acts as a translator. It uses AI and medical knowledge to say, "This rule is about safety, keep it. This rule is about logistics, drop it. This rule is just a characteristic, adjust for it."

By doing this, AERO allows researchers to use real-world hospital data to answer questions that usually require expensive, controlled trials, while ensuring the answer is still accurate and fair. It bridges the gap between the "perfect world" of a lab and the "messy world" of a real hospital.

Technical Summary

1. Problem Statement

The Challenge of Real-World Trial Emulation:
While Randomized Controlled Trials (RCTs) are the gold standard for causal inference, their restrictive eligibility criteria often limit the generalizability of results to broader, real-world populations. Target trial emulation attempts to replicate RCT designs using observational data (e.g., Electronic Health Records - EHRs) to generate Real-World Evidence (RWE). However, a critical bottleneck exists in translating protocol-defined eligibility criteria into EHR-based cohorts:

• Operational Infeasibility: Many trial criteria rely on measurements or assessments unavailable or inconsistently recorded in EHRs.
• Subjectivity and Bias: Adapting these criteria currently relies on manual, expert-driven decisions that are labor-intensive, difficult to reproduce, and prone to subjective bias.
• Selection Bias: Applying criteria verbatim can drastically reduce cohort sizes or exclude clinically relevant patients, distorting comparative effectiveness estimates. Conversely, ad-hoc relaxation of criteria without systematic reasoning can introduce new biases.

2. Methodology: The AERO Framework

The authors propose AERO (AI Agent for Adaptive Eligibility Refinement and Optimization), an agentic framework designed to automate the adaptation of clinical trial eligibility criteria for real-world data.

Core Architecture:
AERO utilizes a Knowledge-Augmented Large Language Model (LLM) (specifically GPT-5 in the study) as its reasoning core. It integrates three heterogeneous external knowledge sources to inform its decisions:

1. Clinical Knowledge Base: UpToDate (for disease background, indications, contraindications).
2. Medical LLM: Claude AI (for contextual clinical interpretation).
3. Drug Knowledge Tool: ToolUniverse (for structured pharmacological and safety data).

The Five-Stage Pipeline:

1. Extraction: Original eligibility criteria are extracted from trial protocols (e.g., ClinicalTrials.gov).
2. Retrieval: Relevant external medical knowledge is retrieved based on the trial context.
3. AI-Driven Optimization: The LLM analyzes each criterion against the protocol intent and external knowledge, classifying it into one of four categories:
   • Strict Inclusion: Essential characteristics defining the target population (e.g., specific diagnosis, ejection fraction).
   • Strict Exclusion: Safety-related or clinically necessary restrictions (e.g., pregnancy, severe liver impairment).
   • Confounder: Variables that influence outcomes but should be handled via statistical adjustment rather than cohort filtering (e.g., recent stroke, pacemaker use, specific medication use).
   • Drop / Operational: Criteria that are redundant, subjective, or infeasible to operationalize in EHRs (e.g., "ability to follow outpatient protocol," "Modified Rankin score").
4. Cohort Construction: The optimized criteria are applied to EHR data (Mayo Clinic Platform). Patients must satisfy all Inclusion criteria and none of the Exclusion criteria. Confounders are retained in the dataset for downstream statistical adjustment (e.g., propensity score matching, Cox regression).
5. Outcome Analysis: Comparative effectiveness is estimated using survival analysis (Cox proportional hazards models).

3. Key Contributions

1. First AI Agent for Eligibility Refinement: AERO is presented as a novel agentic approach specifically designed to systematically refine trial eligibility criteria for real-world emulation, moving beyond manual heuristics.
2. Knowledge-Grounded Reasoning: Unlike purely data-driven filtering, AERO explicitly integrates diverse external medical knowledge sources to ensure decisions are clinically grounded and reproducible.
3. Scalable Pipeline: The framework demonstrates a reproducible pipeline for translating RCT protocols into real-world studies, bridging the gap between controlled trials and routine clinical practice.

4. Results: WARCEF Trial Emulation

The framework was evaluated by emulating the WARCEF trial (Warfarin vs. Aspirin in Reduced Cardiac Ejection Fraction) using Mayo Clinic Platform data from the pre-trial completion period (before July 2014).

Cohort Characteristics:

• Demographics: The AERO-optimized cohort was slightly older and had a different gender distribution compared to the original trial cohort but preserved the treatment assignment ratios.
• Criteria Reclassification:
  • Retained as Strict: Heart failure diagnosis, reduced ejection fraction (EF ≤ 35%).
  • Reclassified as Confounders: Recent stroke, pacemaker insertion, NSAID use, creatinine levels, and need for antiplatelet therapy.
  • Dropped: Functional assessments and life-expectancy judgments not captured in EHRs.

Outcome Analysis:

• Primary Finding: The emulation using AERO-optimized criteria yielded a Hazard Ratio (HR) of 1.561 (p = 0.0605).
• Consistency: This result is consistent with the original WARCEF trial finding (HR = 1.01, p = 0.91), which concluded there was no statistically significant difference between warfarin and aspirin.
• Interpretation: Despite differences in cohort composition, the optimized criteria preserved the original trial's conclusion of neutrality, validating the framework's ability to maintain clinical intent while improving real-world applicability.

Ablation Study (Impact of Handling Decisions):
The authors conducted an ablation study on the criterion "requires continuing therapy with low-molecular-weight heparin (LMWH)."

• Scenario A (Strict Exclusion): Treating LMWH use as a strict exclusion criterion resulted in a statistically significant separation between treatment groups (p = 0.0347).
• Scenario B (Confounder): Treating LMWH use as a confounder (adjusted statistically) resulted in a non-significant difference (p = 0.0605).
• Conclusion: The study demonstrated that eligibility handling decisions materially influence observed treatment effects. Reclassifying a variable from a filter to a confounder prevented selection bias that would have artificially inflated the apparent treatment difference.

5. Significance and Implications

• Methodological Advancement: AERO shifts eligibility adaptation from a subjective, manual preprocessing step to a systematic, transparent, and AI-driven design component of causal inference.
• Bias Mitigation: By distinguishing between safety exclusions and confounders, the framework reduces selection bias caused by over-restrictive filtering in observational data.
• Reproducibility: The structured taxonomy (Inclusion, Exclusion, Confounder, Drop) enhances the reproducibility of target trial emulations across different studies and institutions.
• Future Outlook: As regulatory agencies increasingly rely on RWE, frameworks like AERO are critical for scaling the translation of RCT protocols into real-world evidence generation, ensuring that clinical trial conclusions remain valid when applied to diverse, routine clinical populations.

Limitations Noted: The study relied on a single trial case (WARCEF); broader validation is needed. Additionally, the framework relies on LLM reasoning which may be sensitive to prompt structures, and residual confounding inherent in EHR data remains a challenge.