From Study Design to Executable Code: Automating Target Trial Emulation with Large Language Models

The paper introduces THESEUS, a framework that leverages large language models to automatically translate free-text study descriptions into standardized, executable R scripts for the OHDSI ecosystem, thereby demonstrating high accuracy in parameter extraction and code generation to lower technical barriers in observational research.

The Gist

Imagine you are a chef trying to recreate a famous dish from a food critic's review. The review says, "The soup was simmered for three hours with fresh basil and a pinch of salt."

In the world of medical research, this review is a study design. The "soup" is the analysis of patient data to see if a drug works. The problem is that translating that simple sentence into a working recipe (code) is incredibly hard. If one chef uses a gas stove and another uses an electric one, or if one measures salt in grams and the other in teaspoons, the final soup tastes different. In medical research, this means two teams studying the same drug might get different results just because they wrote their computer code differently.

This paper introduces THESEUS, a new tool that acts like a super-smart, robotic sous-chef to solve this problem.

Here is how it works, broken down into simple steps:

1. The Problem: The "Translation Gap"

Medical researchers often write their study plans in plain English (like the food critic's review). But to run the study on a computer, they need to translate that English into a very specific, rigid computer language called Strategus (part of the OHDSI ecosystem).

• The Old Way: A human researcher has to read the English plan, understand the complex math, and then manually type out hundreds of lines of code. This is slow, prone to typos, and hard to copy-paste between different hospitals.
• The Goal: We want to type a sentence in English and instantly get a perfect, error-free computer program.

2. The Solution: The "Two-Step Robot" (THESEUS)

The researchers built a system called THESEUS that uses Large Language Models (LLMs)—the same AI technology behind chatbots—to do the translation. It works in two distinct phases:

Step A: The "Architect" (Standardization)

First, the AI reads the messy, free-text description of the study (e.g., "We watched patients for one year after they started the drug").

• The Analogy: Imagine the AI is an architect reading a client's vague sketch. It doesn't just guess; it forces the sketch into a strict blueprint (a JSON file).
• It asks: "Did you mean 365 days? Did you mean to start counting the day the drug was given?"
• It fills out a standardized form where every field has a specific rule. This ensures that "one year" is always understood as "365 days" and "start date" is always in the same format.

Step B: The "Builder" (Code Generation)

Once the blueprint is perfect, the AI switches roles to become a construction worker.

• The Analogy: Now that the architect has the perfect blueprint, the builder doesn't need to guess where the walls go. They just follow the blueprint to build the house.
• The AI takes the standardized blueprint and automatically writes the R code (the computer program) needed to run the study.
• The Safety Net: The AI has a "self-check" feature. If the code it wrote has a typo or an error, the AI reads the error message, fixes the code, and tries again until it runs perfectly.

3. The "Human-in-the-Loop" (The Taste Test)

The researchers didn't just let the robot run wild. They built a Graphical User Interface (GUI)—a visual dashboard that looks like a control panel.

• The Analogy: Think of it like a video game character creator. The AI suggests the settings based on your text, but you can look at the screen, see the "Time at Risk" or "Drug Match" settings, and say, "Actually, change that to 2 years," or "No, keep it as is."
• This ensures a human expert always has the final say before the code is generated.

4. The Results: Does the Robot Cook Good Soup?

The team tested this system on 15 real medical studies and 5 studies from outside their network.

• Accuracy: The AI was incredibly good at turning English text into the correct "blueprint" (about 90-98% accurate for studies already using their system).
• Code Success: When the AI wrote the code, it worked on the first try about 80-100% of the time. If it failed, the "self-check" feature fixed it, bringing the success rate to nearly 100%.
• Generalization: It even worked well on studies that didn't originally use their system, proving it can translate ideas from different "kitchens."

Why This Matters

Before this, only experts who knew both medical statistics and advanced computer programming could easily run these studies.

• The Impact: THESEUS lowers the barrier to entry. It allows more researchers (and potentially more diverse teams) to run high-quality, reproducible studies just by describing what they want to do in plain English.
• The Future: It turns the chaotic, messy process of writing research code into a standardized, reliable assembly line.

In short: THESEUS is a translator that turns your "I want to study this drug" into a "Here is the perfect, error-free computer program to study that drug," ensuring that everyone in the world is cooking the same recipe, no matter which kitchen they are in.

Technical Summary

1. Problem Statement

Comparative Effectiveness Research (CER) using observational data, specifically Target Trial Emulation (TTE), is essential for generating real-world evidence when randomized controlled trials are infeasible. However, a critical bottleneck exists in translating conceptual study designs (described in natural language) into executable analytic code.

• Technical Barrier: Researchers require expertise in both causal inference methodology and programming (R/Python) to implement TTE.
• Reproducibility Crisis: Custom code written by different teams for similar study designs often leads to divergent results due to implementation differences (variable naming, data transformations).
• Lack of Standardization: While the OHDSI (Observational Health Data Sciences and Informatics) ecosystem provides a standardized data model (OMOP CDM) and analysis framework (Strategus), the process of converting study protocols into the specific JSON specifications required by Strategus remains manual and error-prone.

2. Methodology: THESEUS Framework

The authors developed THESEUS (Text-guided Health-study Estimation and Specification Engine Using Strategus), a two-step framework that leverages Large Language Models (LLMs) to automate the translation of free-text study descriptions into executable Strategus R scripts.

Step 1: Standardization (Text-to-JSON)

• Input: Free-text descriptions of study designs (e.g., study period, time-at-risk, propensity score adjustment strategies) extracted from published papers.
• Process: An LLM maps the unstructured text into a constrained JSON schema based on the OHDSI Strategus specification.
• Mechanism: The LLM is prompted with the text, a JSON template, and detailed field descriptions (referencing OHDSI guidelines) to ensure accurate extraction of parameters like study start/end dates, risk windows, and matching calipers.
• Output: A structured, machine-readable JSON analysis specification.

Step 2: Code Generation (JSON-to-R)

• Input: The structured JSON specification generated in Step 1.
• Process: A second LLM pass converts the JSON into an executable R script (CreateStrategusAnalysisSpecification.R) using the OHDSI Strategus package.
• Self-Auditing Loop: A critical innovation where the LLM attempts to execute the generated script. If errors occur, the error log is fed back to the LLM to diagnose and correct the code iteratively until it runs successfully.
• Human-in-the-Loop: A graphical user interface (GUI) prototype was developed (mimicking the OHDSI ATLAS interface) allowing researchers to review and validate the LLM-generated specifications before code generation.

Evaluation Design

• Datasets:
  • Internal Validation: 15 OHDSI-based TTE studies (using OMOP CDM).
  • External Validation: 5 non-OHDSI TTE studies (to test generalizability).
• Input Settings: Evaluated under three conditions: (1) Primary analysis text only, (2) Full analyses text (including sensitivity analyses), and (3) Full methods sections.
• Models Tested: Eight proprietary LLMs from four vendors (OpenAI, Google, Anthropic, DeepSeek).
• Metrics:
  • Standardization: Study-level accuracy and field-level sensitivity/precision/false positives.
  • Code Generation: Executability rates (first-run and post-self-auditing).

3. Key Contributions

1. End-to-End Automation: Demonstrated the first framework capable of translating natural language study protocols directly into executable, reproducible R code within the OHDSI ecosystem.
2. Structured Constraint Strategy: Proved that combining a standardized data model (OMOP CDM) with a rigid analysis specification format (Strategus JSON) allows LLMs to overcome the "open-ended" nature of coding, turning it into a deterministic mapping problem.
3. Self-Correction Mechanism: Validated that a self-auditing loop significantly improves code reliability, pushing executability rates to near-perfect levels.
4. GUI Integration: Developed a prototype interface that bridges the gap between natural language input and the complex configuration of OHDSI tools, lowering the barrier to entry for non-programmers.

4. Results

Standardization Performance (Text-to-JSON)

• OHDSI Studies:
  • Primary Analysis: High study-level accuracy across models (0.91–0.98).
  • Full Methods: Accuracy dropped slightly (0.67–0.85) due to increased complexity and noise in full text, but remained robust.
  • Field-Level: High sensitivity (0.73–0.90) with low false positive rates (0.27–0.67 per study) for OHDSI studies.
• Non-OHDSI Studies:
  • Performance was lower but still significant, particularly for primary analysis (0.73–0.93 accuracy).
  • Accuracy dropped significantly (0.27–0.47) when processing full methods sections of non-OHDSI studies, highlighting the challenge of varying terminology outside the OMOP CDM context.
• Top Performers: Claude-Opus-4.5 and DeepSeek-V3.2 generally achieved the highest accuracy.

Code Generation Performance (JSON-to-R)

• Executability:
  • OHDSI Studies: First-run executability ranged from 0.80 to 1.00. After self-auditing, this improved to 0.93–1.00 for all models.
  • Non-OHDSI Studies: First-run executability ranged from 0.60 to 1.00. After self-auditing, all models achieved 1.00 executability.
• Conclusion: The self-auditing step is crucial for reliability, effectively correcting syntax errors and logical mismatches.

5. Significance and Impact

• Democratization of Research: By lowering the technical barrier to entry, THESEUS enables researchers without deep programming expertise to conduct rigorous, reproducible TTE studies using standardized data.
• Enhanced Reproducibility: Automating the translation from protocol to code ensures that the same study design yields identical analytic logic across different research sites, addressing a major source of variability in observational research.
• Scalability: The framework allows for the rapid reconstruction of analytic specifications from published literature, facilitating retrospective verification and meta-analyses.
• Future Directions: While currently limited to the Cohort Method and specific PS adjustment strategies (matching/stratification), the modular architecture suggests potential expansion to other study designs (e.g., patient-level prediction) and integration with cohort definition tools (like Criteria2Query) for full end-to-end automation.

Limitations: The study is limited to TTE designs and specific PS methods; the sample size for evaluation was small (15 OHDSI, 5 non-OHDSI); and it relies on the prerequisite of having data converted to OMOP CDM.

Conclusion: THESEUS demonstrates that the combination of standardized data infrastructure and structured LLM prompting can reliably automate the most labor-intensive step in observational research—coding—thereby accelerating the generation of high-quality real-world evidence.