Evaluating Large Language Models for Translating Multimodal Phenotype Documentations into Executable EHR Phenotyping Algorithms

This study evaluates frontier large language models for translating multimodal clinical phenotype documentation into executable EHR algorithms, finding that while they effectively interpret structured text, their performance significantly degrades with diagram-only inputs, ultimately identifying documentation quality rather than model capability as the primary bottleneck.

The Gist

Imagine you are a master chef trying to recreate a famous dish, but you don't have the recipe. Instead, you have a messy stack of notes, some scribbled on napkins, some drawn as cartoons, and some written in a confusing mix of languages. Your goal is to turn these messy notes into a precise, step-by-step instruction manual that a robot kitchen can follow to cook the dish perfectly.

This paper is about testing two super-smart AI chefs (called Large Language Models, or LLMs) to see if they can do this job for medical research.

The Problem: The "Lost in Translation" Recipe

In medical research, scientists define specific groups of patients (like "people with Type 2 Diabetes") using complex rules. These rules are usually written in human-readable documents that look like a mix of stories, flowcharts, and tables.

To use these rules in a hospital's computer system, a human expert has to manually translate them into a computer language (SQL). This is like translating a poem into a computer code. It takes a long time, is very tedious, and if two different experts do it, they might end up with slightly different results. The researchers wanted to see if AI could do this translation automatically.

The Experiment: Testing the AI Chefs

The researchers picked two of the smartest AI models available (OpenAI's GPT o3 and Anthropic's Claude Opus 4.1) and gave them five different "recipes" (medical definitions for conditions like kidney injury, heart attacks, and diabetes) from a public library called PheKB.

They tested the AI in three different ways, like giving the chef different types of instructions:

1. The Full Package: The AI got the whole document (text, charts, and diagrams).
2. Just the Story: The AI got only the written text and tables, but no pictures.
3. Just the Pictures: The AI got only the diagrams and flowcharts, with no words.

The Results: What Worked and What Didn't

1. The "Picture-Only" Trap
When the AI tried to read only the diagrams (the flowcharts), it failed miserably. It was like asking a chef to cook a complex meal just by looking at a drawing of a pot and a fork, with no text explaining the ingredients or heat levels. The AI missed crucial details, got the timing wrong, and produced instructions that wouldn't work.

2. The "Story" is King
When the AI got the written text (even without the pictures), it did a very good job. It turned out that the written words contained almost all the information needed. The AI could understand the logic and write the computer code accurately.

3. The AI is a Great Draftsman, Not a Final Editor
Both AI models were surprisingly good at understanding the big picture and the logic of the rules. However, they made specific types of mistakes:

• Missing Ingredients: They sometimes forgot to include specific medical codes (like a specific type of medication).
• Wrong Numbers: They might get a threshold wrong (e.g., saying "blood pressure over 140" when the rule was "over 150").
• Making Things Up: Sometimes, the AI invented rules or conditions that weren't in the original document at all (a "hallucination").
• Confusing the Format: When looking at diagrams, they often couldn't figure out how to turn a visual arrow into a logical "if-then" computer command.

The Big Takeaway

The paper concludes that these AI models are not ready to replace human experts yet. They cannot just look at a messy document and spit out a perfect, ready-to-use computer program.

However, they are excellent first draft generators. If you give them clear, structured text, they can write a very good starting point for the code. But because they can make subtle but dangerous mistakes (like getting a number wrong or missing a rule), a human expert must always check their work.

The Final Lesson:
The biggest problem isn't that the AI isn't smart enough; it's that the medical documents aren't written in a way that is easy for computers to read. If doctors and researchers standardized their notes to be clearer and more structured (like writing a recipe in a standard format rather than scribbling on a napkin), the AI would become much more useful. Until then, the AI is a helpful assistant, but the human expert must remain the boss.

Technical Summary

Technical Summary: Evaluating Large Language Models for Translating Multimodal Phenotype Documentations into Executable EHR Phenotyping Algorithms

Problem Statement
The translation of clinical phenotype definitions into executable Electronic Health Record (EHR) database queries is a critical yet labor-intensive bottleneck in biomedical research. While phenotyping algorithms are essential for identifying patient cohorts at scale, they are typically documented in high-level, human-readable formats (narrative text, tables, flowcharts, diagrams) rather than computation-ready forms. Converting these multimodal specifications into structured query language (SQL) tailored to specific data models (e.g., OMOP CDM) requires manual effort, domain expertise, and iterative validation. This process introduces interpretive variability, limiting the portability, comparability, and reuse of cohorts across institutions. Although Large Language Models (LLMs) have demonstrated proficiency in code generation and multi-step reasoning, it remains unclear whether they can faithfully convert real-world, multimodal phenotype documentation into standards-conformant, executable SQL.

Methodology
This study systematically evaluated the capability of two frontier LLMs—OpenAI GPT o3 and Claude Opus 4.1—to translate phenotype documentation into OMOP-compliant SQL queries.

• Dataset: Five established clinical phenotypes were selected from PheKB: Acute Kidney Injury (AKI), Autoimmune Diseases (ADs), Familial Hypercholesterolemia (FH), Major Adverse Cardiovascular Events (MACE), and Type 2 Diabetes Mellitus (T2DM). These were chosen for their diverse clinical domains and varying documentation complexities.
• Input Modalities: To isolate the impact of documentation format, three input scenarios were tested for each phenotype and model:
  1. Full Document: Narrative text, tables, diagrams, and pseudo-code.
  2. Text Only: Narrative text and tables (excluding diagrams and pseudo-code).
  3. Diagram Only: Visual diagrams exclusively.
• Prompting Pipeline: A four-stage chain-of-thought pipeline was employed:
  1. Extraction of a structured pseudo-algorithm from the input.
  2. Verification of the pseudo-algorithm against the source document.
  3. Generation of executable SQL based on the verified pseudo-algorithm.
  4. Review and correction of the generated SQL.
• Evaluation: Two clinical phenotyping experts independently rated the generated queries on a 0–4 scale across seven dimensions: logical/Boolean accuracy, temporal constraint implementation, value-set precision, OMOP schema correctness, coding efficiency, human readability, and readiness for local deployment. Interrater agreement was substantial (Cohen's kappa > 0.79).
• Error Analysis: A qualitative review categorized failures into seven distinct types: logical fidelity errors, concept-set incompleteness, schema/implementation errors, threshold/parameter errors, hallucinations, under-specification/over-generalization, and modality interpretation limitations.

Key Results

• Input Modality Sensitivity: The quality of generated SQL was highly sensitive to input representation. Providing only diagrams resulted in a dramatic degradation in performance across all evaluation dimensions, particularly in value-set precision, temporal constraints, and logical accuracy. Conversely, text-only input yielded performance closely approximating that of full-document input, suggesting that narrative prose captures the majority of necessary information for SQL generation.
• Model Performance: Under full-document and text-only conditions, Claude Opus 4.1 generally achieved better or comparable scores in five of seven dimensions, while OpenAI o3 outperformed in logical/Boolean accuracy and readiness for local deployment. In diagram-only scenarios, performance gaps narrowed, though OpenAI o3 maintained an advantage in logical accuracy.
• Phenotype-Specific Variance: The Autoimmune Disease (ADs) phenotype yielded the highest ratings across both models. This was attributed to its documentation style, which utilized explicit, well-organized pseudo-code, contrasting with the less structured documentation of other phenotypes.
• Error Categories: The analysis identified seven failure modes. Logical fidelity errors (e.g., misinterpreting single diagnoses as sufficient for a condition requiring repeated events) and concept-set incompleteness (missing required concept IDs) were among the most consequential. Hallucinations, where models introduced unsupported clinical concepts or thresholds, were also observed. Modality interpretation limitations were most prominent in diagram-only inputs, where models failed to translate visual exclusion criteria into SQL logic.

Significance and Claims
The paper positions LLMs not as autonomous implementers of clinical phenotypes, but as capable "first-draft generators" whose utility is strictly gated by the quality of input documentation. The study claims that:

1. Documentation is the Primary Bottleneck: The fidelity of LLM-generated phenotyping queries depends more on the standardization and structure of the input documentation than on the inherent capabilities of the models. Structured natural language is necessary and largely sufficient for near-optimal performance, whereas visual diagrams alone are insufficient.
2. Expert Oversight is Mandatory: Due to the diversity and clinical significance of identified errors (including hallucinations and threshold inaccuracies), LLM-generated queries are not yet suitable for autonomous deployment. Mandatory expert review remains essential, particularly for temporal constraints and concept-set precision.
3. Path to Scalability: Standardizing phenotype documentation into implementation-oriented formats (such as explicit pseudo-code) offers a promising path to improving translation fidelity without requiring further advances in model architecture.
4. Collaborative Paradigm: The findings suggest a collaborative workflow where LLMs accelerate the initial drafting of phenotyping algorithms, which are then refined through structured human oversight, potentially enhancing the portability and reuse of clinical cohorts across research networks.

The authors acknowledge limitations, including the evaluation of only five phenotypes and two models, the reliance on expert scoring rather than gold-standard chart review for cohort accuracy, and the lack of systematic prompt optimization. Future work is suggested to focus on longitudinal benchmarking, larger review panels, and validation against gold-standard datasets.