💬 NLP

CUICurate: A GraphRAG-based Framework for Automated Clinical Concept Curation for NLP applications

The paper introduces CUICurate, a scalable and cost-effective GraphRAG framework that leverages a UMLS knowledge graph and large language models to automate the generation of comprehensive, clinician-reviewable concept sets for clinical NLP applications, outperforming manual curation in recall and completeness.

Original authors: Victoria Blake, Jamie Novak, Mathew Miller, Sze-yuan Ooi, Blanca Gallego

Published 2026-07-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Victoria Blake, Jamie Novak, Mathew Miller, Sze-yuan Ooi, Blanca Gallego

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Lost in Translation" Library

Imagine the Unified Medical Language System (UMLS) as the world's biggest, most chaotic medical library. It has over 4 million books (concepts) describing every possible medical condition, symptom, and treatment.

When doctors write notes or when computers try to read them, they need to find the right "book" for a specific idea. For example, if a doctor writes "heart failure," the computer needs to know exactly which books in the library count as "heart failure." Does it include "congestive heart failure"? What about "left-sided heart failure"?

The Problem: Currently, building these "lists of related books" (called concept sets) is like trying to find every single book about a specific topic in that massive library by walking the aisles yourself. It takes a long time, it's exhausting for the experts (clinicians), and different experts often end up with different lists. This makes it hard for computers to compare data from different hospitals reliably.

The Solution: CUICurate (The Smart Librarian)

The authors built a new tool called CUICurate. Think of it as a super-smart, automated librarian that uses two special skills to build these lists for you:

The Map (GraphRAG): Instead of just searching for keywords, this librarian looks at a giant map of how all the medical books are connected. If you ask for "Heart Failure," the map shows the librarian not just the main book, but all the books linked to it (like subtypes, symptoms, and related conditions).
The Expert Brain (Large Language Models): Once the librarian finds a pile of potentially relevant books using the map, a highly intelligent AI (like GPT-5 or Qwen3) reads the summaries of those books. It acts like a senior doctor reviewing the list, saying: "Yes, this book definitely belongs on the list," or "No, this one is too vague, let's leave it out."

How It Works (The Recipe)

The paper describes a three-step process:

Building the Map: They took the UMLS library and turned it into a digital network (a graph) where every medical concept is a dot, and the relationships between them are lines connecting the dots. They also gave every dot a "digital fingerprint" (embedding) so the computer understands the meaning of the words, not just the spelling.
The Search (Retrieval): When they want a list for a specific condition (like "Ischemic Stroke"), the system uses the map to find the starting point and then walks along the connecting lines to find related concepts. It grabs a "seed" list of candidates.
The Review (Filtering): The AI reads this candidate list. It filters out the junk and organizes the good stuff into two piles:
- Definitive: "This is definitely heart failure."
- Context-Dependent: "This might be heart failure, depending on other details."

What They Found (The Results)

The team tested this on five different medical conditions (like heart failure, stroke, and poor mobility). Here is what happened:

It found way more than humans: The lists created by CUICurate were much bigger and more complete than the lists created by human experts manually browsing the library. The human lists missed many important variations.
It was very accurate: The AI (specifically GPT-5) kept almost all the "definitive" concepts (over 95%) that the gold-standard experts agreed on.
It was cheap and fast:
- Humans: It took a doctor 1–2 hours to build one list manually.
- CUICurate: It cost less than $10 to set up the map and less than $1 to generate a single list. It took about 20 minutes of computer time.
The "Missed" Items: Some items were missed by the computer, but when the team checked 10,000 real patient notes, they found that many of the "missed" items never actually appeared in the notes anyway. So, missing them didn't matter much in practice.

The Analogy: Finding the Perfect Playlist

Imagine you want to create a playlist for a "90s Rock" party.

The Old Way (Manual): You ask a music fan to write down every song they think is 90s rock. They might forget some deep cuts or only remember the hits. It takes them all day.
The CUICurate Way: You give a robot a map of the entire music industry showing how songs are related. The robot finds 500 songs that might fit. Then, a music critic AI listens to all 500 and picks the best 100, ensuring no great song is left out, but no pop song sneaks in. It does this in minutes for pennies.

The Bottom Line

The paper concludes that CUICurate is a scalable, cost-effective way to build the "lists" that computers need to understand medical text. It doesn't replace the doctor; instead, it gives the doctor a pre-sorted, high-quality list to review, saving them hours of work and ensuring the computer understands the medical data much better than before.

Key Takeaway: By combining a "map" of medical knowledge with an "AI expert" to filter the results, the authors created a system that builds better medical concept lists faster and cheaper than humans can do alone.

Technical Summary: CUICurate

Problem Statement
Clinical Natural Language Processing (NLP) applications frequently rely on mapping free-text clinical terms to Unified Medical Language System (UMLS) Concept Unique Identifiers (CUIs). While the UMLS provides broad coverage, its inclusivity creates challenges: concepts often overlap, use ambiguous wording, or lack consistent hierarchical links. For many downstream tasks, such as phenotyping, the clinically meaningful unit is not a single CUI but a "concept set"—a collection of related synonyms, subtypes, and associated concepts. Currently, constructing these sets is a labor-intensive, manual process reliant on specialist clinical judgment and iterative searching. Existing tools, such as the UMLS Metathesaurus Browser or the OHDSI Athena platform, offer limited support for automated, coherent concept set generation, particularly for NLP pipelines that operate on free text rather than structured coded data. Furthermore, relying solely on hierarchical expansion is insufficient, as relevant variants may exist outside direct parent-child relationships or across different branches of the ontology.

Methodology
The authors present CUICurate, a GraphRAG-based framework designed to automate the curation of UMLS concept sets. The framework operates through a three-stage pipeline:

Knowledge Graph Construction and Embedding:
- A bi-directed knowledge graph is constructed from UMLS Rich Release Format (RRF) files, utilizing SNOMED CT–US, NCI Thesaurus (NCIt), and Medical Subject Headings (MeSH).
- Nodes represent CUIs, and edges represent semantic relationships.
- Nodes are embedded using three models: text-embedding-3-large (OpenAI), SapBERT (domain-specific), and e5-large-v2 (open-source general-purpose).
- Embeddings are indexed using FAISS (Facebook AI Similarity Search) for semantic retrieval.
Graph Retrieval (Candidate Generation):
- For a given target concept, a recall-oriented description is embedded to retrieve the top- $k$ semantically similar "seed" CUIs.
- The system performs hierarchical graph expansion (recursive child expansion and one-hop parent/synonymous expansion) to capture related concepts.
- Candidate sets are constrained by semantic type filters and neighborhood limits (max_neighbors) to ensure the resulting set is computationally manageable for downstream processing.
LLM-Based Curation (Filtering and Classification):
- Retrieved CUIs are processed by Large Language Models (LLMs) in a two-stage workflow:
  - Filtering: Retaining concepts that indicate or strongly suggest the target phenotype.
  - Classification: Categorizing included CUIs as either "definitive" (explicitly indicating the concept) or "context-dependent" (requiring additional clinical information).
- Two models were evaluated: GPT-5 (via OpenAI API) and Qwen3-32B (open-source, locally deployed).

Evaluation Strategy
The framework was evaluated on five lexically heterogeneous clinical concepts: Chronic Heart Failure (CHF), Ischaemic Stroke (IS), Fluid Overload (FO), Left Ventricular Systolic Dysfunction (LVSD), and Poor Mobility (PM).

Benchmarks: Manual concept sets (M) were created by clinicians using the UMLS browser.
Gold Standard: An adjudicated gold-standard set ( $G_a$ ) was constructed using pooled relevance assessment, combining candidates from multiple retrieval strategies and embedding models, then independently annotated by clinicians.
Metrics: Retrieval was evaluated using recall against $G_a$ . Filtering performance was measured using precision, recall, and F1-score against the subset of $G_a$ present in the retrieved candidates.
Observability: Missed concepts were analyzed for their presence in 10,000 MIMIC-III clinical notes to assess practical significance.

Key Results

Concept Set Size and Completeness: CUICurate generated concept sets substantially larger and more complete than manual benchmarks. Manual sets ranged from 30–130 CUIs, whereas gold-standard sets ranged from 163–675 CUIs.
Retrieval Performance: The text-embedding-3-large model outperformed domain-specific and other open-source models. A single configuration (500 seed CUIs, one-hop expansion, max 1,000 neighbors) achieved high recall for definitive concepts (exceeding 0.90 for all concepts except Poor Mobility) while keeping candidate sets manageable (~750 CUIs per concept).
LLM Filtering Performance:
- GPT-5 outperformed manual curation across all concepts. It retained at least 95% of definitive gold-standard CUIs for all five target concepts, despite lower recall for the full set of context-dependent concepts.
- Qwen3-32B achieved comparable but slightly lower performance than GPT-5, yet still significantly outperformed manual curation in recall.
Clinical Observability: A significant portion of missed gold-standard CUIs (particularly context-dependent ones) were not observed in the sampled MIMIC-III notes, suggesting that some recall losses may have limited practical impact on clinical text mining.
Cost and Efficiency: The pipeline is cost-efficient. One-time embedding cost was approximately $8.20 USD. Per-query processing with GPT-5 cost roughly $0.88 USD and took ~21 minutes, compared to 1–2 hours of clinician time required for manual curation.

Significance and Claims
The paper claims that CUICurate offers a scalable, reproducible, and cost-efficient approach to generating clinician-reviewable UMLS concept sets. Its primary contribution is demonstrating that a GraphRAG framework can effectively constrain the search space of the massive UMLS ontology, allowing LLMs to prioritize clinically relevant concepts for expert review.

The authors emphasize that this approach addresses the limitations of purely hierarchical retrieval and manual curation. By automating the initial expansion and filtering, CUICurate reduces the manual burden while producing concept sets that are more comprehensive than those created via traditional UMLS browser searches. The framework is positioned not as a fully automated "black box" solution, but as an iterative design tool that enables researchers to rapidly generate study-specific concept sets tailored to different analytical objectives, balancing sensitivity and specificity with minimal computational cost.