💬 NLP

Retrieve, Then Classify: Corpus-Grounded Automation of Clinical Value Set Authoring

This paper introduces Retrieval-Augmented Set Completion (RASC), a framework that combines retrieval of similar existing value sets with code classification to effectively automate clinical value set authoring, demonstrating superior performance and reduced irrelevant candidates compared to direct generation by large language models or simpler retrieval-only approaches on a new large-scale benchmark.

Original authors: Sumit Mukherjee, Juan Shu, Nairwita Mazumder, Tate Kernell, Celena Wheeler, Shannon Hastings, Chris Sidey-Gibbons

Published 2026-04-17

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Sumit Mukherjee, Juan Shu, Nairwita Mazumder, Tate Kernell, Celena Wheeler, Shannon Hastings, Chris Sidey-Gibbons

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a librarian tasked with creating a "Special Book Club" list for a very specific topic, like "Books about 19th-century French poets who wrote about rain."

Your library is massive—it has millions of books (the medical codes). You need to pick out the exact few hundred books that fit this description to make your list perfect.

The Old Way: The "Guessing Game"

In the past, a human expert had to walk through the aisles, read thousands of titles, and guess which ones fit. It was slow and exhausting.

Then, people tried using a super-smart AI (like a very well-read robot) to just write the list from scratch. The robot would say, "Here are the books you need!"

The Problem: The robot is smart, but it doesn't have a physical library. It relies on its memory. When asked for specific, obscure codes, it starts hallucinating. It makes up book titles that don't exist in the library at all. In this paper, the robot made up nearly 50% of the titles it suggested!

The New Solution: RASC (The "Smart Assistant" System)

The authors of this paper propose a new two-step method called RASC (Retrieval-Augmented Set Completion). Think of it as hiring a smart assistant who works in two stages:

Stage 1: The "Similarity Search" (The Curator)

Instead of asking the AI to invent the list, the system first looks at other lists that already exist in the library.

Analogy: You ask the assistant, "I need a list for 'French poets about rain.' Show me the 10 other lists we have that are most similar to that."
The assistant pulls out 10 existing lists about "French literature," "Poetry," and "Weather."
It combines all the books from those 10 lists into a small, manageable pile (maybe 500 books).
Why this helps: We know the answer is almost certainly inside this small pile. We don't need to search the whole million-book library anymore. We've shrunk the problem from "Find a needle in a haystack" to "Find a needle in a shoebox."

Stage 2: The "Binary Classifier" (The Judge)

Now, the system takes that small pile of 500 books and asks a Judge (a specialized AI model) to look at each one individually.

The Question: "Does this specific book belong in the 'French poets about rain' list?"
The Judge says "Yes" or "No" for every single book in the pile.
Why this helps: It is much easier for an AI to say "Yes/No" to a specific item than to invent a whole list from scratch. The Judge is very good at spotting the right books and ignoring the wrong ones.

The Results: Why It Works Better

The researchers tested this on over 11,000 real medical lists. Here is what they found:

The Robot (Zero-Shot LLM): When asked to just "make the list," it failed miserably. It made up fake codes 48% of the time and missed most of the real ones.
The Old Way (Just Retrieval): If you just took the 10 similar lists and dumped them all together, you'd get the right answers, but you'd also have a lot of junk (irrelevant books) mixed in.
The RASC System: By using the "Curator" to find the pile and the "Judge" to filter it, they got the best of both worlds.
- They found almost all the correct codes.
- They drastically reduced the amount of junk (irrelevant codes) the human had to review.
- The system got smarter as the lists got bigger. The more complex the topic, the better RASC performed compared to the robot guessing from memory.

The Big Takeaway

The paper argues that in fields like medicine, where information is structured and vast, we shouldn't ask AI to "create" things from thin air. Instead, we should ask AI to find existing related information and then critically evaluate it.

It's the difference between asking a student to "Write a history essay from memory" (which leads to made-up facts) versus giving them a stack of relevant history books and asking, "Which of these paragraphs are true?" The second approach is faster, cheaper, and much more accurate.

In short: Don't let the AI guess the answer. Let the AI find the clues, and then let it decide which clues are the real deal.

1. Problem Definition

Clinical Value Set Authoring is the task of identifying all codes within a standardized medical vocabulary (e.g., SNOMED-CT, ICD-10-CM) that define a specific clinical concept (e.g., "Type 2 Diabetes").

The Challenge: The vocabulary universe ( $U$ ) is massive ( $10^5$ – $10^6$ codes), version-controlled, and structured. Manual authoring is a bottleneck, requiring experts to search vast databases to create precise, complete lists.
The LLM Limitation: Directly prompting Large Language Models (LLMs) to generate these codes fails because:
1. Medical codes are structured identifiers, not natural language, and are not reliably memorized during pretraining.
2. LLMs hallucinate significantly (48.6% of GPT-4o's returned codes were absent from the official VSAC corpus).
3. Ignoring existing expert-curated corpora wastes valuable prior knowledge.

2. Proposed Methodology: RASC

The authors propose Retrieval-Augmented Set Completion (RASC), a two-stage framework that shifts the problem from generative (creating codes from scratch) to discriminative (selecting codes from a candidate pool).

Stage 1: Semantic Retrieval

Goal: Reduce the search space from the full vocabulary $U$ to a small candidate pool $C$ .
Mechanism:
- The system embeds the target value set title using SAPBert (a BERT model pretrained on UMLS synonym pairs).
- It retrieves the $K$ (set to 10) most similar existing value sets from a curated corpus of 11,803 public VSAC sets.
- The union of codes from these $K$ sets forms the candidate pool $C$ .
Theoretical Benefit: This shrinks the effective output space from $N$ (total vocabulary) to $K$ (candidate pool), theoretically reducing sample complexity from $O(\log N)$ to $O(\log K)$ .

Stage 2: Binary Classification

Goal: Determine which codes in the candidate pool $C$ actually belong to the target set.
Mechanism: A lightweight binary classifier evaluates each $(value\_set, candidate\_code)$ pair.
Feature Representation: The input vector concatenates:
1. SAPBert embedding of the value set title.
2. SAPBert embedding of the candidate code's display name.
3. One-hot encoding of the code system (e.g., SNOMED, ICD-10).
4. The retrieval similarity score.
Models Tested:
1. LightGBM: Gradient-boosted trees (non-neural baseline).
2. MLP: Three-layer Multi-Layer Perceptron with independent embeddings.
3. Cross-Encoder: Fine-tuned SAPBert where the title and code are processed jointly through 12 transformer layers, allowing fine-grained token-level attention.

3. Key Contributions

Theoretical Framework: Formalized the set completion problem and derived a sample complexity gap showing that retrieval-restricted prediction requires significantly fewer training samples than direct generation ( $O(\log K)$ vs. $O(\log N)$ ).
Benchmark Creation: Constructed the first large-scale benchmark for this task using 11,803 VSAC value sets, including a rigorous train/validation/test split that holds out entire publisher organizations to test cross-organization generalization.
Empirical Validation: Demonstrated that RASC significantly outperforms zero-shot LLM generation and retrieval-only baselines.
Open Source: Released code for dataset construction, model training, and the benchmark itself to ensure reproducibility.

4. Experimental Results

The evaluation was conducted on a test set of 2,011,009 code pairs (7.5% positive rate).

Model	AUROC	Value-Set F1	Precision	Recall
Retrieval-Only	—	0.136	0.092	0.553
LightGBM	0.745	0.190	0.173	0.316
MLP	0.799	0.250	0.226	0.470
Cross-Encoder	0.852	0.298	0.272	0.483
GPT-4o (Zero-Shot)	—	0.105	0.169	0.100

Performance Gains: The Cross-Encoder achieved the highest F1 (0.298), outperforming the MLP by +0.048 and the retrieval-only baseline by +0.162.
Hallucination: Zero-shot GPT-4o hallucinated 48.6% of its outputs (codes not in the official corpus).
Scalability: As value set size increased, GPT-4o performance degraded monotonically (F1 dropped to 0.014 for sets >150 codes), while RASC classifiers improved, validating the theoretical advantage of the retrieval-constrained approach.
Generalization: The model maintained performance on held-out publishers (Clinical Architecture, CSTE Steward), indicating it learned clinically meaningful structures rather than memorizing authoring patterns.
LLM-as-Classifier: When GPT-4o was provided the candidate pool (acting as a classifier rather than a generator), its F1 jumped from 0.089 to 0.294, proving that LLMs can recognize relevant codes but cannot reliably generate them from memory.

5. Significance and Conclusion

Paradigm Shift: RASC demonstrates that for structured, version-controlled vocabularies, retrieval + classification is superior to direct generation. It transforms an intractable generation problem into a tractable classification problem.
Practical Impact: The system reduces the number of irrelevant candidates per true positive from ~12.3 (retrieval only) to ~3.2 (Cross-Encoder), drastically reducing the manual review burden for clinical informaticists.
Future Directions: The framework is domain-agnostic and applicable to other tasks involving large structured vocabularies and small target sets, such as gene panel construction and systematic review inclusion. The primary bottleneck remains retrieval quality; improving the initial candidate pool is the key to further gains.

In summary, the paper establishes that leveraging existing expert-curated corpora via retrieval, followed by a specialized classifier, is the most effective path toward automating clinical value set authoring, outperforming even state-of-the-art LLMs in accuracy and reliability.