💬 NLP

RASC+: Retrieval-Constrained LLM Adjudication for Clinical Value Set Authoring

This paper introduces RASC+, a stage-wise framework that improves clinical value set authoring by combining enhanced retrieval methods to maximize candidate recall with a constrained LLM adjudicator for precise selection, thereby significantly outperforming previous baselines while ensuring all outputs originate from an auditable candidate pool.

Original authors: Sumit Mukherjee

Published 2026-06-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sumit Mukherjee

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a librarian tasked with creating a "Best Books" list for a specific topic, like "Heart Health." However, you aren't allowed to just make up book titles. You must choose strictly from a massive, ever-changing library catalog containing millions of entries. If you pick a book that doesn't exist in the catalog, the list is useless. If you miss a crucial book that is in the catalog, your list is incomplete.

This is the challenge of Clinical Value Set Authoring. Doctors and health systems need lists of specific medical codes (like "Heart Attack" or "Diabetes") to track patient health. These codes come from huge, strict dictionaries (like SNOMED-CT or ICD-10). The goal is to build a perfect list of codes for a specific medical concept without inventing fake codes or missing the right ones.

The paper introduces a new method called RASC+ to solve this problem. Here is how it works, broken down into simple steps:

The Problem: The "Zero-Shot" Mistake

Previously, researchers tried asking a smart AI (a Large Language Model) to just "write the list" from scratch.

The Analogy: It's like asking a student to write a bibliography from memory without looking at the library.
The Result: The student (the AI) often invents fake book titles or misses real ones because they haven't memorized the entire library. In the medical world, this is dangerous because a fake code is like a fake medicine—it can't be used.

The Solution: A Two-Step "Search and Select" Team

The authors realized it's better to split the job into two distinct roles: The Scout and The Judge.

Step 1: The Scout (Building the Candidate Pool)

The first job is to find every possible book that might belong on the list. The goal here is Recall (finding everything), even if it means grabbing a few books that don't quite fit.

The Old Scout: Used a basic search tool that only looked at the title of the book. It missed many relevant books, only finding about 55% of the correct codes.
The New Scout (RASC+): The authors upgraded the Scout with three superpowers:
1. Smarter Search: It uses a more advanced brain (Qwen3) to understand the meaning of the topic, not just the keywords.
2. Vocabulary Expansion: If the Scout finds a book about "Heart Attack," it automatically checks the library's map to see if there are related books about "Angina" or "Cardiac Arrest" nearby, even if the title didn't match perfectly.
3. Display Rescue: Sometimes the library catalog uses different words for the same thing. The Scout also searches by the actual description of the code, not just the title, to catch things the first search missed.
The Result: The new Scout finds 73% of the correct codes (up from 55%). It creates a much larger "shortlist" of candidates.

Step 2: The Judge (Adjudication)

Now that the Scout has a huge, messy pile of potential candidates, the second job is to pick the exact right ones. The goal here is Precision (being accurate).

The Old Judge: A specialized medical classifier (SAPBert). When given the new, larger pile of candidates, it got confused. It started picking too many wrong books because the pile was noisier. Its score dropped slightly.
The New Judge (GPT-5): The authors used a powerful AI (GPT-5) to act as the Judge.
- The Constraint: Crucially, the Judge is not allowed to invent codes. It can only look at the list the Scout provided and say "Yes, this belongs" or "No, this doesn't."
- The Result: The new Judge is much better at sorting the messy pile. It successfully picked the right codes from the expanded list, achieving a score of 54.9% (compared to the old Judge's 28.7%).

Why This Matters

The paper shows that this "Search then Select" approach is much better than asking an AI to "just write the list."

Safety First: Because the AI (the Judge) can only pick from the list the Scout found, it can never invent a fake medical code. It stays grounded in the real library catalog.
Handling the Unknown: The test included "held-out" publishers (libraries the AI had never seen before). The new system handled these unknown libraries much better than the old methods, proving it can generalize to new situations.

The Bottom Line

The paper demonstrates that by splitting the task into a high-recall Scout (who finds everything possible) and a smart, constrained Judge (who picks the best from that list), we can build much better medical code lists.

Old Way: AI guesses from memory (High error, fake codes).
New Way (RASC+): AI searches a massive net, then carefully picks the best fish from the net.

The result is a system that is safer (no fake codes) and more accurate (finds more real codes), acting as a powerful assistant for human experts rather than a replacement that might hallucinate.

Technical Summary: RASC+ – Retrieval-Constrained LLM Adjudication for Clinical Value Set Authoring

Problem Definition

Clinical value sets define the standardized terminology codes (e.g., SNOMED-CT, ICD-10-CM, RxNorm) required for electronic quality measurement, phenotyping, and clinical decision support. Authoring these sets is not a standard text generation task; it requires selecting a precise subset from a massive, version-controlled terminology universe.

The Retrieval-Augmented Set Completion (RASC) benchmark previously demonstrated that direct zero-shot Large Language Model (LLM) generation is ill-suited for this task. LLMs struggle to memorize the vast, versioned code systems and frequently generate codes absent from the authoritative Value Set Authority Center (VSAC) corpus. The original RASC approach framed the problem as a two-stage process: retrieving similar prior value sets to form a candidate pool, then classifying candidates. However, the original pipeline, relying on SAPBert embeddings and a cross-encoder, achieved a macro F1 of only 0.298 on the full test set, with significant performance drops on out-of-distribution (OOD) publishers.

Methodology

This work proposes RASC+, a stage-wise optimization of the retrieve-then-select formulation. The authors decouple the problem into two distinct objectives:

Stage 1 (Candidate-Pool Construction): Maximize recall (exposing as many true codes as possible) even at the cost of precision, ensuring the "recall ceiling" is high.
Stage 2 (Candidate Adjudication): Maximize precision by selecting the correct subset from the expanded pool, preserving the safety constraint that all outputs must originate from the auditable candidate pool.

Stage 1: Enhanced Pool Construction

The authors replace the original SAPBert title retriever with a more robust pipeline:

Semantic Retrieval: Utilizes Qwen3-Embedding-0.6B to retrieve the top $K=15$ nearest neighbor value sets (increased from $K=10$ ) based on cosine similarity of normalized title embeddings.
Vocabulary-Aware Expansion: To capture codes structurally close to retrieved seeds but absent from the source sets, the system expands the pool using vocabulary-specific graphs:
- ICD-10-CM: Child and sibling links.
- RxNorm & SNOMED-CT: Relation-neighbor links.
- Constraints: Expansion is capped (max 3 new codes per seed, 60 per vocabulary) to maintain pool reviewability.
Code-Display Rescue Retrieval: To address phrasing mismatches between target titles and training corpus titles (a critical failure mode for OOD publishers), the system indexes code display strings using BM25. The top 10 matches per vocabulary are added as seeds, followed by the same graph-based expansion.

The final candidate pool ( $C_v$ ) is the union of source-derived codes, vocabulary-expanded codes, and code-display rescue codes.

Stage 2: Constrained Adjudication

Two selectors are evaluated on the same expanded pool:

SAPBert Cross-Encoder: The original RASC classifier applied to the expanded pool without retraining or threshold retuning. This serves as a conservative control to test if a larger pool alone improves performance.
GPT-5 Adjudication: A constrained LLM that receives the target value set metadata and the candidate pool. It is strictly forbidden from generating new codes; it must return a JSON list of candidate IDs from the supplied pool.
- Blinding: The prompt excludes OIDs, publisher names, and ground truth labels to prevent memorization or leakage.
- Chunking: For pools exceeding 1,200 candidates, the pool is split into deterministic chunks. The model processes each chunk independently, and final predictions are the union of selected IDs.

Key Contributions

Recall Ceiling Elevation: The authors demonstrate that optimizing retrieval and expansion significantly raises the recall ceiling. On the full test set, candidate-pool recall increased from the original RASC baseline of 0.553 to 0.730. For held-out publishers (OOD), recall improved from 0.543 to 0.655.
Adjudication Superiority: The study shows that a constrained LLM (GPT-5) can effectively filter the larger, noisier pool where traditional cross-encoders fail. Replacing SAPBert with GPT-5 on the expanded pool increased the full-test macro F1 from 0.287 to 0.549.
OOD Robustness: The approach significantly improves generalization to unseen publishers. On held-out publishers, GPT-5 achieved a macro F1 of 0.533, compared to 0.233 for SAPBert on the same pool.

Results

The paper reports the following key metrics on the 3,744-value-set test split:

Metric	Original RASC (SAPBert)	Expanded Pool + SAPBert	Expanded Pool + GPT-5
Pool Recall (Full Test)	0.553	0.730	0.730
Macro F1 (Full Test)	0.298	0.287	0.549
Macro F1 (OOD)	0.233*	0.233	0.533
Macro Precision (OOD)	-	0.229	0.642
Macro Recall (OOD)	-	0.415	0.564

*Note: The original RASC OOD F1 is inferred from the text as the baseline for comparison, though the specific OOD split for the original SAPBert was not explicitly broken down in the original manuscript; the paper compares the new SAPBert result on the expanded pool to the original full-test result.

The results indicate that while the expanded pool introduces more false positives (lowering retrieval-only precision), the GPT-5 adjudicator successfully filters these while retaining a high fraction of true positives. The "pool-restricted recall" (fraction of true codes in the pool that are selected) for GPT-5 is 0.757, indicating the model misses some recoverable codes but performs significantly better than the cross-encoder.

Significance and Claims

The paper claims that retrieval-constrained LLM adjudication is a superior paradigm for clinical value set authoring compared to unconstrained generation or static cross-encoders.

Safety and Auditability: By constraining the LLM to select only from a retrieved candidate pool, the system preserves the "corpus-grounded safety" property. It prevents the generation of invalid or hallucinated codes, a critical requirement for clinical applications.
Decomposition of Tasks: The work validates the hypothesis that separating recall (pool construction) and precision (adjudication) allows for specialized optimization. A high-recall pool provides the necessary "ceiling" for performance, which a strong LLM can then navigate more effectively than a traditional classifier.
Decision Support Role: The authors position the system as a decision support tool rather than an autonomous publisher. It reduces the human review burden by presenting a ranked, provenance-rich candidate set, though it acknowledges that human curators must still validate stewardship-specific modeling choices and potential omissions in the source terminology.

The paper concludes that the RASC benchmark serves as a rigorous testbed for these systems and that the combination of high-recall retrieval strategies with constrained LLM selection offers a promising path forward for clinical value set authoring.