💬 NLP

Retrieval-Augmented Large Language Models for Schema-Constrained Clinical Information Extraction

This paper proposes a modular retrieval-augmented generation pipeline utilizing GPT-5.2 and schema-constrained prompting to extract and normalize clinical observations from nurse-patient transcripts into a structured format, achieving an 80.36% F1 score while demonstrating that retrieval augmentation and second-pass auditing significantly enhance schema adherence.

Original authors: A H M Rezaul Karim, Ozlem Uzuner

Published 2026-05-18

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: A H M Rezaul Karim, Ozlem Uzuner

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a busy nurse spending their day talking to patients. These conversations are full of important medical details, but right now, a nurse has to manually type all that information into a computer system later. This is like trying to transcribe a whole movie script by hand while the movie is still playing—it's slow, tiring, and takes time away from actually helping the patient.

The MEDIQA-SYNUR task is a challenge to build a robot (an AI) that can listen to these nurse-patient conversations and automatically write down the key facts in a very specific, organized format. But there's a catch: the computer system the AI is writing to is extremely picky. It's like a strict librarian who will only accept books if they are placed on the exact right shelf, with the exact right label, and no extra notes allowed.

Here is how the researchers at George Mason University (MasonNLP) built their robot to solve this problem:

1. The "Cheat Sheet" Strategy (Retrieval-Augmented Generation)

Instead of just asking the AI to "guess" the answer from its general memory, the researchers gave it a Cheat Sheet.

The Analogy: Imagine taking a difficult math test. Instead of relying only on what you studied, you are allowed to look at a few examples of similar problems and their solutions right next to you.
How it works: The AI looks at the current conversation, finds similar past conversations from its training data, and uses those examples to figure out how to format the new information correctly. The paper found that giving the AI these "examples" (Retrieval-Augmented Generation, or RAG) consistently made it smarter and more accurate.

2. The "Menu" Problem (Schema Constraints)

The AI has to choose from a massive menu of 193 different medical categories (like "Pain Level," "Mobility," or "Nausea").

The Analogy: Imagine asking a waiter, "What do you have?" If you give them a menu with 193 items, a small, inexperienced waiter might get overwhelmed and order the wrong thing. But a very experienced, high-end waiter might actually prefer seeing the entire menu to make the perfect choice.
The Discovery: The researchers tested two types of "waiters" (AI models):
- The Smaller Model (Llama-4): This model got confused by the full menu. It worked much better when the researchers gave it a Pruned Menu—a shortened list of only the most likely options. This helped it focus and avoid mistakes.
- The Larger Model (GPT-5.2): This "super-smart" model actually performed worse with the shortened menu. It needed the Full Menu to understand the nuances and make the best decisions.
- The Lesson: You can't use the same "menu" for every AI. You have to tailor the rules to the specific brain you are using.

3. The "Second Opinion" (Second-Pass Auditing)

Even the best AI makes small mistakes, like writing "3" instead of "03" or picking a value that isn't on the approved list.

The Analogy: Think of this as a teacher grading a test. The first pass is the student taking the test. The second pass is the teacher reviewing the answers, crossing out the ones that don't follow the rules, and fixing the formatting.
The Result: This "Second-Pass Audit" didn't completely rewrite the answers, but it cleaned up the small errors, giving the system a slight but helpful boost in accuracy.

The Final Score

By combining these three ingredients—using examples (RAG), giving the right size menu (Schema), and adding a final check (Audit)—the team built a system that got an 80.36% score on the challenge.

In summary:
The paper proves that to turn messy nurse conversations into clean, structured data, you can't just rely on a smart AI. You have to:

Show it similar examples first.
Give it a list of options that matches its intelligence level (short list for smaller brains, full list for big brains).
Have a second set of eyes check the work for formatting errors.

This approach helps reduce the "documentation burden" (the paperwork headache) for nurses by automating the translation of their speech into the strict computer language hospitals need.

Technical Summary: MasonNLP at MEDIQA-SYNUR 2026

Problem Definition

The paper addresses the challenge of converting conversational nurse-patient transcripts into structured, schema-adherent clinical observations. While clinical documentation is vital for care continuity, it imposes a significant burden on clinicians, who spend substantial time on electronic health record (EHR) tasks rather than direct patient care. The specific task, MEDIQA-SYNUR, requires systems to extract clinically salient observations from transcripts and normalize them into a predefined schema of 193 observation concepts. This schema includes strict value-type constraints (numeric, string, single-select, multi-select) and enumerated validity requirements for categorical values. The difficulty lies in the large, heterogeneous output space and the need for the model to avoid hallucinating concept names or violating type constraints, which are common failure modes in large language models (LLMs) under strict schema conditions.

Methodology

The authors propose a modular, retrieval-augmented generation (RAG) pipeline designed to condition LLM outputs on both retrieved exemplars and schema constraints. The system comprises the following components:

Retrieval Corpus and Knowledge Base:
- The training set serves as the sole retrieval corpus.
- Each training instance is represented by two textual views: the original transcript and a "concept-name summary" derived by mapping gold-label concept IDs to their schema names.
- Embeddings are generated using Sentence-Transformers all-MiniLM-L6-v2 for the Llama-4 backbone and OpenAI text-embedding-3-large for the GPT-5.2 backbone.
Exemplar Retrieval (RAG):
- A candidate pool is retrieved via cosine similarity on transcript embeddings.
- The pool is re-ranked using a weighted score combining transcript similarity ( $w_t=0.70$ ), concept-name summary similarity ( $w_c=0.25$ ), and lexical overlap via Jaccard similarity ( $w_l=0.05$ ).
- Top- $k$ exemplars (varying $k \in \{3, 5, 10, 20, 30, 50\}$ ) are inserted into the prompt as few-shot demonstrations, pairing the exemplar transcript with its gold-label JSON output.
Schema Constraints:
The study compares two prompting strategies for conditioning the LLM on the schema $S$ :
- Full Schema Prompting: The model receives the complete list of 193 concepts, including identifiers, names, value types, and allowable categorical values.
- Pruned Candidate Schema Prompting: A per-instance candidate set $C(x)$ is constructed by scoring concepts based on lexical matches with the transcript, semantic similarity to concept names, and retrieval expansion (adding concepts found in retrieved exemplars). This limits the output space to a compact subset of relevant concepts.
Inference and Postprocessing:
- The LLM is instructed to output only a JSON list of (id, v) pairs.
- Deterministic Expansion: To prevent hallucinations of names or types, the system post-processes the output by looking up the canonical name and value_type for each predicted id from the schema.
- Schema Validation: Predictions are validated against the schema; unknown IDs are dropped, and values are normalized (e.g., converting numeric strings to floats, enforcing enumerated lists for categorical types).
Second-Pass Auditing:
A refinement stage where an auditor LLM reviews the original transcript, the schema, and the first-pass prediction. It is instructed to remove unsupported items, correct schema adherence issues, and add only clearly supported missing observations.

The pipeline was evaluated using two LLM backbones: Llama-4-Scout-17B-16E-Instruct (open-weight) and GPT-5.2 (closed-weight).

Key Results

The experiments, summarized in Table 2 of the paper, yield the following performance metrics (F1 scores):

Best Configuration: The highest performance was achieved using GPT-5.2 with Full Schema prompting, RAG (with $k=30$ exemplars), and Second-Pass Auditing, resulting in an F1 score of 80.36%.
Impact of RAG: Retrieval augmentation consistently improved performance across both models and schema configurations by reducing false negatives and reinforcing the expected JSON structure and value normalization.
Schema Interaction:
- Llama-4 (Open-weight): Performed best with the Pruned Candidate Schema (F1 73.20% with RAG). Reducing the output space helped the smaller model adhere to constraints and focus on relevant concepts.
- GPT-5.2 (Closed-weight): Performed best with the Full Schema (F1 80.27% with RAG). The larger model benefited from the full context to disambiguate related concepts, whereas pruning led to a drop in precision (a "checklist-like bias" encouraging over-extraction).
Second-Pass Auditing: Provided modest additional gains (e.g., +0.09 F1 for GPT-5.2), primarily by correcting residual schema-adherence and normalization errors rather than significantly altering extraction coverage.

Error Analysis

The error analysis (Table 3 and 4) reveals that most errors are not near-miss value disagreements but rather:

False Positives (608 instances): 81.9% were spurious concept predictions (IDs not in the ground truth). Common over-predicted concepts included "Secondary diagnosis" and "Patient identification," which were clinically plausible but rarely annotated in the gold standard.
False Negatives (485 instances): 78.8% were missed concepts entirely.
Systematic Issues: A significant portion of errors stemmed from representation-level mismatches, such as leading zeros in concept IDs (e.g., "03" vs. "3") and encoding differences in categorical tokens (e.g., temperature units). These issues highlight the brittleness of strict string matching in heterogeneous schemas.

Significance and Contributions

The paper claims three primary contributions to the field of Clinical Information Extraction (IE):

Model-Aware Schema Constraint: The study demonstrates that the optimal strategy for constraining LLM outputs is not universal but depends on the model's scale and architecture. Pruned candidate schemas are beneficial for smaller open-weight models to reduce the decision space, whereas larger closed-weight models benefit from full schema context to leverage their reasoning capabilities.
RAG and Schema Interaction: The work provides systematic evidence that Retrieval-Augmented Generation consistently improves schema-constrained extraction, but its efficacy is modulated by how the schema is presented to the model.
Refinement vs. Core Component: The authors show that second-pass auditing yields modest improvements by fixing residual normalization errors but does not substitute for the core mechanisms of RAG and schema constraints. It is best utilized as a final refinement stage.

Overall, the findings suggest that effective schema-constrained observation extraction requires a strategy that jointly selects retrieval and schema-constraint approaches tailored to the specific capabilities of the underlying LLM.