💬 NLP

HealthNLP_Retrievers at ArchEHR-QA 2026: Cascaded LLM Pipeline for Grounded Clinical Question Answering

The HealthNLP_Retrievers team achieved competitive results in the ArchEHR-QA 2026 shared task by employing a cascaded pipeline powered by Gemini 2.5 Pro that integrates query reformulation, evidence scoring, grounded response generation, and alignment to deliver precise, evidence-based answers to patient questions derived from electronic health records.

Original authors: Md Biplob Hosen, Md Alomgeer Hussein, Md Akmol Masud, Omar Faruque, Tera L Reynolds, Lujie Karen Chen

Published 2026-04-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Md Biplob Hosen, Md Alomgeer Hussein, Md Akmol Masud, Omar Faruque, Tera L Reynolds, Lujie Karen Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive, complicated medical file (your Electronic Health Record, or EHR) written in a secret code of doctor-speak. Now, imagine a patient trying to ask a question about their health using everyday, emotional, and sometimes messy language. The challenge is to take that messy question, find the exact right sentences in the giant medical file, and write a clear, honest answer that only uses what's actually in the file—no making things up.

This paper describes a team called HealthNLP_Retrievers who built a "digital assembly line" to solve this problem for a competition called ArchEHR-QA 2026. Think of their system not as a single super-brain, but as a relay race with four specialized runners, each passing the baton to the next.

Here is how their system works, step-by-step:

1. The Translator (Question Interpretation)

The Problem: Patients often ask questions like, "I feel so scared and my chest hurts like a heavy rock, and I took that pill my aunt gave me..." This is too wordy and emotional for a computer to search efficiently.
The Solution: The first runner is a "Clinical Administrative Assistant." It listens to the messy patient story and rewrites it into a super-short, professional search query (max 15 words).
The Result: It turns "I'm scared and my chest hurts..." into "Causes of persistent chest pain."
The Win: This team came in 1st place for this step. They were the best at cleaning up the noise without losing the important medical meaning.

2. The Detective (Evidence Scoring)

The Problem: The medical file is huge. You can't read every word. You need to find the specific sentences that answer the question.
The Solution: The second runner acts like a strict judge. It reads every sentence in the file and gives it a score from 1 to 5 (like a Likert scale).

5: This sentence has the direct answer.
4: This sentence gives important context (like a test result).
1-2: This sentence is irrelevant (like "The patient arrived at 9 AM").
The Strategy: The team told the detective to be "safety-first." It's better to grab a few extra sentences that might be useful (high recall) than to miss the one sentence that holds the answer.
The Result: They ranked 7th. They caught almost all the right answers, though they grabbed a few extra "maybe" sentences along the way.

3. The Writer (Grounded Answer Generation)

The Problem: Now you have the right sentences, but you need to write a clear answer for the patient.
The Solution: The third runner is a "Clinical Documentation Specialist." It takes the selected sentences and writes a short, professional answer (max 75 words).
The Rules:

No Hallucinations: It is strictly forbidden to use outside knowledge. If it's not in the selected sentences, it cannot be in the answer.
No Fluff: It must be direct and objective.
The "Soft Cut": If the answer gets too long, the system has a special trick to cut it off at the end of a sentence so it doesn't look broken.
The Result: They ranked 5th. They were excellent at simplifying complex medical jargon into plain English, even if they didn't match the exact wording of the "gold standard" answers perfectly.

4. The Auditor (Answer-Evidence Alignment)

The Problem: How do we prove the answer is true? We need to show exactly which sentence in the file supports each part of the answer.
The Solution: The final runner is a "Conservative Auditor." It looks at the answer and the file, then draws a map connecting every sentence of the answer to the specific evidence in the file.
The Strategy: This runner is very strict. It only connects an answer to evidence if it's 100% sure. It would rather miss a connection than make a weak one.
The Result: They ranked 9th. Their system was very precise (high accuracy) but missed some connections because it was being too cautious.

The Big Picture: What Did They Learn?

The team found that breaking the problem into these four small steps worked better than trying to do everything at once.

The Good: Their "Translator" was the best in the competition. By cleaning up the question first, the rest of the system worked much better.
The Trade-off: They had to choose between being thorough (catching everything) and being precise (only catching the exact right things).
- In the "Detective" phase, they chose to be thorough (catching more info, even if some was extra).
- In the "Auditor" phase, they chose to be precise (only linking what they were sure of).
The Limitation: Because they used a specific, closed AI model (Gemini 2.5 Pro) that lives on a company's server, they can't easily share their exact code for others to run on their own computers. Also, if the first runner makes a mistake, the mistakes get passed down the line to the next runners.

In summary: The HealthNLP_Retrievers team built a team of specialized AI workers to turn messy patient questions into clean, evidence-based medical answers. They proved that having a structured, step-by-step process is a powerful way to make AI trustworthy in healthcare, even if it's not perfect yet.

1. Problem Statement

The paper addresses the challenge of Grounded Clinical Question Answering (QA) over Electronic Health Records (EHRs). While patient portals provide access to EHRs, patients often struggle to interpret complex, technical clinical notes due to a "vocabulary gap" between informal, verbose patient inquiries and dense, specialized medical documentation.

The ArchEHR-QA 2026 shared task requires systems to:

Interpret subjective patient narratives.
Identify minimal supporting evidence within complex clinical notes.
Generate professional, grounded answers strictly restricted to the provided evidence.
Align generated answers with specific source sentences.

The core difficulty lies in bridging the semantic divide between patient language and clinical reality without introducing hallucinations or unjustified medical information.

2. Methodology

The HealthNLP_Retrievers system employs a four-stage cascaded pipeline powered by the Gemini 2.5 Pro large language model (LLM). Unlike standard Retrieval-Augmented Generation (RAG) which often relies on single-pass generation, this approach decomposes the task into structured intermediate steps to improve precision and grounding.

Stage 1: Few-Shot Query Interpretation (Question Reformulation)

Goal: Transform verbose, emotional, and informal patient queries into concise, professional clinical search queries.
Mechanism: A "clinical administrative assistant" persona is used with few-shot prompting (3 examples from the training set).
Constraints: The output is strictly truncated to a maximum of 15 words. It prohibits diagnostic language and preambles, focusing solely on the core clinical intent to reduce noise for downstream retrieval.

Stage 2: Heuristic-Based Evidence Scoring

Goal: Rank clinical sentences to prioritize recall while filtering noise.
Mechanism: The model acts as a "clinical evidence scorer," assigning a 1–5 Likert scale to every sentence in the EHR note:
- 5 (Critical): Direct answers, confirmed diagnoses, specific medication outcomes.
- 4 (High Context): Qualifying findings (e.g., test result context).
- 3 (Relevant): Background, admission dates, history.
- 1–2 (Irrelevant): Normal findings, headers, unrelated systems.
Dynamic Fallback: To prevent "evidence starvation" (false negatives), the system uses a dynamic filter:
- First, it isolates strict evidence (scores $\ge$ 4).
- If no strict evidence is found, it falls back to a lenient set (scores $\ge$ 3).
- In case of API failure, it defaults to the first three sentences of the note.

Stage 3: Grounded Response Generation

Goal: Synthesize a professional answer strictly based on the filtered evidence.
Mechanism: A "clinical documentation specialist" persona is used. The prompt includes three context blocks: (1) the interpreted query, (2) the filtered evidence pool, and (3) the full clinical note as a safety net for implicit references.
Constraints: The output is capped at 75 words.
Truncation Heuristic: A "soft-cut" logic is applied. If the text exceeds 75 words, it attempts to cut at the end of a sentence. If the resulting length is at least 60% of the limit, it is accepted; otherwise, a hard cut is enforced with a manually appended period to ensure grammatical integrity.

Stage 4: Answer-Evidence Alignment

Goal: Map generated answer sentences to their specific source sentences in the EHR.
Mechanism: A many-to-many alignment framework. The model is prompted to link evidence conservatively, citing only sentences that directly support the answer.
Output: A JSON array mapping answer sentence indices to evidence sentence indices.

3. Key Contributions

Cascaded Pipeline Architecture: Moving beyond standard RAG, the system decomposes the complex reasoning task into four distinct stages (Interpretation, Scoring, Generation, Alignment), allowing for specialized optimization at each step.
Persona-Driven Query Reformulation: The use of a specific "clinical administrative assistant" persona with strict 15-word constraints effectively bridges the vocabulary gap, significantly improving downstream retrieval precision.
Recall-Biased Scoring with Dynamic Fallback: The 1–5 Likert scoring system, combined with a dynamic fallback mechanism, prioritizes preventing false negatives (missing critical evidence) over strict precision, ensuring the generator has sufficient context.
Precision-Constrained Alignment: A strict many-to-many alignment framework that acts as a verification layer, ensuring generated answers are explicitly grounded in the provided text to mitigate hallucinations.

4. Results

The system achieved competitive results across all four subtasks of the ArchEHR-QA 2026 challenge:

Subtask	Rank	Primary Metric (Score)	Key Insight
Question Interpretation	1st	Overall: 31.2	The few-shot reformulation strategy successfully stripped noise while retaining clinical intent (BERTScore: 46.8).
Evidence Identification	7th	Strict Micro F1: 60.2	Designed for high recall; strict precision was lower (58.4) but lenient precision jumped to 75.7, confirming the "safety-first" retrieval strategy.
Answer Generation	5th	Overall: 34.6	Achieved a top-tier SARI score (59.2), outperforming 1st place in text simplification, proving the effectiveness of the "soft-cut" and persona constraints.
Answer-Evidence Alignment	9th	Micro F1: 76.9	High precision (83.8) indicates the model successfully filtered out irrelevant noise, though recall (71.1) was lower due to strict exclusion of implicit support.

Ablation Study Findings:

Few-Shot vs. Zero-Shot: Few-shot prompting significantly improved MEDCON scores (medical entity retention) from 9.20 to 17.20.
Query Anchors: Using the reformulated clinician query (15 words) for retrieval yielded a higher F1 score (62.90) compared to using raw patient narratives (60.20), confirming that concise queries filter noise better.

5. Significance and Limitations

Significance:
The paper demonstrates that integrating Large Language Models within a structured, multi-stage pipeline significantly improves the grounding, precision, and professional quality of patient-oriented health communication. The approach effectively balances the trade-off between accessibility (simplifying language) and medical accuracy (retaining specific entities).

Limitations:

Proprietary Dependency: The system relies entirely on the Gemini 2.5 Pro API, limiting reproducibility and adaptability in privacy-sensitive environments where data cannot be sent to external servers.
Error Propagation: The cascaded nature means errors in early stages (e.g., imperfect query reformulation) can degrade performance in later stages.
Generalizability: Evaluation is confined to the ArchEHR-QA 2026 benchmark; performance on other EHR systems is unexplored.
Trade-offs: The system prioritizes high precision in alignment and generation, which sometimes comes at the cost of recall (missing implicit evidence) and specific medical entity retention in the summary phase.

Future Work:
The authors plan to explore iterative self-correction to reduce error propagation and distill the pipeline into open-weight, locally deployable models for privacy-preserving clinical use.