💬 NLP

Healthier LLMs: Retrieval-Augmented Generation for Public Health Question Answering

This paper extends the PubHealthBench benchmark to a retrieval-augmented setting, demonstrating that hybrid retrieval and careful context selection significantly improve the accuracy and reliability of public health question answering by enabling smaller models to outperform larger ones while introducing a validated LLM-as-a-judge framework for evaluating free-form responses.

Original authors: Felix Feldman, Joshua Harris, Timothy Laurence, Leo Loman, Ollie Higgins, Fan Grayson, Poonam Soma, Bethany Pace-Bonello, Michael Borowitz, Toby Nonnenmacher

Published 2026-07-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Felix Feldman, Joshua Harris, Timothy Laurence, Leo Loman, Ollie Higgins, Fan Grayson, Poonam Soma, Bethany Pace-Bonello, Michael Borowitz, Toby Nonnenmacher

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant but slightly forgetful librarian named "LLM." This librarian has read millions of books and can answer almost any question instantly. However, there are two big problems:

The "Hallucination" Problem: Sometimes, when the librarian doesn't know the answer, they confidently make one up.
The "Outdated" Problem: The librarian's memory is frozen in time. If a new health rule comes out today, the librarian won't know about it until they are retrained years from now.

This paper is about giving this librarian a magic index card system (called Retrieval-Augmented Generation, or RAG) so they can look up the exact current rules before answering a question. The authors tested this system using a massive library of UK Government public health guides.

Here is the breakdown of their findings, using simple analogies:

1. The Search Strategy: How to Find the Right Page

The researchers tested different ways for the librarian to find the right page in the library.

The "Keyword" Search (Sparse): Like looking for a book by searching for specific words (e.g., "flu," "mask"). It's good, but sometimes misses the point if you use different words.
The "Semantic" Search (Dense): Like asking the librarian, "I need info about protecting yourself from viruses," and them understanding the meaning behind the words. This is usually better.
The "Hybrid" Search: This was the winner. It's like having a super-smart librarian who checks both the specific keywords and the meaning of your question at the same time.
- The Result: Mixing these two methods consistently found the right information better than using just one. It was so effective that a smaller, cheaper librarian (a smaller AI model) using this hybrid search could outperform a giant, expensive librarian who had to guess without looking anything up.

2. The "Chunk" Size: How Big Should the Page Be?

The library's books were cut into smaller pieces (chunks) to make them easier to search. The researchers found that size matters.

Too Small: You might miss the context.
Too Big: If a chunk is a whole chapter, it's like trying to find a needle in a haystack. The librarian gets confused by too much extra text.
The Sweet Spot: They found that medium-sized chunks worked best. Interestingly, they tried summarizing the long chunks into short notes to help the search. While this helped a little, it didn't fix the problem entirely. The best approach was to keep the chunks at a manageable size and use the "Hybrid" search method.

3. The Multiple-Choice Test vs. The Real Conversation

The researchers tested the librarians in two ways:

Multiple Choice (MCQA): Like a quiz where the librarian picks from options A, B, C, or D.
- Result: When the librarian could look up the answer first, even the small, cheap models got nearly perfect scores (around 99%). They beat the giant models that weren't allowed to look anything up.
Free-Form Answers: Like a real conversation where the librarian has to write a full paragraph.
- Result: This was harder. While the librarians could find the right info, they sometimes got "chatty." They would include extra advice that wasn't strictly necessary or mix in details from other parts of the book.
- The "Faithfulness" Issue: The biggest problem wasn't finding the wrong info; it was the librarian adding too much info. If the right answer was on page 50, but the librarian also read pages 1–49, they might accidentally mix in advice from page 10 that didn't apply to the specific question. The further down the list the right answer was, the more likely the librarian was to get "confused" and add extra, potentially misleading details.

4. The "Judge" Problem: Who Grades the Answers?

To see if the librarians were doing a good job, the researchers used a "Judge" (another AI) to grade the answers. They also had human experts grade the same answers to check if the AI Judge was fair.

What the Judge Got Right: The AI Judge was very good at spotting if the librarian stuck to the script (Faithfulness) and if they covered all the main points (Completeness).
What the Judge Struggled With: The AI Judge was not very good at spotting if the librarian was being clear (Clarity) or if the facts were perfectly accurate (Factual Consistency). Even the human experts disagreed on these points sometimes.
The Lesson: You can trust the AI Judge to tell you if the librarian stayed on topic, but you can't fully trust it to tell you if the answer is perfectly clear or factually airtight.

The Bottom Line

The paper concludes that for public health questions, how you search for the information is more important than how big the AI model is.

Small + Smart Search > Big + No Search: A small, affordable AI model with a great "Hybrid" search system is safer and more accurate than a giant, expensive AI model that relies only on its memory.
Keep it Focused: To get the best results, you need to feed the AI just the right amount of information (not too little, not too much) and make sure the most relevant info is at the very top of the list.
Safety First: By grounding the AI in official, up-to-date government documents, we can prevent it from making things up or giving outdated advice, which is crucial for public health.

In short: Don't just make the librarian smarter; give them a better, more organized index card system.

Technical Summary: Healthier LLMs: Retrieval-Augmented Generation for Public Health Question Answering

Problem Statement

Large Language Models (LLMs) demonstrate promise in medical question answering but face critical limitations in public health applications: hallucinations and the inability to keep pace with rapidly evolving official guidance. Unlike clinical decision support which often targets individual patients, public health guidance is population-level, precautionary, and frequently updated. Relying solely on parametric memory risks disseminating outdated or inaccurate advice, which can lead to harmful individual decision-making or dangerous population-level behaviors. While Retrieval-Augmented Generation (RAG) offers a solution by grounding responses in an external corpus, its end-to-end performance in public health depends heavily on specific retrieval configurations and evaluation methods that go beyond standard multiple-choice formats.

Methodology

The authors extend PubHealthBench, a benchmark of 7,929 multiple-choice questions (MCQA) and 760 free-form questions derived from UK Government public health guidance, into a retrieval-augmented setting.

1. Dataset and Corpus

Source: 687 UK Government public health guidance documents covering 10 topic areas.
Corpus Construction: Documents were converted to Markdown and split into chunks based on header hierarchy, resulting in 5,358 chunks.
Corpus Variants:
- CF (Full Corpus): Original chunks.
- CS (Summary-only): Chunks summarized by GPT-4o-mini for retrieval.
- CR (Reduced Corpus): Chunks >512 tokens replaced with summaries to fit embedding model context limits.

2. Retrieval Strategies

The study systematically evaluated five retrieval configurations across multiple embedding models (e.g., NV-Embed-v2, Multilingual-E5, ModernBERT) and sparse indices (BM25, TF-IDF):

Dense-only: Cosine similarity using dense embeddings.
Sparse-only: Keyword overlap (BM25/TF-IDF).
Hybrid: Weighted Reciprocal Rank Fusion (RRF) of dense and sparse results.
Summary-based Hybrid: Hybrid retrieval performed on summaries, with full chunks passed to the generator.
Reduced-corpus Hybrid: Hybrid retrieval on a corpus where long chunks are summarized.

3. Generation and Evaluation

Models: 11 LLMs ranging from 1B to 70B parameters (e.g., Llama 3.3, Phi-4, Gemma-3, MedGemma).
MCQA Evaluation: Measured accuracy on PubHealthBench-Full with varying context chunk counts ( $k \in \{1, 3, 5, 10\}$ ).
Free-form Evaluation: Introduced a rubric-based LLM-as-a-Judge (GPT-OSS-120B) to evaluate responses on four criteria: Faithfulness (no new claims), Completeness (all key points), Factual Consistency (accuracy relative to guidance), and Clarity.
Validation: The LLM judge was validated against dual human expert annotations on a sample of 100 responses using Cohen's $\kappa$ and macro-F1.

Key Results

Retrieval Performance

Hybrid Superiority: Hybrid retrieval consistently outperformed dense-only and sparse-only methods across all metrics (Recall@k, MRR, nDCG@10).
Model Sensitivity: While NV-Embed-v2 was the strongest dense model, hybrid retrieval significantly narrowed the performance gap between large and small embedding models (e.g., improving smaller encoders by 5–10 percentage points).
Chunk Length Impact: Ranking quality (Precision@1, MRR) degraded sharply as target chunk length exceeded 700–800 words. Summary-based retrieval partially mitigated this but did not eliminate it.
Topic Variance: While recall was largely independent of topic, ranking quality showed systematic deviations across topics (e.g., Tuberculosis vs. Blood Safety), suggesting topic-specific language or structure affects embedding performance.

Generation Performance (MCQA)

Retrieval Impact: Providing retrieved context substantially increased MCQA accuracy. Smaller open-weight models (e.g., Phi-4-14B, Gemma-3-12B) with high-quality retrieval matched or exceeded the performance of larger models (e.g., GPT-4.5) operating without retrieval.
Optimal Context: Maximum accuracy was typically achieved with $k=3$ to $k=5$ chunks. Increasing $k$ to 10 introduced noise with diminishing returns.
Rank Sensitivity: Retrieving the correct chunk at rank 1 yielded the largest gains. Smaller models were more sensitive to rank position than larger models, which could extract relevant info from noisier contexts.

Free-Form Evaluation & Judge Validation

Judge Reliability: The LLM judge showed strong agreement with humans on Faithfulness ( $\kappa \approx 0.64$ ) and Completeness ( $\kappa \approx 0.59$ ). Agreement was weak for Factual Consistency ( $\kappa \approx 0.06$ ) and Clarity, indicating these dimensions are difficult to automate reliably.
Model Behavior: Medium and large models achieved high completeness and clarity but struggled with Faithfulness; even top models only met the faithfulness criterion in ~64% of cases, often incorporating extraneous guidance.
Rank Effect on Faithfulness: Faithfulness dropped sharply as the target chunk rank increased, confirming that models rely heavily on early context and hallucinate when the most relevant evidence is buried.

Significance and Claims

The paper claims that retrieval configuration is a primary lever for reliable public health QA, often more impactful than model size alone.

RAG as a Stabilizer: Grounding LLMs in an explicitly maintained corpus of official guidance mitigates hallucinations and outdated advice, enabling systems to remain aligned with evolving public health policy without model retraining.
Efficiency via RAG: A well-designed retrieval stack allows smaller, open-weight models to outperform larger, closed models that rely solely on parametric memory. This offers a practical pathway for public health institutions with budget and infrastructure constraints.
Evaluation Nuance: The study highlights that standard MCQA benchmarks may overestimate capability due to answer-option cues. Free-form evaluation is necessary but requires careful rubric design; currently, automated evaluation of "factual consistency" remains unreliable, necessitating human oversight or further method development.
Practical Guidance: For deployment, the authors recommend hybrid retrieval, careful chunking (avoiding excessive length), and limiting context to the top 3–5 chunks for optimal performance-to-cost ratios.

The work concludes that while RAG significantly improves reliability, future benchmarks must address multi-evidence support and out-of-corpus queries to fully test system robustness in real-world public health scenarios.