💬 NLP

Evaluating Factual Density in Multi-Source RAG: A Study in Medical AI Accuracy

This paper introduces Factual Density (FD*), a novel retrieval signal that optimizes Medical RAG systems by prioritizing documents with a high proportion of verified atomic claims over lexically similar text, thereby achieving superior evidence saturation and factual accuracy compared to traditional keyword-based methods.

Original authors: Michael R. DeMarco

Published 2026-06-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Michael R. DeMarco

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Noisy Library"

Imagine you are looking for a specific, life-saving fact in a massive library. You ask the librarian (the AI) a question like, "Does exercise lower heart disease risk?"

The librarian has two ways to find books:

The Keyword Match: They look for books that use the exact same words as your question.
The Vibe Match: They look for books that "sound" like your question when read by a computer brain.

The Paper's Discovery: Both of these methods have a blind spot. They love long, chatty books that repeat the same words over and over. They often ignore short, dense, scientific abstracts that are packed with hard facts, simply because the short ones don't have as many words to match your query.

The authors call this the "Expert Blindness Effect." It's like the librarian ignoring a 200-word summary written by a Nobel Prize-winning scientist because it's too short, while handing you a 2,000-word blog post that just repeats the word "exercise" a hundred times without giving any real data.

The Solution: A "Fact Density" Score

To fix this, the researchers invented a new way to rank books called Factual Density (FD)*.

Think of a document not as a pile of words, but as a smoothie.

Standard AI looks at the size of the cup (how long the text is).
FD* looks at how many real fruit chunks (verified facts) are inside that cup.

If you have a huge cup of water with one tiny grape, it's a "low-density" smoothie. If you have a tiny cup packed with 15 different fruits, it's a "high-density" smoothie. The paper argues that for medical questions, you want the tiny, fruit-packed cup, not the big, watery one.

How They Tested It

The researchers built a "Ghost Audit" pipeline. Before the AI ever sees a document, they run it through a special scanner that:

Extracts the facts: It pulls out every single specific, verifiable claim (e.g., "A 2021 study found 45% efficacy").
Scores them: It gives points based on how specific and quantifiable the claim is.
Calculates the Density: It divides the total "fact points" by the number of words.

The "Length Trap" Fix:
At first, they noticed a glitch: Short documents always scored higher just because they had fewer words (the denominator was small). It was like saying a cup with 1 grape is better than a cup with 10 grapes just because the cup is smaller.
To fix this, they used a statistical trick called Z-score normalization. Imagine sorting all the books into "Small," "Medium," and "Large" bins. They only compared the density of books within their own bin. This ensured they were comparing apples to apples, not apples to watermelons.

The Results: Finding the Gold

They tested this new system against the old "Vibe Match" system using a benchmark of 750 health claims verified by real medical experts.

The Old System: When asked about exercise and heart health, the old system pulled up a mix of long articles and some scientific papers. It missed the most authoritative source (a Cochrane Systematic Review) because that review was ranked 8th or 12th.
The New System (FD):* The new system immediately found that Cochrane Review and put it at the very top. In fact, it was the only system to fill the top 5 results with high-quality, expert-verified systematic reviews.

The paper found that the new system successfully surfaced the "fruit-packed" evidence that the old system buried.

Important Caveats (What the Paper Didn't Say)

The authors are very careful about what they claim:

It's a "Reranker," not a replacement: They didn't throw out the old AI. They just added a "second opinion" step. The AI still finds the relevant topics, but then the FD* score bumps the fact-heavy ones to the front of the line.
The "Alignment" Problem: The researchers admitted a major hurdle. They tried to test this on 50 different questions, but their library of documents didn't actually contain answers for enough of those questions. They only had enough data to prove the point on 7 specific questions.
Not a Magic Cure: They explicitly state that while the results look promising, they haven't done the massive statistical test (on 50 questions) yet to prove it works everywhere. They also noted that the "weights" they gave to different types of sources (like Cochrane vs. general PubMed) were guesses, not proven numbers.

The Bottom Line

This paper argues that in medical AI, quality of information matters more than quantity of words.

They propose a simple, low-cost upgrade: Before an AI answers a health question, it should check how many verified facts are packed into the source text. If it does this, it stops handing you long, empty articles and starts handing you the short, dense, expert summaries that actually contain the truth.

They call this "Factual Density," and they say it fixes the "Expert Blindness" where AI ignores the smartest sources because they are too concise.

Technical Summary: Evaluating Factual Density in Multi-Source RAG

Problem Statement

Retrieval-Augmented Generation (RAG) has become the industry standard for grounding Large Language Models (LLMs) in external knowledge. However, standard retrieval mechanisms—specifically Cosine Similarity (semantic proximity) and BM25 (lexical frequency)—lack an inherent mechanism to measure the factual quality of retrieved content.

The paper identifies a structural failure mode termed the Expert Blindness Effect. This phenomenon occurs when standard RAG pipelines systematically bury high-density, expert-validated evidence (e.g., Cochrane Systematic Reviews containing specific, quantified clinical outcomes) in favor of lexically dominant but factually sparse text (e.g., long-form articles that repeatedly reference a topic without providing specific data). Current evaluation frameworks focus on faithfulness (whether the model hallucinated relative to retrieved context) rather than retrieval quality (whether the retrieved context was factually dense or expert-validated in the first place).

Methodology

1. Corpus Construction and Preprocessing

The study utilized a 600-chunk corpus derived from three tiers of medical evidence:

Cochrane Systematic Reviews (COCHRANE_SR): 200 structured abstracts.
Randomized Controlled Trials (CLINICAL_RCT): 200 abstracts filtered by NLM tags.
General PubMed Abstracts (SCIENTIFIC_PMC): 200 peer-reviewed abstracts.

Prior to ingestion, the corpus underwent the NexusAgentics Ghost Audit preprocessing pipeline. This pipeline performs:

Atomic Claim Extraction: Using Gemini 2.5 Flash to identify unique, standalone, verifiable factual statements while discarding hedged language and anecdotes.
Probabilistic Factuality Scoring: Assigning a prob_fact score (0.2 to 1.0) based on verifiability and specificity, not empirical truth.

2. Factual Density (FD*) Formulation

The core metric, Factual Density (FD)*, is defined as the proportion of verified atomic claims relative to token count.

Initial Formulation ( $FD_{raw}$ ): Sum of prob_fact scores divided by token count.
- Issue: Experiment 1 revealed a severe document-length confound (Pearson $R = -0.8636$ ), where shorter documents artificially scored higher due to the denominator effect.
Corrected Formulation ( $FD^*$ ): To resolve this, the authors applied Z-score normalization within document length bins (Short, Medium, Long).
- $FD^*(d) = [(FD_{raw}(d) - \mu_{bin}) / \sigma_{bin}] \times a_{source}$
- This correction reduced the length correlation to non-significant levels ( $p = 0.0749$ ), validating FD* as a length-independent signal.

3. Retrieval Conditions

Three conditions were evaluated against the HealthFC benchmark (750 expert-labeled health claims):

Condition A: Cosine Similarity (Semantic search).
Condition B: BM25 (Lexical search).
Condition C: FD-Optimized Retrieval.* A two-stage hybrid pipeline:
1. Cosine Similarity pre-filter (Top-20).
2. Reranking using a composite score: $0.5 \times \text{CosineSim} + 0.5 \times FD^*_{rescaled}$ .

4. Evaluation Constraints

The study faced significant corpus-benchmark alignment constraints. Despite two corpus iterations, coverage against the HealthFC evaluation set remained below 20% (14% in the final iteration). Consequently, the planned statistical validation (Wilcoxon signed-rank test on $n=50$ queries) could not be executed. Results are reported as directional evidence based on available ground truth mappings.

Key Results

1. Validation of Scoring and Correction

Ghost Audit: Validated that the scoring framework produces meaningful variance across source tiers (Standard Deviation $\ge 0.10$ ), ensuring the reranking signal is not collapsed.
Length Confound: Confirmed that naive token-normalized density metrics are vulnerable to document length bias. Z-score normalization within bins was proven to be an effective, computationally inexpensive correction.

2. The Expert Blindness Effect (Single-Query Audit)

On a specific query regarding exercise and cardiovascular risk, the retrieval conditions yielded distinct results:

Cosine Similarity: Retrieved 40% Cochrane reviews and 60% RCTs.
BM25: Retrieved 80% Cochrane reviews but failed to rank them consistently high on all metrics.
FD-Optimized:* Achieved 100% systematic review saturation in the top-5 results.
- Mechanism: FD* successfully elevated high-density evidence that was ranked outside the top ten by semantic similarity alone (e.g., a Cochrane abstract with a cosine score of 0.7308 was elevated to Rank 4 due to a high FD* score of 9.660).
Ground Truth Verification: The top-5 results from the FD* condition mapped to nine confirmed instances across five distinct HealthFC-supported claims, all labeled "Supported" by medical experts.

3. Corpus Alignment Findings

The study identified that seeded pre-check samples systematically overestimate corpus coverage. A sample of 12 chunks showed 38.62% alignment, while the full 600-chunk corpus showed only 14%. This highlights a critical gap in current RAG evaluation practices where source selection often assumes alignment without full-corpora verification.

Significance and Contributions

The paper posits two co-equal contributions:

Factual Density (FD) as a Retrieval Signal:*
FD* is proposed as a source-agnostic, length-independent, and computationally inexpensive optimization signal. It can be integrated into existing RAG pipelines as a 50/50 composite with cosine similarity without requiring changes to embedding models, vector stores, or retrieval infrastructure. It addresses the gap where semantic proximity fails to correlate with informational concentration.
Documentation of the Expert Blindness Effect:
The paper formally names and empirically demonstrates the systematic bias in standard retrieval methods that buries high-authority evidence. This has direct safety implications for health AI, where surfacing a 2,000-word article over a Cochrane review can lead to dangerous misinformation.
Methodological Contribution to RAG Evaluation:
The study introduces a dual-filter pre-check methodology (requiring cosine similarity $\ge 0.75$ and keyword overlap $\ge 2$ ) as a prerequisite for health RAG corpus construction. It argues that corpus-benchmark alignment is a critical, often overlooked prerequisite for valid RAG evaluation, and that relying on small, topically seeded samples leads to false confidence in coverage.

Limitations and Future Work

The authors explicitly state that findings are preliminary due to the inability to execute the full $n=50$ statistical test caused by low corpus-benchmark alignment. The "Expert Blindness Effect" is demonstrated on a single query and not statistically generalized.

Future work is outlined to:

Execute formal statistical testing using a corpus with $\ge 40\%$ HealthFC coverage.
Empirically derive optimal source authority weights ( $a_{source}$ ) rather than using placeholders.
Explore FD* as an anchor node signal in Graph RAG architectures to compound evidence structure with factual density.

The paper concludes that while FD* is not a complete solution, it represents a necessary step toward ensuring AI systems retrieve facts rather than just lexical matches, particularly in high-stakes domains like healthcare.