💬 NLP

MedFabric and EtHER: A Data-Centric Framework for Word-Level Fabrication Generation and Detection in Medical LLMs

This paper introduces MedFabric, a data-centric pipeline for generating realistic word-level medical fabrications, and ETHER, a modular detector that leverages this dataset to significantly outperform existing state-of-the-art methods in identifying subtle factual deviations within medical LLM outputs.

Original authors: Tung Sum Thomas Kwok, Qian Qian, Xiaofeng Lin, Dongxu Zhang, Jun Han, Zhichao Yang, Davin Hill, Tamer Soliman, Sanjit Singh Batra, Robert Tillman, Guang Cheng

Published 2026-05-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Tung Sum Thomas Kwok, Qian Qian, Xiaofeng Lin, Dongxu Zhang, Jun Han, Zhichao Yang, Davin Hill, Tamer Soliman, Sanjit Singh Batra, Robert Tillman, Guang Cheng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, well-read robot assistant that can write medical advice. It's great at sounding confident and using the right big words. But sometimes, it confidently makes things up. In the medical world, this is dangerous. If the robot says a drug cures a disease when it doesn't, that's a "fabrication."

This paper introduces a new system to catch these specific types of lies, called EtHER, and a new training ground to teach it how to spot them, called MedFabric.

Here is the story of how they built it, explained simply:

The Problem: The "Style" Trap

The authors noticed that previous attempts to catch robot lies had a major flaw. They were like security guards who only looked at a person's clothing, not their ID card.

The Old Way: When researchers made fake medical answers to test detectors, they often changed the writing style too much. The fake answers sounded robotic and clunky, while the real answers sounded human.
The Mistake: The detectors learned to spot the "robotic style" rather than the actual lie. They would flag a fake answer just because it sounded weird, but they would miss a fake answer that sounded perfect.
The Result: When the robot wrote a lie that sounded exactly like a human (same sentence structure, same tone), the detectors got confused and failed.

The Solution Part 1: MedFabric (The "Perfect Forger")

To fix this, the authors built a new training dataset called MedFabric. Think of this as a master forger in a spy movie.

Step 1: The Style Shift: They took real, human-written medical facts and asked an AI to rewrite them. The goal wasn't to change the meaning, but to make the human text sound exactly like the AI's natural writing style. Now, the "real" and "fake" answers look identical in terms of style.
Step 2: The Subtle Lie: Next, they asked the AI to create a lie based on that rewritten text. But here's the trick: the lie had to be tiny. It wasn't a whole new story; it was just one or two words changed.
- Real: "This drug works on mice."
- Fake: "This drug works on rats."
Step 3: The Quality Check: They used a strict filter to ensure the fake answer was so similar to the real one that a human might not even notice the difference immediately.

The result is MedFabric: a collection of medical sentences where the "truth" and the "lie" are twins. They look, sound, and feel the same, except for a tiny factual error.

The Solution Part 2: EtHER (The "Word-Level Detective")

Now that they had these tricky "twin" sentences, they needed a detective that wouldn't get fooled by style. They built EtHER (EvaluaTe Hallucination with TablE decomposition and woRd masking).

Instead of reading the whole paragraph and guessing, EtHER breaks the problem down into three steps, like a forensic accountant:

Text2Table (The Organizer):
Imagine the medical text is a messy pile of papers. EtHER sorts this pile into a neat spreadsheet (a table). It pulls out specific facts (like "Drug X" and "Works on Mice") and puts them in their own boxes. This stops the detective from getting distracted by fancy sentence structures.
Word Masking and Filling (The Puzzle):
EtHER takes those neat boxes and plays a "fill-in-the-blank" game. It covers up a word (like "Mice") and asks the AI to guess what word should be there based on a trusted medical database.
- If the original sentence said "Rats" (the lie), but the database says "Mice," the AI will try to fill the blank with "Mice."
- When EtHER compares the original lie ("Rats") with the filled-in truth ("Mice"), the mismatch is obvious.
Hybrid Evaluation (The Double Check):
Finally, EtHER uses two tools to make a final decision:
- The Math Tool: It measures the distance between the words using math (embeddings) to see if they are different.
- The Brain Tool: It asks a smart AI to reason about the difference.
  By combining these two, EtHER avoids the "random guessing" that happens when you just ask an AI a question once.

The Results: Why It Matters

The authors tested EtHER against other top detectors using their new "twin" dataset (MedFabric).

The Old Detectors: When the lies were subtle (high similarity), the old detectors failed miserably, dropping their accuracy to below 50% (basically guessing). They were still looking at the "clothing" (style) instead of the "ID" (facts).
EtHER: Because EtHER looked at the specific words and facts, it stayed strong. It outperformed the best existing detectors by over 15%.

In short: The paper argues that to catch a smart liar, you can't just look at how they talk. You have to build a system that breaks their sentences down to the individual words and checks them against the facts, one by one. EtHER does exactly that, using a dataset (MedFabric) that forces the system to learn this skill.

Technical Summary: MedFabric and EtHER

Problem Statement

Large Language Models (LLMs) possess strong semantic and reasoning capabilities but frequently hallucinate in expert domains like medicine. Among various hallucination types, fabrication—the generation of fluent yet factually incorrect statements—poses the highest risk in high-stakes fields.

Existing approaches to detecting medical hallucinations face two critical limitations:

Data Deficiencies: Current medical hallucination datasets (e.g., Med-HALT, MedHallu) inadequately capture fabrication phenomena. They often rely on gathering correct and incorrect answers from disparate sources (human vs. LLM), creating stylistic disparities and distributional drift. Detectors trained on these datasets tend to overfit to superficial stylistic cues (e.g., distinguishing human vs. AI writing styles) rather than learning to identify genuine factual inconsistencies.
Model Fragility: Existing detectors (embedding-based or LLM-as-a-judge) perform poorly when structural similarity between ground truth and fabricated samples is high. Embedding models fail to distinguish subtle word-level deviations, while zero-shot LLM evaluators exhibit high variance and inconsistency due to sampling randomness and prompt sensitivity.

Methodology

The authors propose a unified, data-centric framework comprising a new dataset generation pipeline (MedFabric) and a modular detection system (EtHER).

1. MedFabric: A Data-Centric Generation Pipeline

To address the lack of realistic fabrication data, the authors developed a three-stage pipeline to generate word-level fabrications that preserve syntactic and stylistic fidelity while introducing subtle factual deviations:

Stylistic Alignment: Human-authored ground truths are rewritten by an LLM to align with LLM stylistic distributions, eliminating the "human vs. AI" stylistic shortcut.
Evidence-Conditioned Fabrication: Fabrications are generated by conditioning the LLM on the rewritten ground truth and specific evidence chunks. This ensures the fabricated text remains contextually coherent but introduces fine-grained factual errors (e.g., changing a species from "human" to "rat").
Quality Control: A two-step evaluation process ensures the generated fabrications are challenging:
- Structural Similarity: Measured via ROUGE-L recall; samples with scores below 0.7 are discarded to ensure the fabrication is not too divergent.
- Semantic Difficulty: A self-play preference optimization (SPPO) strategy is used where an LLM agent attempts to distinguish the fabrication from the rewritten ground truth. Only fabrications that "blind" the agent (i.e., are misclassified) are retained, ensuring the dataset captures genuine failure modes.

2. EtHER: A Modular Word-Level Fabrication Detector

To address the limitations of existing detectors, the authors introduce EtHER (EvaluaTe Hallucination with TablE decomposition and woRd masking), a framework designed for stable, interpretable, and word-level factuality assessment. It consists of three modules:

Text2Table Decomposition: Unstructured medical statements are converted into structured entity–description tables. This decomposes complex sentences into entity-context pairs (e.g., "TGFBI expression" $\to$ "unfavorable lymphatic recurrence"), enhancing semantic interpretability and reducing contextual ambiguity.
Word Masking and Filling: The system generates masked versions of the entity-description tables. An LLM, guided by a domain-specific Retrieval-Augmented Generation (RAG) system, fills these masks with verbatim phrases from retrieved evidence. This creates paired reconstructions that allow for direct, fine-grained comparison between the model's output and verified knowledge.
Hybrid Sentence Pair Evaluation: To minimize LLM randomness and improve stability, the system employs a hybrid approach:
1. Embedding-based filtering: Sentence pairs are encoded to measure semantic distance; pairs exceeding a threshold are flagged.
2. LLM-based reasoning: Flagged candidates are re-evaluated by an LLM agent to distinguish factual conflicts from benign lexical variations.
3. Rule-based Aggregation: Final decisions are aggregated via a rule-based layer to determine response-level fabrication.

Key Results

Empirical evaluations on the MedFabric dataset and existing benchmarks (MedHallu, Rewritten Ground Truth) demonstrate the following:

Superior Performance on Subtle Fabrications: EtHER outperforms state-of-the-art detectors (including SAPLMA, RelD, TSV, and GCA) by over 15% on word-level fabrication benchmarks. Specifically, on MedFabric (high structural similarity, ROUGE-L $\approx$ 0.77), EtHER achieves an overall F1 of 60.6%, compared to an average of 44.3% for other models.
Improved Recall: EtHER retrieves nearly twice as many fabrications as the model average (68.6% vs. 34.8%) while maintaining comparable performance on truthful answers.
Robustness to Structural Similarity: Unlike fine-tuned models that degrade sharply as structural similarity increases (dropping from ~69% F1 on low-similarity data to ~42% on MedFabric), EtHER maintains stable performance (60–70% F1) across datasets with varying structural similarities.
Low Variance: Despite utilizing multiple generation steps, EtHER exhibits prediction variance comparable to single-step zero-shot LLM evaluators, attributed to the constrained, precise nature of its sub-tasks (masking and filling) which reduces autoregressive randomness.

Significance and Claims

The paper claims to establish a unified paradigm for studying and mitigating fabrication-type medical hallucinations. Its primary contributions are:

Uncovering Stylistic Bias: Highlighting that existing detectors often rely on stylistic patterns between ground truth and fabricated samples rather than factual content, leading to spurious classification.
Realistic Data Simulation: Providing a data generation pipeline that simulates realistic medical fabrication scenarios, resulting in the MedFabric dataset, which offers a rigorous benchmark with high structural similarity and subtle semantic shifts.
Word-Level Detection Framework: Proposing EtHER, a detector that aligns with word-level fabrications through structured reasoning and factual alignment, offering a scalable foundation for trustworthy LLM systems in high-stakes medical applications.

The authors position this work as a step toward reducing the workload of medical experts in verifying patient consultations and improving the reliability of LLMs in domains where factual precision is critical.