Trust but Verify: Mitigating Medical Hallucinations via Post-Hoc Adversarial Auditing and Multi-Agent Feedback Loops

Technical Summary: Trust but Verify – Mitigating Medical Hallucinations via Post-Hoc Adversarial Auditing and Multi-Agent Feedback Loops

Problem Statement
Large Language Models (LLMs) are increasingly deployed in healthcare for clinical decision support, yet their probabilistic nature leads to "hallucinations" that pose severe safety risks. A critical, often overlooked failure mode is regulatory knowledge obsolescence: LLMs frequently recommend pharmaceuticals that were historically correct but are now banned, withdrawn, or carry black box warnings by regulatory bodies (e.g., FDA, EMA). The paper highlights that even state-of-the-art (SOTA) models equipped with Retrieval-Augmented Generation (RAG) and web browsing capabilities fail to prioritize current safety constraints over parametric training data, leading to dangerous clinical recommendations. Existing mitigation strategies, such as static semantic guardrails or simple RAG, often lack the active, adversarial verification required to detect these temporal knowledge gaps.

Methodology
The authors propose a modular, five-agent architecture titled "Trust but Verify," designed to enforce regulatory compliance through a post-hoc adversarial auditing loop. The system utilizes a single LLM backbone (tested across GPT-OSS, Llama-3, and Falcon-3 families) but differentiates agents via prompt-based persona redirection rather than heterogeneous model weights.

The workflow consists of five functional roles:

1. Router Agent: Classifies queries as "Medical" or "General." Only medical queries trigger the safety pipeline, optimizing latency and cost.
2. Medical Clinical Agent: Acts as the primary reasoning engine, generating candidate treatment recommendations based on the input query.
3. Entity Extractor Agent: Parses the clinical agent's natural language output into a structured JSON format (extracting drug and condition) to facilitate machine-readable verification.
4. Safety Auditor Agent: The core safety component. It performs real-time web-grounded searches (via Tavily API) against authoritative regulatory sources (FDA, NIH) to verify the current status of the extracted drug. If a drug is flagged as banned or withdrawn, the auditor triggers a feedback loop, sending the violation back to the Clinical Agent to force a retry or a safe refusal.
5. General Chat Agent: Handles non-critical interactions to conserve resources.

The system allows up to three retry attempts. If no safe alternative is found among the options, the system is designed to issue a "safe refusal" rather than a hallucinated recommendation.

Experimental Setup
The study utilized a curated adversarial dataset of 103 clinical Multiple Choice Questions (MCQs). In these questions, the historically correct answer corresponds to a drug that has since been withdrawn or banned. The dataset was constructed using DrugBank, ensuring that the "correct" label in the training data context is actually a safety violation in the current regulatory context.

Three experimental configurations were evaluated:

• Experiment I (Vanilla): Single-shot inference where the LLM acts as a clinical expert without external auditing.
• Experiment II (Agentic): The proposed five-agent pipeline with the adversarial feedback loop.
• Experiment III (SOTA Evaluation): Testing leading proprietary models (ChatGPT-5.1/5.3, Gemini 3 Flash/3.1 Pro) with native browsing enabled to assess if current commercial solutions mitigate this specific failure mode.

Key Metrics
Performance was evaluated using:

• Label Accuracy: Standard correctness against the dataset's ground truth (which is inherently unsafe in this adversarial setup).
• Pointwise Score (PS): A specialized metric penalizing definitive recommendations for banned drugs (-0.25), rewarding safe refusals (0.0), and rewarding safe correct predictions (+1.0).
• Hallucination Error Rate (HER): The frequency of definitive recommendations for withdrawn substances.
• Component Fidelity (CF): The ratio of successful functional outputs to total agent invocations, measuring the reliability of the multi-agent orchestration.

Results

• Vanilla Performance: All tested open-access models exhibited high "accuracy" (matching the banned drug option) but correspondingly high HERs (ranging from 90.29% to 99.03%). Pointwise scores clustered near -0.25, indicating maximum safety penalties.
• Agentic Performance: The proposed architecture significantly reduced HERs across all models by approximately 53% on average (e.g., Llama-3-70B dropped from 99.03% to 37.86%). While raw label accuracy decreased (as the system correctly refused to select the banned "correct" answer), the Pointwise Score shifted from -0.25 toward 0.0, indicating a successful transition to safe refusals.
• SOTA Failure: Experiment III revealed that even proprietary models with advanced reasoning and browsing capabilities failed to prevent the recommendation of banned drugs. These models often acknowledged the withdrawal status in their search results but ultimately prioritized their parametric knowledge to select the banned drug as the answer, demonstrating a "clinical safety regression."
• System Integrity: The Component Fidelity scores for the agentic pipeline ranged between 73.82% and 85.71%, confirming the robustness of the multi-agent orchestration.

Significance and Claims
The paper claims to demonstrate a practical, model-agnostic approach for deploying autonomous AI in safety-critical healthcare settings. Its primary contributions are:

1. Identification of a Critical Gap: Highlighting that current RAG and SOTA models fail to integrate real-time regulatory data effectively, leading to the recommendation of banned substances.
2. Architectural Innovation: Proposing a non-linear, adversarial audit loop that treats safety as a deterministic verification process rather than a probabilistic generation task. By decoupling clinical reasoning (LLM) from factual truth (real-time auditor), the system ensures compliance without requiring model retraining.
3. Safety over Fluency: The framework prioritizes patient safety by converting dangerous recommendations into appropriate refusals, shifting the system's objective from "answering correctly" to "answering safely."
4. Scalability: The use of a single backbone with prompt-based agent differentiation offers a scalable solution that does not require parameter expansion or expensive retraining.

The authors conclude that while the multi-agent approach introduces inference latency, this trade-off is necessary for deterministic safety in high-stakes clinical environments. They position this work as a step toward integrating real-time regulatory data into LLM pipelines to support, rather than replace, clinical decision-making.