💬 NLP

Do No Harm? Hallucination and Actor-Level Abuse in Web-Deployed Medical Large Language Models

This paper presents a large-scale assessment of over 6,000 web-deployed medical large language models, revealing significant risks of hallucinations, policy violations, and privacy gaps while introducing new evaluation frameworks and a dataset to guide future safety improvements.

Original authors: Sunday Oyinlola Ogundoyin, Muhammad Ikram, Rahat Masood

Published 2026-05-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sunday Oyinlola Ogundoyin, Muhammad Ikram, Rahat Masood

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the internet's "App Store" for AI, but instead of games or productivity tools, it's filled with thousands of digital doctors. These are custom AI chatbots (called MedGPTs) that anyone can build and publish to give medical advice, diagnose symptoms, or explain treatments.

The paper you provided is like a massive, undercover health inspection of this digital marketplace. The researchers didn't just look at the top-rated, popular doctors; they audited over 6,000 of them to see if they are safe, honest, or if they are actually dangerous.

Here is what they found, broken down into simple concepts:

1. The "Confident Liar" Problem (Hallucinations)

Imagine a tour guide who speaks with absolute confidence but makes up facts about history. In the medical world, this is called a hallucination.

The Finding: About 25% to 30% of these AI doctors are "lying" or making up medical facts. They might confidently tell you to take a dangerous drug or ignore a serious symptom.
The Twist: You might think the most popular doctors are the safest. But the study found that popularity is a bad lie detector. The "famous" AI doctors were just as likely to be making things up as the obscure ones. In fact, the less popular ones were often even worse.
The User Blind Spot: The researchers found that users don't seem to notice these lies. If an AI gives a smooth, confident answer, people give it 5 stars, even if the advice is medically wrong. It's like giving a 5-star rating to a chef who serves you a delicious-looking but poisonous meal.

2. The "Bad Actor" Problem (Design Abuse)

Some of these AI doctors aren't just accidentally wrong; they are designed to be risky.

The Finding: Nearly 50% of the AI doctors had "red flags" in how they were built.
The Metaphor: Imagine a restaurant owner who puts up a sign saying "We are a hospital" (even though they aren't), hides the menu, and refuses to show you where the food comes from.
The Specifics:
- Some creators named their bots things like "Instant Cancer Diagnosis" to trick people into thinking they are real doctors.
- Many of these bots have a feature called "Actions" (which lets them connect to outside websites), but 57% of them didn't have a privacy policy. It's like walking into a clinic where the doctor takes your blood sample but refuses to tell you what they will do with it.
- Even when they did have a privacy policy, nearly 70% of them were broken, vague, or didn't actually protect your data.

3. The "Open Source" vs. "Store-Bought" Showdown

The researchers also compared these store-bought AI doctors to "open-source" ones (models that developers share freely, like a community recipe book).

The Store-Bought (MedGPTs): These were generally better at sounding smart and accurate (they got the facts right more often). However, they were less stable—sometimes they gave a great answer, and the next time you asked the same question, they gave a totally different, confusing answer.
The Open-Source: These were more consistent (they gave the same answer every time), but they were less accurate on the facts.
The Lesson: Neither type is perfect. The "fancy" ones sound better but wobble; the "community" ones are steady but might get the details wrong.

4. The "Trust Signals" Are Broken

In a normal app store, if an app has bad reviews or low ratings, you avoid it.

The Finding: In this medical AI store, ratings and reviews tell you nothing about safety.
The Metaphor: It's like a car dealership where the cars with the most test drives and the highest "5-star" ratings are actually the ones with broken brakes. The researchers found that how much people talked to the AI had almost zero connection to whether the AI was actually telling the truth.

The Bottom Line

The paper concludes that we cannot trust these AI doctors just because they are popular or because they sound confident.

The System is Broken: The current way of checking these bots (relying on user reviews and star ratings) is failing.
The Danger: Because patients often don't have medical training, they can't spot when the AI is lying or when the doctor is "faking" authority.
The Solution: We need a new kind of "health inspector" that checks the AI's facts, its design, and its privacy rules before it is allowed to talk to patients. The researchers have released a new dataset and a set of tools to help build these inspectors.

In short: The digital medical marketplace is full of confident liars and risky designs, and the "customer reviews" aren't helping us spot them. We need better safety checks before we let these AI doctors into our lives.

Technical Summary: Do No Harm? Hallucination and Actor-Level Abuse in Web-Deployed Medical Large Language Models

1. Problem Statement

The rapid deployment of custom medical Large Language Models (MedGPTs) on commercial platforms like the OpenAI Store presents significant safety risks that are often overlooked. While these models offer diagnostic suggestions and health advice, they face two primary, interconnected threats:

Clinical Hallucination: The generation of fluent but factually incorrect, incomplete, or fabricated medical information (e.g., unsafe treatments, incorrect drug advice). Unlike general LLM errors, these hallucinations use domain-specific language, making them difficult for non-expert patients to detect.
Actor-Level Abuse: Intentional design choices by developers that predispose models to unsafe or unethical use. This includes deceptive metadata (names, descriptions), the circumvention of safety safeguards via tools (e.g., web browsing), and the lack of functional privacy disclosures, particularly in models enabled with "Actions" (external API calls).

Current literature largely focuses on hallucination in base models or generic misuse in general-purpose apps, leaving a gap in the systematic evaluation of deployed, user-facing healthcare GPTs regarding both content accuracy and developer-driven structural risks.

2. Methodology

The authors conducted a large-scale audit of the OpenAI GPT Store, introducing a dual-layer evaluation framework.

Data Collection

Scope: A crawler extracted metadata from 9,245 unique GPTs using OpenAI's public search API and a domain-specific taxonomy of 171 medical keywords.
Filtering: After manual curation to ensure healthcare relevance, a final dataset of 6,233 MedGPTs was established.
Sampling: Due to platform rate limits, a stratified sample of 1,500 MedGPTs was selected for deep evaluation: the Top 1,000 (by conversation count), a random Middle 250, and a Bottom 250.
Comparison Group: 10 prominent open-source medical LLMs (e.g., Galactica, PMC-LLaMA, MedAlpaca) were evaluated using the same metrics for comparison.

Evaluation Frameworks

MedGPT-HEval (Hallucination Detection):
- Input: Each model was queried five times with standardized clinical scenarios from the MedQA benchmark.
- Metrics: Responses were scored using four complementary metrics:
  - G-Eval: A chain-of-thought framework using Gemini 3.1 Pro to assess factual accuracy, consistency, and citation reliability.
  - BARTScore: Measures the likelihood of the generated text given the input prompt to assess faithfulness.
  - Semantic Entropy: Quantifies model uncertainty; higher values indicate greater instability and hallucination risk.
  - Cosine Similarity: Measures semantic alignment between the response and the clinical input using BioBERT embeddings.
- Validation: Automated scores were validated against human annotation on a stratified subset (300 models), achieving 90.5–94.0% agreement (Cohen's kappa = 0.8163).
Actor-Level Abuse Evaluation:
- Method: An LLM-as-a-judge pipeline (using Gemini 3.1 Pro) analyzed metadata (names, descriptions, conversation starters) against OpenAI's usage policies.
- Categories: Risks were classified into health consultations, scams, privacy violations, and illicit activities.
- Thresholding: K-means clustering ( $k=2$ ) was applied to misuse risk scores to determine a compliance threshold of 0.45. This was validated by expert review (Cohen's kappa = 0.8188).
- Privacy Audit: Specifically for the 170 models with "Actions" enabled, the authors assessed the accessibility and compliance of linked privacy policies.

3. Key Results

Clinical Hallucination

Prevalence: 25–30% of MedGPTs across all popularity tiers scored below 0.8 in G-Eval.
Tier Disparities: Bottom- and middle-tier models exhibited the highest hallucination risks and weakest contextual alignment. Top-tier models performed better but still showed significant variance.
Metric Specifics:
- Only 37.27% of MedGPTs achieved a BARTScore $\ge -3.5$ .
- Only 41.07% reached a Cosine Similarity $\ge 0.4$ .
- 59.87% had semantic entropy $< 2$ , indicating moderate stability, though bottom-tier models showed high variability.
User Perception: There was a near-zero correlation between user engagement (conversation counts, star ratings) and hallucination metrics (e.g., $r \approx -0.03$ for G-Eval). Users reward activity, not accuracy; less reliable models often received more positive feedback.

Actor-Level Abuse and Privacy

Misuse Rates: 49.8% of all evaluated MedGPTs violated operational policies.
- Top 1,000: 54.3% non-compliant.
- Middle 250: 48.0% non-compliant.
- Bottom 250: 33.6% non-compliant.
Privacy Gaps: Among 170 Action-enabled models:
- 57.06% lacked accessible privacy policies.
- Of the 42.94% with accessible policies, nearly 70% scored below the compliance threshold (missing disclosures on data sharing, cookies, user rights).
- Many policies were broken links, generic templates, or redirected to unrelated domains.

Open-Source vs. Web-Deployed Models

Accuracy vs. Stability: MedGPTs demonstrated higher factual accuracy (Avg G-Eval: 0.9238) and semantic alignment (Avg Cosine: 0.4054) compared to open-source models (Avg G-Eval: 0.4558; Avg Cosine: 0.3731).
Consistency: Open-source models were more stable and predictable (lower semantic entropy: 1.6129 vs. 1.9272 for MedGPTs), suggesting a trade-off where web-deployed models may prioritize coherence over the stability found in community-driven alternatives.

4. Key Contributions

HAA-MedGPT Dataset: The release of the first large-scale, structured dataset containing metadata, interaction logs, and evaluation scores for 6,233 custom web-deployed MedGPTs.
Dual-Layer Safety Analysis: The introduction of MedGPT-HEval for hallucination detection combined with an actor-level misuse evaluator, addressing both output-level errors and design-level risks.
Systemic Governance Findings: Empirical evidence that platform trust indicators (ratings, reviews) do not correlate with safety or accuracy, and that nearly half of deployed models violate operational policies.
Privacy Risk Quantification: The first empirical assessment showing that the majority of Action-enabled medical GPTs lack functional privacy disclosures.

5. Significance and Claims

The paper argues that the current deployment of medical LLMs on open marketplaces suffers from systemic governance failures. The authors claim that:

Popularity is not a proxy for safety: High conversation counts and positive reviews do not indicate factual reliability or adherence to safety policies.
Risk is structural: Unsafe behaviors are often embedded at the design level by developers (e.g., deceptive names, missing privacy policies) rather than being incidental generation errors.
Multi-metric evaluation is essential: Relying on a single metric (like accuracy) is insufficient; a combination of factual alignment, semantic stability, and policy compliance is required to assess medical LLM safety.
Urgent need for safeguards: The findings highlight the necessity for proactive safety assessments, enforceable policy standards, and multi-metric evaluation frameworks before models are deployed to the public.

The authors conclude that without these interventions, the "trust gap" between perceived reliability and actual safety will continue to expose patients to clinical and privacy risks.