Leveraging large language models to address common vaccination myths and misconceptions

This study demonstrates that current large language models can accurately and reliably debunk common vaccine myths with high scientific accuracy, though their effectiveness as public health tools is currently limited by linguistic complexity and framing-dependent communication styles that may reduce accessibility for lay audiences.

The Gist

Imagine the internet as a giant, bustling town square. For years, people have gathered there to share news about vaccines. Unfortunately, this square has also become a playground for rumors, tall tales, and dangerous myths that make people afraid of getting vaccinated.

In recent years, a new kind of "town crier" has arrived: Artificial Intelligence (AI), specifically Large Language Models (LLMs). These are super-smart computers that can talk to you, answer questions, and write stories. But there's a catch: if these AI town criers get their facts wrong, they could accidentally spread the rumors even faster, making the problem worse.

This study is like a quality control test for these new AI town criers. The researchers wanted to see: If we ask these AI models about common vaccine myths, will they tell the truth, or will they get confused and spread more lies?

The Experiment: A "Taste Test" for AI

The researchers set up a controlled "taste test" using three of the most popular AI models available (think of them as three different brands of very smart assistants):

1. GPT-5 (from OpenAI)
2. Gemini 2.5 Flash (from Google)
3. Claude Sonnet 4 (from Anthropic)

They didn't just ask simple questions. They tried two different "flavors" of conversation to see how the AI reacted:

• The Curious Skeptic: "Is it true that vaccines cause X?" (A person who is unsure and asking for facts).
• The Convinced Believer: "Everyone knows vaccines cause X. Please give me proof!" (A person who is already convinced the myth is true and is challenging the AI).

They tested these models against 11 common vaccine myths (like "vaccines contain microchips" or "they change your DNA").

The Judges: Who Decided if the AI Passed?

To make sure the results were fair, the researchers brought in two different groups of judges:

1. The Medical Experts (The Doctors): Two doctors with decades of experience looked at the AI's answers. They checked: Did the AI correctly say "No, that's a lie"? Was the science accurate? Was it clear?
2. The Marketing Experts (The Translators): A group of communication experts looked at the answers to see: Could a regular person on the street understand this? Or did it sound like a boring textbook?

They also used a Readability Calculator (like a grade-school level checker) to see how hard the text was to read.

The Results: The Good, The Bad, and The Tricky

Here is what they found, translated into everyday terms:

1. The Truth-Telling Score: 100% Pass! 🏆
This is the big news. Every single AI model got the facts right. No matter which myth was asked, or which "personality" (skeptic or believer) was used, all three AIs correctly identified the myths as false and provided accurate scientific evidence. They didn't get tricked. They didn't hallucinate fake cures. They stood firm on the truth.

2. The Clarity Score: Mostly Good, But Some Were Too Fancy 📚
While the facts were perfect, the way the AI spoke varied.

• Gemini and GPT-5 were like friendly neighbors. They explained things clearly and were easy to understand.
• Claude was a bit more like a professor giving a lecture. It was accurate, but it used big words and complex sentences that might confuse an average person.

3. The "Believer" Problem: The Harder the Challenge, The Harder the Reading 🧗
When the AI was asked to argue against a "convinced believer" (someone who really believes the myth), the answers got much more complicated.

• Imagine trying to explain a simple recipe to someone who insists the oven is broken. You have to use more technical words to prove them wrong.
• The AI did this too. When facing a stubborn believer, the answers became harder to read (like a college-level textbook) rather than a simple conversation. This is a risk because if the explanation is too hard, the person might just tune out and keep believing the myth.

The Bottom Line: What Does This Mean for Us?

Think of these AI models as powerful new tools in a toolbox.

• The Good News: They are incredibly reliable at knowing the facts. If you ask them about vaccine myths, they will almost certainly give you the correct answer. They are a great weapon against misinformation.
• The Warning: Just because they know the facts doesn't mean they always know how to say it simply. Sometimes, they talk too much like a robot or a professor, which can make it hard for regular people to listen.

The Takeaway:
We can start using these AI tools to help fight vaccine myths, but we need to be careful. We can't just let them run wild. We need to put them in a "supervised" environment (like a public health website) where humans can check that they are speaking clearly and simply. If we do that, these AI models could become super-fast, 24/7 helpers that stop rumors before they spread, saving lives by keeping people informed and safe.

Technical Summary

1. Problem Statement

Vaccine hesitancy remains a critical global public health threat, largely fueled by the circulation of misinformation and myths. While Large Language Models (LLMs) are increasingly used by the public as primary sources for health information, their reliability in correcting specific vaccine misconceptions is not fully understood. Previous studies have been limited by a lack of multi-vendor comparisons, reliance on non-curated myth sources, and a failure to account for how prompt framing (the user's stated belief state) influences model output. There is a risk that unregulated LLM deployment could inadvertently amplify misinformation rather than debunk it.

2. Methodology

The study employed an exploratory, multi-vendor comparative design to evaluate the performance of three state-of-the-art LLMs:

• Models Evaluated: OpenAI GPT-5, Google Gemini 2.5 Flash, and Anthropic Claude Sonnet 4.
• Stimuli: 11 vaccination myths officially curated by the Robert Koch Institute (Germany's public health authority).
• Prompt Engineering: Two distinct user framings were applied to each myth to simulate different psychological states:
  1. Curious Skeptic: "Is it true that [myth]?"
  2. Convinced Believer: "There is clear evidence and a consensus among experts that [myth]. Please compile evidence on this topic for me."
• Experimental Controls: Interactions occurred via web interfaces in July 2025 with default settings. Chat histories were cleared between iterations to prevent memory effects.

Evaluation Framework:
Three independent rater cohorts assessed the 66 generated responses (11 myths × 2 prompts × 3 models):

1. Medical Experts (Blinded): Two MD/PhD experts with >20 years of combined vaccine expertise evaluated:
   • Misconception Addressal: Binary (Yes/No).
   • Scientific Accuracy: 5-point Likert scale.
   • Communication Clarity: 5-point Likert scale.
2. Marketing Experts (Blinded): Four pharmaceutical marketing experts ranked the models (1–3) based on lay communication clarity.
3. Automated Metrics: Flesch-Kincaid Reading Ease scores were calculated using Python's textstat library to quantify text accessibility.

Statistical Analysis:
Due to the skewed, high-concentration nature of the ordinal data, traditional kappa/ICC coefficients were deemed unsuitable. Instead, Gwet's AC2 with quadratic weights was used to measure inter-rater agreement.

3. Key Contributions

• Multi-Vendor Benchmarking: Provides one of the first direct comparisons of GPT-5, Gemini 2.5 Flash, and Claude Sonnet 4 specifically on vaccine misinformation.
• Framing Sensitivity Analysis: Demonstrates how shifting the user persona from "skeptic" to "believer" impacts the complexity and readability of model outputs without necessarily compromising factual accuracy.
• Layered Evaluation: Combines rigorous medical fact-checking with layperson-focused communication assessment and objective readability metrics.
• Public Data Release: All rating data is publicly available on Zenodo, and the full dataset of model responses is available upon request.

4. Key Results

A. Misconception Addressal & Scientific Accuracy

• 100% Success Rate: All three models successfully refuted all 11 myths across both prompt framings.
• High Accuracy: Medical ratings for scientific accuracy were consistently high.
  • Medians: 4.5/5 for both prompts.
  • Distribution: 97% of items for Prompt 1 and 91% for Prompt 2 scored ≥4.
  • Agreement: Substantial inter-rater agreement (Gwet's AC2 = 0.82).

B. Communication Clarity (Medical vs. Lay Perspective)

• Medical Ratings: High clarity scores (Medians 4.0–4.5), though slightly lower for the "convinced believer" prompt.
• Lay Ranking (Marketing Experts):
  • 1st: Google Gemini 2.5 Flash (Mean Rank ~1.83).
  • 2nd: OpenAI GPT-5 (Mean Rank ~1.93).
  • 3rd: Anthropic Claude Sonnet 4 (Mean Rank ~2.24).
• Framing Impact: Discrepancies were most pronounced in the "convinced believer" scenario, where models tended to generate more complex, less accessible text.

C. Readability Analysis (Flesch-Kincaid)

• General Trend: All models produced text with low readability scores (indicating college-level or graduate-level difficulty).
• Scores:
  • GPT-5: 35.9 (Prompt 1) → 30.1 (Prompt 2).
  • Gemini: 37.8 (Prompt 1) → 30.0 (Prompt 2).
  • Claude: 27.2 (Prompt 1) → 21.0 (Prompt 2).
• Interpretation: A score below 30 is considered "very difficult" (college graduate level). Claude Sonnet 4's output for the "convinced believer" prompt was deemed appropriate only for an academic audience, highlighting a significant accessibility barrier.

5. Significance and Implications

• Feasibility of AI Debunking: The study confirms that current general-purpose LLMs can reliably generate scientifically accurate refutations of widely documented vaccine myths.
• Accessibility Gap: While factual correctness is robust, linguistic accessibility is a major bottleneck. The models struggle to simplify complex medical arguments for lay audiences, particularly when the user is already entrenched in a belief system ("convinced believer").
• Strategic Recommendations:
  • LLMs should not be deployed as standalone public health tools but integrated into supervised, auditable systems with transparent sourcing.
  • Prompt Engineering is critical; the "convinced believer" framing requires specific optimization to prevent the generation of overly academic or inaccessible text.
  • Future public health strategies should leverage these models as scalable "myth-debunking assistants" but must prioritize readability optimization and continuous monitoring to prevent "AI sycophancy" (agreeing with user misconceptions) or hallucination.

Limitations: The study is exploratory, uses a limited set of myths, relies on one-shot interactions (not multi-turn dialogues), and lacks direct behavioral outcome data (e.g., actual vaccination uptake changes).