Investigating Vaccine Buyer's Remorse: Post-Vaccination Decision Regret in COVID-19 Social Media Using Politically Diverse Human Annotation

This paper introduces a novel, politically diverse annotated dataset from YouTube to quantify and analyze the prevalence, causes, and narrative patterns of COVID-19 vaccine regret, revealing that while it constitutes less than 2% of public discourse, it is concentrated among vaccine-skeptic influencers and primarily expressed through first-person accounts of adverse health events.

The Gist

Imagine the global conversation about vaccines as a massive, chaotic town square. For years, everyone was arguing about whether to enter the square (get vaccinated). But this paper asks a different, quieter question: What happens to people after they've already walked through the door?

Specifically, the authors are investigating "Vaccine Buyer's Remorse." Think of it like buying a new car. You drive it off the lot, but then you hear a weird noise, or you realize the gas mileage is terrible, and you think, "Oh no, I made a mistake. I wish I hadn't bought this."

Here is a breakdown of how the researchers investigated this feeling, using simple analogies:

1. The Digital "VAERS" (The Data Collection)

Official health agencies have a system called VAERS where people can voluntarily report bad reactions to vaccines. It's like a suggestion box at a restaurant. But in the digital age, people don't just write letters; they scream into the void on YouTube.

The researchers went to YouTube, specifically to the comment sections of three major news channels (Fox, CNN, MSNBC) and popular online influencers. They gathered 80 million comments—a digital ocean of chatter. From this ocean, they filtered out the waves to find the specific droplets of water that talked about vaccines, side effects, or regret.

2. The "Politically Diverse Jury" (The Human Annotation)

This is where the study gets clever. Labeling something as "regret" is tricky. If someone says, "I hate this vaccine," are they saying they regret taking it, or just that they hate the idea of it? And does a Republican see regret differently than a Democrat?

To avoid bias, the researchers didn't just ask one person to judge the comments. They assembled a jury of 201 different people from across the political spectrum (Democrats, Republicans, and Independents).

• The Analogy: Imagine a courtroom where the judge, the prosecutor, and the defense attorney all have to agree on what a witness said before it counts as evidence.
• The Result: They found that while political views changed how people labeled who was talking (the "subject"), they mostly agreed on whether someone actually expressed regret.

3. The "Robot Detectives" (The AI Models)

You can't read 80 million comments by hand. So, the team trained "Robot Detectives" (Large Language Models, or LLMs) to do the heavy lifting.

• Stage 1 (The Bouncer): A fast, simple robot scanned the comments to see if they were even about vaccines. If a comment was about a basketball game, the bouncer kicked it out.
• Stage 2 (The Detective): The comments that passed the bouncer went to a smarter, more expensive robot. This detective read the text carefully to answer three questions:
  1. Who is this about? (Is it the person writing, someone they know, or just a vague "people"?)
  2. Did they get the shot?
  3. Do they regret it?

4. The Big Discoveries (The Findings)

Once the robots and the human jury did their work, some surprising patterns emerged:

• The "Rare Bird" Effect: Even though it feels like everyone is talking about vaccine regret, it's actually quite rare. Only about 1.1% of the relevant comments expressed regret. It's a loud minority, but a small one.
• The Echo Chamber: Regret wasn't spread evenly. It was three times more common on channels run by "vaccine-skeptic" influencers compared to mainstream news or "pro-vaccine" influencers.
  • Analogy: If you go to a party where everyone is complaining about the food, you'll think the food is terrible. If you go to a different party where everyone loves the food, you'll think it's great. The "vaccine-skeptic" channels were like the first party, amplifying the few voices of regret.
• Who is Regretting? Most of the regret came from people talking about themselves (First-Person). They were sharing personal stories of getting sick or feeling unwell after the shot.
• The "Why": The main reasons for regret were:
  1. Adverse Health Events: "I got sick after the shot." (The most common reason).
  2. Lack of Efficacy: "I got the shot but still got COVID."
  3. Coercion: "I only got it because my boss forced me."

5. Why This Matters

The authors argue that while regret is a small part of the total conversation, it is a loud part. In the world of social media, one dramatic story about a side effect can travel faster than a thousand statistics saying the vaccine is safe.

By understanding who is feeling regret, why they feel it, and where they are saying it, public health officials can stop shouting general messages and start having specific conversations. Instead of saying "Vaccines are safe," they might need to say, "We hear your concerns about side effects, and here is what we are doing to help."

In a nutshell: This paper used a mix of human juries and robot detectives to listen to the quiet whispers of regret in a noisy digital crowd. They found that while regret exists, it's mostly concentrated in specific corners of the internet and is driven by personal stories of health issues or feeling forced. Understanding this helps us build better trust in the future.

Technical Summary

1. Problem Statement

While extensive research exists on pre-vaccination hesitancy, there is a significant gap in understanding post-vaccination regret (or "vaccine buyer's remorse"). This phenomenon involves the distress individuals feel after making a health-related choice, often driven by adverse events, perceived coercion, or disillusionment with efficacy.

• The Challenge: Public health surveillance systems (like VAERS) rely on voluntary reporting, which is passive and limited. Social media offers a vast, informal analog, but analyzing it presents methodological hurdles:
  • Distinguishing between first-person (personal experience) and vicarious (reporting on others) narratives.
  • Capturing the nuanced, often ambivalent emotion of "regret," which traditional sentiment analysis often misses.
  • Mitigating political bias, as vaccine discourse is highly polarized, and annotator political affiliation can systematically influence labeling.

2. Methodology

The authors developed a multi-stage hybrid inference pipeline and a novel benchmark dataset to analyze COVID-19 vaccination discourse on YouTube.

A. Data Collection & Curation

• Corpus: 80.3 million comments from 65,886 videos across three major US cable news networks (Fox News, CNN, MSNBC) and 981 videos from prominent influencers (both pro-vaccine and vaccine-skeptic).
• Filtering: A keyword-based filter reduced the corpus to ~1.37 million vaccine-relevant comments.
• Benchmark Dataset ($D_{bench}$): A stratified sample of 2,000 comments was created to ensure a balanced representation of regret vs. non-regret.
• In-the-Wild Dataset ($D_{wild}$): A processed corpus of 600,000 comments (300k from news, 300k from influencers) used for large-scale analysis.

B. Human Annotation Strategy

To address political bias, the study employed a politically diverse annotation panel:

• Recruitment: 201 unique US-based annotators (67 Democrats, 67 Republicans, 67 Independents).
• Process: Each comment in $D_{bench}$ was annotated by three raters (one from each political group).
• Resolution: Final labels were determined by majority vote.
• Labels:
  1. Regret: Positive (Explicit or Implicit) vs. Negative (Neutral/Unrelated).
  2. Narrative Perspective: First-Person (Self), Third-Party/Vicarious (Other), or Unspecified.
  3. Vaccination Status: Vaccinated vs. Not/Unclear.

C. Computational Pipeline

The authors utilized a two-stage hybrid inference pipeline to balance accuracy and computational efficiency:

1. Stage 1: Relevance Filter: A Natural Language Inference (NLI) model (ModernBERT-large-nli) filters out irrelevant comments.
   • Hypothesis: "This comment mentions or discusses anything related to vaccines..."
   • Performance: F1-score of 0.868.
2. Stage 2: Expert Reasoner: A fine-tuned Large Language Model (LLM) performs multi-label classification (Subject, Vaccinated, Regret) and reason extraction.
   • Model Selection: Eight LLMs were evaluated (Zero-shot, Few-shot, and Fine-tuned). Llama-3.1:70b was selected as the final "Expert Reasoner" due to its superior F1-scores on Regret (0.83) and Vaccinated (0.87) tasks and faster inference speed compared to Mixtral.
   • Fine-tuning: Models were fine-tuned using LoRA (Low-Rank Adaptation) on the training split of the benchmark dataset.

D. Reason Extraction

For comments identified as expressing regret, a zero-shot Llama-3.1:70b model was used to categorize the primary reason into:

• Adverse Health Event
• Perceived Coercion
• Lack of Efficacy
• Shift in Beliefs
• Vague/Unspecified

3. Key Contributions

1. Novel Dataset: Creation of the first large-scale, politically diverse annotated benchmark for vaccine regret ($D_{bench}$).
2. Methodological Innovation: A hybrid pipeline combining NLI filtering with fine-tuned LLMs to handle nuance and scale.
3. Bias Mitigation: Demonstrated that a politically diverse annotation panel is crucial for labeling subjective, politicized content, revealing that political affiliation significantly impacts the classification of "Subject" but less so on factual "Vaccinated" status.
4. Granular Analysis: Differentiated between first-person and vicarious narratives and quantified the drivers of regret across different media ecosystems.

4. Key Results

Prevalence of Regret

• Overall Rate: Vaccine regret appears in only 1.1% of relevant comments, suggesting it is a persistent but minority narrative compared to the volume of total discourse.
• Source Disparity:
  • Mainstream News: 0.7% regret rate.
  • Influencer Channels: 1.9% regret rate (more than double).
  • Vaccine-Skeptic Influencers: 2.9% regret rate (nearly 3x higher than pro-vaccine influencers).
• Temporal Trend: Regret commentary among vaccine-skeptic influencers showed a rising trajectory, peaking during the "Mandate Peak" (Zone B) and remaining high, whereas mainstream news trends remained relatively stable.

Narrative Perspectives

• First-Person Dominance: 67.9% of regretful comments were first-person narratives, indicating that personal anecdotes are the primary driver of regret sentiment.
• Vicarious Relationships: In third-party accounts, Family Members (29.6%) were the most cited subjects, followed by Unspecified (28.5%), highlighting the role of intimate social networks in spreading health narratives.

Drivers of Regret

• Adverse Health Events: The most common reason overall (55.0%), dominating influencer channels (64.2%).
• Lack of Efficacy: A prominent secondary reason in mainstream news (26.9%).
• Perceived Coercion: Significantly more likely to be cited in first-person accounts (11.6%) compared to vicarious ones (4.9%), suggesting feelings of lost autonomy are a deeply personal driver.

Model Performance & Bias

• Pipeline Accuracy: The final pipeline achieved an Exact Match of 62.00% on the test set.
  • Regret F1: 0.82
  • Vaccinated F1: 0.87
  • Subject F1: 0.80 (lower due to the "Unspecified" class being harder to distinguish from irrelevant noise).
• Political Alignment: The model did not show systematic bias toward any political group when resolving annotator disagreements (alignment was statistically similar across Democrats, Republicans, and Independents).
• Inter-Annotator Agreement: Moderate agreement (Fleiss' Kappa ~0.45–0.51), confirming the subjective nature of the task. Political affiliation significantly influenced "Subject" classification ($p < 0.001$).

5. Significance and Implications

• Public Health Monitoring: The study provides a scalable, automated method for monitoring public sentiment and identifying the specific drivers of vaccine regret (e.g., side effects vs. mandates) in real-time.
• Targeted Communication: The stark difference in regret drivers between mainstream news (efficacy concerns) and influencer channels (adverse events/coercion) suggests that public health messaging must be tailored to specific online ecosystems.
• Trust Building: Understanding that "perceived coercion" is a major driver for first-person regret highlights the need for public health agencies to address autonomy concerns to maintain institutional trust.
• Limitations: The study relies on self-selected social media data (similar to VAERS), meaning it cannot infer causality or true population incidence rates. It is also limited to English-language YouTube comments.

In conclusion, the paper demonstrates that while vaccine regret is a small fraction of total discourse, it is highly concentrated in specific communities and driven by distinct, actionable factors. The proposed computational framework offers a robust tool for navigating this complex information landscape.