Technical skepticism and institutional mistrust drive COVID-19 vaccine refusal: a global LLM- assisted survey

Technical Summary: Technical Skepticism and Institutional Mistrust Drive COVID-19 Vaccine Refusal

Problem Statement
While vaccine hesitancy is widely studied using closed-ended survey instruments with predefined response categories, the nuanced motivations underlying specific vaccine refusal remain poorly characterized. Existing research often relies on researchers' a priori assumptions, potentially missing concerns that fall outside traditional medical or religious justifications. Furthermore, the rapid development of COVID-19 vaccines and the political polarization surrounding mandates have reshaped the drivers of hesitancy, creating a need to understand specific technical objections and the erosion of trust in public health institutions. The study aims to identify and categorize the specific concerns driving COVID-19 vaccine refusal by analyzing free-text survey responses from a global cohort.

Methodology
The study, titled PROVES (Participant Responses on Vaccination and Evolving Skepticism), was a cross-sectional, self-reported online survey conducted between September 3, 2022, and April 16, 2024.

• Sampling: Participants were recruited via exponential non-discriminative snowball sampling across 95 countries. Inclusion criteria required adults (18+) with working English proficiency.
• Data Collection: The survey collected demographic data and reasons for vaccination status. Crucially, unvaccinated participants were asked to provide free-text explanations for their refusal in an "other (specify)" field, alongside predefined options for medical conditions and religious beliefs.
• LLM-Assisted Classification: The study utilized the Mistral Large Language Model (LLM) to classify 7,254 free-form responses from 3,756 unvaccinated individuals. A custom pipeline was developed using the ollama Python library.
  • Thematic Subcategories: Seven predefined subcategories were established a priori: (1) safety concerns (e.g., mRNA as experimental/gene therapy), (2) institutional mistrust, (3) autonomy/freedom of choice, (4) natural immunity, (5) perceived lack of need, (6) prior vaccine injury/allergy, and (7) pregnancy-related concerns.
  • Human Verification: All LLM-generated classifications were subsequently reviewed and manually curated by the research team to ensure accuracy, employing a hybrid approach of automated scalability and human contextual judgment.
• Statistical Analysis: Data were de-identified using SHA-1 hashing. Comparisons between subgroups (sex and vaccination status) utilized t-tests, Mann-Whitney U tests, and Shapiro-Wilk tests for normality.

Key Results

• Cohort Demographics: Of 7,541 participants from 95 countries, 6,916 provided sufficient data for analysis. The majority (80.07%) were unvaccinated. The cohort was highly educated, with over 87% holding a college degree. The geographic distribution was heavily weighted toward the USA (61.7%), Canada, the UK, and Australia.
• Primary Drivers of Refusal: The LLM analysis of free-text responses revealed that the most frequently cited reason for refusal was safety concerns regarding mRNA technology (3,033 responses; 41.8%). Respondents characterized mRNA vaccines as experimental, gene therapy, or lacking sufficient clinical data.
• Secondary Drivers: Institutional mistrust was the second most common reason (18.7%), followed by autonomy and freedom of choice (9.5%). Other categories included natural immunity (4.4%), perceived lack of need (1.2%), prior vaccine injury (0.7%), and pregnancy-related concerns (0.3%).
• Demographic Nuances:
  • Education: Contrary to common associations between hesitancy and lower education, this cohort was highly educated. However, vaccinated participants were more likely to hold postgraduate degrees (18.0% vs. 12.7%).
  • "Zero-Dose" Status: Only two participants reported receiving no vaccines since childhood, indicating that the majority of this hesitant cohort is not broadly "anti-vaccination" but rather specifically skeptical of COVID-19 vaccines.
  • Perceived Mortality: Unvaccinated participants were significantly more likely to report knowing someone who died after receiving a vaccine compared to vaccinated participants, suggesting exposure to different information ecosystems.

Key Contributions

1. Granular Categorization of Hesitancy: By utilizing free-text analysis powered by LLMs, the study moves beyond binary or limited-choice surveys to identify specific technical objections (e.g., the "gene therapy" narrative) rather than general anti-vaccination sentiment.
2. Challenging Educational Assumptions: The findings challenge the notion that vaccine hesitancy is primarily driven by informational deficits or low educational attainment, as the hesitant cohort was highly educated.
3. Hybrid Classification Framework: The paper demonstrates a methodology combining LLM classification with manual curation to process large volumes of qualitative data, offering a scalable approach for future public health research.
4. Distinction Between Ideology and Specific Skepticism: The data suggests that for this cohort, refusal is driven by specific technical concerns and mistrust of institutions rather than a blanket rejection of all vaccination.

Significance and Claims
The authors claim that COVID-19 vaccine refusal in this global, self-selected cohort was driven primarily by specific technical concerns about mRNA vaccine safety and institutional mistrust, rather than general anti-vaccination ideology, religious objection, or low educational attainment.

The study concludes that these findings suggest transparent communication addressing specific safety concerns (such as the nature of mRNA technology and long-term safety data) may be more effective than enforcement-based approaches or mandates, which appear to have hardened resolve rather than increased uptake. The authors emphasize that because the vast majority of this hesitant cohort has a history of vaccination (only two were "zero-dose"), they remain within reach of the healthcare system, and the patient-provider relationship should be prioritized for restoration and education.

Limitations Acknowledged
The paper explicitly notes several limitations affecting generalizability:

• Sampling Bias: The snowball sampling method resulted in a cohort that was 80% unvaccinated, exceeding population baselines, and was self-selected.
• Language and Geography: The survey was conducted only in English, limiting representation of non-Anglophone populations, and was heavily skewed toward high-income countries (USA, Canada, UK, Australia).
• LLM Reliability: While human-reviewed, formal inter-rater reliability metrics (e.g., Cohen's kappa) between the LLM and human reviewers were not calculated.
• Self-Reported Data: All data, including infection history and perceptions of vaccine-associated mortality, were self-reported and subject to recall bias.
• Temporal Evolution: The study spanned 19 months during which pandemic conditions and public discourse evolved, but results were not formally stratified by time of response.