Belief-Sim: Towards Belief-Driven Simulation of Demographic Misinformation Susceptibility

The paper introduces BeliefSim, a framework that leverages psychology-informed belief profiles to successfully simulate demographic variations in misinformation susceptibility using Large Language Models, achieving up to 92% accuracy.

The Gist

Imagine you are trying to predict who is most likely to believe a fake news story.

In the past, researchers tried to guess this by looking at demographics (who the person is). They thought, "Maybe older people believe more fake news," or "Maybe people with less education are more susceptible." It's like trying to guess someone's favorite movie just by knowing their age and zip code. Sometimes it works, but often it's a wild guess that leads to stereotypes.

This paper introduces a new, smarter way to simulate human behavior using AI. They call it Belief-Sim.

Here is the core idea, explained through a simple analogy:

The "Personality Profile" vs. The "Resume"

Think of a person's Demographics (Age, Gender, Location) as their Resume. It lists the facts about them, but it doesn't tell you how they think.

Think of a person's Beliefs (Do they trust science? Are they worried about the future? Do they value tradition?) as their Personality Profile. This tells you how they process information.

The authors argue that if you want an AI to act like a specific group of people, giving it the Resume isn't enough. You have to give it the Personality Profile.

How They Did It (The Recipe)

The researchers built a system called Belief-Sim to test this. Here is how they cooked it up:

1. The Ingredients (The Taxonomy): They created a "menu" of 7 different types of beliefs, like a psychological menu.

   • Worldview: How you see your place in the world.
   • Trust: Who do you trust? (Scientists? The government? Your neighbor?)
   • Thinking Style: Do you think logically, or go with your gut?
   • Conspiracy: Do you think there are secret plots everywhere?
   • Values: What is right and wrong to you?
   • Emotions: Are you easily scared or angry?
   • Shortcuts: Do you believe things just because you've heard them before?

2. The Simulation (The AI Chef): They used Large Language Models (LLMs)—the same kind of AI that writes essays and chats with you.

   • Old Way: They told the AI, "Pretend you are a 65-year-old woman living in a rural area."
   • New Way (Belief-Sim): They told the AI, "Pretend you are a 65-year-old woman living in a rural area who deeply trusts science, is very worried about the economy, and prefers to think things through logically before believing news."

3. The Test: They fed the AI real news headlines (some true, some fake) and asked, "Is this true or false?" They then compared the AI's answer to what real humans actually said.

The Big Discovery

The results were surprising and very clear:

• Demographics alone are weak. Just telling the AI the person's age or gender didn't help much. In fact, it sometimes made the AI worse at guessing, because it started relying on stereotypes (e.g., "Old people must believe this").
• Beliefs are the secret sauce. When the AI was given the "Personality Profile" (the beliefs), it became incredibly accurate—up to 92% accurate in some cases.
• The "Belief" matters more than the "Person." The study found that what you believe is a much stronger predictor of whether you'll fall for fake news than who you are.

The "Two-Step" Trick (BAFT)

The researchers also found a clever way to train the AI so it doesn't get confused. Imagine you are teaching a student:

• Step 1: First, teach the student how different groups of people generally think (using survey data).
• Step 2: Then, teach the student how to apply those thoughts to specific news stories.

They call this BAFT (Belief-Adapter Fine-Tuning). It's like building a solid foundation of "how people think" before asking the AI to solve specific problems. This stopped the AI from cheating by just memorizing stereotypes.

Why This Matters

This isn't just about making AI smarter; it's about saving us from fake news.

If we want to stop misinformation, we can't just target people based on their age or location. We need to understand their beliefs.

• If someone believes in conspiracies, they need a different kind of fact-check than someone who just doesn't trust the media.
• If we know why a group believes a lie (e.g., because they are scared or because they distrust science), we can design better messages to stop it.

In a nutshell: To understand why people believe fake news, stop looking at their ID card (demographics) and start listening to their inner voice (beliefs). The AI learned this, and now we can too.

Technical Summary

Here is a detailed technical summary of the paper "Belief-Sim: Towards Belief-Driven Simulation of Demographic Misinformation Susceptibility."

1. Problem Statement

Misinformation poses a significant societal threat, but susceptibility to it varies widely across demographic groups (e.g., age, gender, education). While Large Language Models (LLMs) are increasingly used to simulate human behavior, current approaches often rely on coarse demographic personas or broad political ideologies. These methods suffer from two main limitations:

1. Insufficient Predictive Power: Demographics alone are often poor predictors of misinformation susceptibility because they do not capture the underlying psychological mechanisms (beliefs) driving judgment.
2. Stereotype Reliance: Models conditioned solely on demographics tend to learn spurious correlations and stereotypes rather than genuine causal factors, leading to "shortcut learning" where the model guesses based on demographic labels rather than reasoning.

The paper addresses the gap in simulating demographic-aware misinformation susceptibility by treating beliefs as the primary driving factor, rather than demographics alone.

2. Methodology

The authors propose Belief-Sim, a framework that constructs demographic belief profiles using psychology-informed taxonomies and survey priors. The methodology is divided into three main components:

A. Belief Taxonomy Construction

The authors define a structured belief taxonomy comprising seven core dimensions derived from psychological and cognitive science research:

1. Worldview & Identity: Group-based and political identities.
2. Epistemic Trust: Beliefs about credible sources (science, media, institutions).
3. Cognitive Style: Analytical vs. intuitive reasoning preferences.
4. Conspiracy Mentality: Tendency to perceive hidden plots.
5. Morals & Values: Core cultural and ethical judgments.
6. Emotion-Related: Fear, risk perception, and threat sensitivity.
7. Heuristics: Reliance on mental shortcuts (e.g., familiarity).

B. Data Construction

The framework utilizes two types of belief data:

• Observed Beliefs: Individual-level judgments from existing datasets (PANDORA and MIST-1) where participants rated specific claims.
• Imputed Beliefs: Group-level priors inferred from the World Values Survey (WVS). These are mapped to the belief taxonomy using exploratory factor analysis to create demographic-specific belief distributions (e.g., the typical "Epistemic Trust" profile of a rural, older female).

C. Simulation Strategies

The paper evaluates two distinct modeling approaches:

1. Prompt-Based Conditioning (In-Context Learning):

   • The LLM is prompted with a persona defined by demographic attributes and a set of beliefs (either observed or imputed).
   • The model is instructed to judge news claims as "True" or "False" while staying consistent with the persona's prior beliefs.
   • Finding: Using modal responses (most frequent answers) from surveys as beliefs works better than using full probability distributions, likely due to reduced noise.

2. Post-Training Adaptation (BAFT - Belief-Adapter Fine-Tuning):

   • To avoid the inconsistencies of mixing individual observed beliefs with group-level imputed beliefs in prompts, the authors propose a two-phase fine-tuning approach:
     • Phase 1 (Belief Modeling): A lightweight adapter head is trained on WVS data to predict demographic-conditioned belief distributions. The base LLM is frozen.
     • Phase 2 (Susceptibility Fine-Tuning): A second lightweight head is trained on misinformation susceptibility data. It takes the semantic representation of the claim and the frozen belief embedding from Phase 1 as input.
   • This decouples belief learning from susceptibility prediction, preventing "label leakage" and reducing reliance on demographic shortcuts.

3. Key Contributions

1. Belief-Sim Framework: A novel simulation framework that integrates a structured psychological belief taxonomy with demographic data to model misinformation susceptibility.
2. Empirical Validation of Beliefs: Demonstrates that belief priors are a stronger predictor of susceptibility than demographics alone.
3. BAFT Architecture: Introduces a two-phase adapter fine-tuning method that effectively decouples belief modeling from task prediction, improving generalization and robustness.
4. Counterfactual Evaluation: Develops a rigorous evaluation suite including Flip-Rate Analysis to detect when models rely on demographic shortcuts (stereotypes) versus genuine belief-based reasoning.

4. Key Results

Performance Accuracy

• Belief vs. Demographics: Incorporating belief information consistently outperforms zero-shot and demographic-only baselines.
  • Accuracy: The best configuration (Imputed Beliefs + Demographics) achieved up to 92.6% accuracy on the PANDORA dataset and 72.6% on MIST.
  • Impact: Beliefs alone provided gains of 1–16 percentage points over demographic-only prompts.
• Model Comparison: Qwen-2.5-14B consistently outperformed Llama-3-8B and Mistral-7B across all settings.
• Fine-Tuning (BAFT): The two-phase BAFT approach significantly improved cross-study generalization (MIST-2 dataset), achieving 92.4% accuracy with Qwen, compared to a sharp drop in performance for 1-phase baselines.

Counterfactual & Shortcut Analysis

• Demographic Sensitivity: Models conditioned only on demographics showed high "flip rates" (changing predictions when demographics were swapped), indicating reliance on stereotypes.
• BAFT Robustness: The BAFT approach reduced flip rates to near 0% (even for Llama models which previously had high flip rates), proving that decoupling belief modeling reduces spurious demographic dependencies.
• Complementarity: Demographics provided only weak complementary value when belief information was present, suggesting that beliefs are the dominant signal.

Thematic Analysis

• Susceptibility is topic-dependent. Demographic gaps were negligible for obvious/opinion-based claims but significant for complex topics like science/health and government, where older and more educated groups generally showed higher accuracy.

5. Significance and Implications

• Beyond Demographics: The study proves that simulating human behavior requires modeling internal cognitive states (beliefs) rather than just external labels (demographics).
• Intervention Design: By identifying which specific belief dimensions (e.g., Epistemic Trust, Emotion) drive susceptibility in specific groups, future interventions can be more targeted and effective.
• Robust AI Simulation: The BAFT methodology offers a blueprint for creating LLM agents that simulate human behavior without reinforcing harmful stereotypes or relying on shortcut learning.
• Evaluation Standards: The paper establishes counterfactual flip-rate analysis as a critical metric for evaluating the fairness and robustness of demographic simulations in LLMs.

In conclusion, Belief-Sim demonstrates that treating beliefs as the primary driving factor allows LLMs to simulate demographic misinformation susceptibility with high accuracy and reduced bias, offering a scalable alternative to expensive human data collection for studying misinformation dynamics.