Technical Summary: What Helps Language Models Predict Human Beliefs: Demographics or Prior Stances?

Problem Statement

As Large Language Models (LLMs) become ubiquitous, their ability to understand and reason about human beliefs has significant implications for privacy, personalized persuasion, and stereotyping. While beliefs form complex, interrelated networks shaped by logical dependencies and social processes, it remains unclear how off-the-shelf LLMs capture this correlational structure. Specifically, when predicting an individual's stance on a new proposition, it is uncertain whether demographic information (who they are) or prior expressed stances (what else they believe) provides more predictive value, or if a combination of both is optimal. Existing benchmarks often focus on narrow sociopolitical contexts or rely on fine-tuning, leaving a gap in understanding how base models leverage social information without task-specific adaptation.

Methodology

The study evaluates the belief prediction capabilities of nine open-weight LLMs (ranging from 3B to 14B parameters) using a dataset derived from Debate.org (DDO), spanning October 2007 to September 2018.

Data Construction

• Source: The DDO dataset provides user demographics, debate topics with opposing arguments, and user stance votes ("agree" or "disagree").
• Belief Extraction: Stance votes were combined with debate titles to generate structured belief statements (e.g., "I agree with the following: [Topic]"). GPT-4 was used to filter out titles that could not be interpreted as coherent human beliefs.
• Preprocessing: Users with fewer than five total belief statements were excluded. Data was chronologically split for each user: the oldest 80% served as context beliefs (training), and the remaining 20% served as test beliefs. To prevent data leakage, context beliefs with a cosine similarity >0.8 to test beliefs (via Sentence-BERT) were removed.
• Final Dataset: 119,118 unique belief statements from 5,775 unique users.

Experimental Design

The task was framed as binary classification (predicting "agree" or "disagree") under four distinct informational conditions:

1. Blind: No user information (relying solely on population-level priors).
2. Demographics Only: User attributes (age, gender, ethnicity, income, political/religious ideology, etc.).
3. Context Beliefs Only: The user's prior expressed stances.
4. Combined: Both demographics and context beliefs.

Implementation

• Models: Nine open-weight models were evaluated, including DeepSeek R1 Distill variants, Gemma 3, Llama 3.1/3.2, Mistral, Phi-4, and Qwen3.
• Prompting: The study utilized DSPy for structured prompt engineering, incorporating Chain-of-Thought (CoT) reasoning to generate intermediate steps before the final binary prediction.
• Evaluation: Performance was measured using Macro F1 scores. Results were aggregated via two methods: the mean of individual model scores and a majority vote (ensemble) across all models.

Key Results

The study reveals that while both demographics and prior beliefs improve prediction over a blind baseline, their relative utility varies significantly across belief domains. The results categorize belief domains into three distinct groups based on the optimal input condition:

1. Group 1 (Demographics + Beliefs): Categories such as Religion, Politics, and Society perform best when both information types are provided. For these identity-linked domains, performance generally improves in the order: Blind → Demographics → Context Beliefs → Combined.
2. Group 2 (Context Beliefs Only): Idiosyncratic domains like Sports, Entertainment, Cars, and Places/Travel achieve peak performance with context beliefs alone. In these categories, adding demographic information often degrades performance compared to using prior beliefs alone, though demographics still outperform the blind baseline.
3. Group 3 (Demographics Only): A smaller set of categories performs best with demographics alone, though the ordering of other conditions varies.

Quantitative Findings:

• Overall Performance: The combined condition yielded the highest Macro F1 score for 8 out of 9 models. The best-performing model, Phi-4, achieved a score of 63.5% in the combined condition.
• Relative Gains: On average, demographics improved the blind baseline by 6 percentage points, context beliefs by 6.5 points, and the combined condition by 7.8 points.
• Non-Monotonicity: Adding a second information source does not always improve accuracy. In some cases (e.g., "Funny" or "Music"), demographics performed worse than the blind baseline, and adding demographics to context beliefs reduced accuracy in specific domains.
• Ensemble Performance: Majority-vote aggregation consistently outperformed mean aggregation, suggesting that different models make complementary errors.

Significance and Claims

The paper claims to advance the understanding of how off-the-shelf LLMs reason about human beliefs without task-specific fine-tuning. The key contributions and implications include:

• Domain-Specific Reasoning: The findings suggest that LLMs leverage different types of social information depending on the nature of the belief. Beliefs tightly coupled with social identity (Politics, Religion) rely heavily on demographic priors, whereas idiosyncratic preferences (Sports, Entertainment) rely more on individual history (prior stances).
• Coherent Belief Systems: The variation in accuracy supports the theoretical concept of coherent belief systems. LLMs perform best in domains where beliefs form densely interconnected, mutually reinforcing networks (e.g., Religion, Science) and struggle more where associations are looser.
• Limitations of Current Models: Despite improvements, absolute accuracy remains moderate. The paper attributes this ceiling to noise in user data (inconsistency between recorded stances and true beliefs), temporal gaps in the dataset (2007–2018), and inherent biases in training corpora that may privilege majority viewpoints.
• Ethical Considerations: The ability to infer attitudes from limited cues carries risks regarding privacy and stereotyping. The authors caution that overconfident inferences could misrepresent individuals, emphasizing the need for transparency about uncertainty and clear constraints on the deployment of belief prediction tools.

The study concludes that while LLMs can effectively leverage social information to predict beliefs, their performance is not uniform and is deeply influenced by the structural coherence of the belief domain and the specific type of social cue provided. Future work is needed to establish human baselines and explore the directionality of belief inference across different domains.