Replicating Human Motivated Reasoning Studies with LLMs

Technical Summary: Replicating Human Motivated Reasoning Studies with LLMs

Problem Statement
Motivated reasoning is a well-documented human phenomenon where individuals process information to either arrive at accurate beliefs (accuracy motivation) or to reach desired, predetermined conclusions (directional motivation). While recent work suggests Large Language Models (LLMs) can exhibit human-like behaviors when assigned specific personas (e.g., political affiliation), it remains unclear whether base LLMs—without persona induction—are susceptible to motivational manipulations in the same way humans are. This uncertainty challenges the validity of using LLMs as proxies for human opinion formation and argument assessment in social science research.

Methodology
The authors replicated four prior political science studies on motivated reasoning (Bolsen et al., 2014; Bolsen and Druckman, 2015; Druckman et al., 2013; Mullinix, 2016) using ten base LLMs across five model families (OpenAI, Google, Anthropic, Mistral, and Qwen).

• Experimental Design: The study utilized a "one-shot" approach, providing models with the same information, motivation manipulations (e.g., directional cues like party endorsement vs. accuracy cues like evenhanded evaluation), and questions as the original human participants. No personality traits or demographics were induced in the primary experiments to isolate the effects of motivational cues.
• Sampling: To manage computational costs while preserving study structure, the authors downsampled the original human participant counts. They selected the smallest condition sample size from the original studies (44) as the baseline, generating proportional numbers of responses for other conditions, resulting in 3,574 total generations across all studies.
• Evaluation Metrics: The authors compared LLM outputs to human baselines using:
  • Kendall's Tau: To measure correlation between average LLM and human responses across conditions.
  • Sign Accuracy: To determine if LLMs correctly predicted the direction of opinion change relative to a control condition.
  • Argument Assessment: Evaluating the correlation between LLMs' perceived argument strength (pro vs. con) and their final opinions.
  • Non-Response Analysis: Measuring "opt-out" rates (instances where models refused to answer) using Chi-Square tests to identify if specific information manipulations triggered non-response.
• Robustness Checks: Additional experiments included prompting models to provide reasoning (Chain-of-Thought) and, in a subset of Study 2, inducing demographic personas to test if these factors altered the findings.

Key Contributions

1. Base LLMs Do Not Mimic Human Motivated Reasoning: The study provides evidence that base LLMs, without persona induction, do not replicate the directional or accuracy-driven shifts in opinion observed in humans when exposed to motivational cues.
2. Misalignment in Argument Assessment: When provided with policy pros and cons, LLMs often display misalignment between their assessment of argument strength and their resulting opinions, failing to emulate the human tendency to weigh arguments consistently with their final stance.
3. Selective Non-Response: The research identifies that information manipulations significantly affect a model's likelihood to "opt-out" of responding. This non-response is not uniform; it varies based on the specific study conditions and information provided, limiting the reliability of LLMs as survey proxies in certain settings.
4. Persona vs. Base Behavior: The findings suggest that the "motivations" previously observed in LLMs are heavily dependent on explicit persona induction rather than an inherent capacity for motivated reasoning.

Results

• Correlation with Humans: There were no strong positive correlations (Kendall's Tau) between any model's average responses and human averages across the four studies. Most correlations were weak, negative, or insignificant.
• Sign Accuracy: LLMs demonstrated low accuracy in predicting the direction of opinion change relative to control conditions. For Study 1 and Study 3, accuracy scores were generally low, with only a few models (e.g., Mistral 8x7B in Study 1) showing moderate accuracy.
• Argument Strength: Models failed to consistently assess argument strength in a manner resembling humans. While some models showed moderate correlations in Study 3, all models showed weak or insignificant correlations in Study 4.
• Opt-Out Behavior: Non-response rates were significant and non-uniform. For instance, in Study 1, motivation manipulations caused high non-response rates in Gemini 2.5 Flash and Claude 4.5 Haiku, whereas in Study 3, control conditions triggered high non-response in multiple models. This suggests models may abstain from responding to controversial or specific informational cues rather than simulating a human opinion.
• Robustness: Even when prompted to provide reasoning or when given demographic personas, LLMs continued to fail to mimic human motivated reasoning outcomes, reinforcing the primary findings.

Significance and Claims
The paper claims that these findings are critical for researchers in social science and computer science who utilize LLMs as survey proxies or for predicting public opinion. The results indicate that LLMs may not be accurate forecasters of public opinion when motivated reasoning is a factor, particularly in controversial or polarized settings where models may simply abstain from responding.

Furthermore, the work highlights limitations for the AI and NLP communities regarding tasks such as automated argument assessment. The authors suggest that current base LLMs struggle to emulate human-like reasoning processes, and that relying on them for tasks like opinion replication or argument evaluation requires caution. The study emphasizes the importance of understanding the gap between base LLM behavior, persona-steered LLM behavior, and actual human behavior, noting that simply prompting for reasoning or adding demographics does not bridge this gap. Future work should focus on developing advanced systems (e.g., using few-shot prompting or hybrid human-LLM assessments) to improve reliability in these domains.