Technical Summary: Synthetic Socratic Debates

Problem Statement

As Large Language Models (LLMs) are increasingly deployed in morally sensitive domains, there is a critical need to understand how assigned persona traits influence their moral reasoning and persuasive behaviors. While previous research has examined LLM moral judgments, it has largely relied on synthetic dilemmas with high annotator agreement or manipulated single persona traits in isolation. This leaves a gap in understanding how rich, intersecting identities (e.g., the combination of political ideology, personality, and demographics) shape moral judgment and how these identities interact during multi-turn moral debates. Furthermore, existing studies often overlook the dynamic evolution of persuasion strategies and consensus formation in open-ended, multi-agent settings.

Methodology

The authors present a large-scale study utilizing a 6-dimensional persona space to simulate structured AI-AI debates over real-world moral dilemmas.

1. Dataset and Scenarios:
The study utilizes 131 interpersonal moral dilemmas drawn from the SCRUPLES ANECDOTES corpus. These scenarios were selected for their high controversy and significant human annotation disagreement, ensuring they lack clear-cut answers. The dilemmas focus on relationship-based conflicts (e.g., pet ownership vs. child safety, weight-related comments in relationships).

2. Persona Modeling:
The researchers define a 6-dimensional persona space, sampling 500 unique personas without replacement from the Cartesian product of the following attributes:

• Age: 20, 30, 40, 50, 60.
• Gender: Male, Female, Non-binary.
• Country: China, United States, Brazil, France, Nigeria, India.
• Social Class: Lower, Middle, Upper.
• Political Ideology: Libertarian Left, Libertarian Right, Authoritarian Left, Authoritarian Right.
• Big Five Personality: High/Low levels of Openness, Conscientiousness, Extraversion, Agreeableness, and Neuroticism.

3. Experimental Setup:

• Models: The study evaluates four LLMs: GPT-4o, Claude-3.5-Sonnet, LLaMA-4-Maverick, and Qwen3-235B-A22B.
• Phase 1 (Single-Turn Judgment): Agents with assigned personas provide initial moral judgments on the dilemmas, rated on a 5-point Likert scale (1 = strongly blame the author, 5 = strongly blame others).
• Phase 2 (Multi-Turn Debate): Pairs of agents with conflicting initial stances engage in debates of up to 5 turns. Agents alternate arguments, update their Likert ratings after each turn, and are prompted to explain their reasoning. The debate terminates upon consensus or reaching the turn limit.
• Evaluation Metrics:
  • Moral Judgment: Likert scores and Moral Foundations Theory (MFT) dimensions (Care, Fairness, Loyalty, Authority, Sanctity, Liberty).
  • Persuasion Dynamics: Consensus rate, self-alignment rate (whether the final answer matches the initial one), and efficiency (turns to consensus).
  • Rhetorical Strategy: Analysis of Aristotle's modes of persuasion (Ethos, Pathos, Logos) using an LLM-as-a-judge to score the presence of these appeals in agent responses.

Key Contributions and Results

1. Persona Impact on Moral Judgment (RQ1):

• Systematic Bias: All models exhibited a consistent bias toward blaming the author of the dilemma compared to human judgments (which centered at neutrality), though the magnitude varied by model.
• Dominant Predictors: Political ideology and personality traits were the strongest predictors of moral stances.
  • Ideology: Libertarian-left personas were the most lenient, while authoritarian-right personas were the harshest. Conservatives emphasized binding foundations (loyalty, authority, sanctity), whereas liberals prioritized individualizing foundations (care, fairness).
  • Personality: High openness and agreeableness correlated with less blame, while high conscientiousness correlated with more blame.
• Demographics: Age, gender, and country showed significant but generally weaker effects compared to ideology and personality. For instance, older personas tended to assign more blame, and female personas blamed the author less than male personas.

2. Persuasion Dynamics in Debates (RQ2):

• Consensus and Efficiency: Persuasive success varied significantly by persona. Libertarian personas, particularly those with high openness and agreeableness, reached consensus more efficiently and frequently. Conversely, authoritarian personas engaged in longer debates with lower consensus rates.
• Rhetorical Shifts:
  • Confidence: Model confidence (measured by log probabilities) consistently increased throughout the debate, regardless of whether the agent changed its stance.
  • Persuasion Modes: The intensity of all rhetorical appeals (Ethos, Pathos, Logos) generally diminished over the course of the debate. Pathos (emotional appeals) showed the largest decrease, suggesting a shift toward more tempered, less affective argumentation as debates progressed.
• Persona-Specific Strategies: Libertarian-left personas utilized more emotional appeals (Pathos), while authoritarian-right personas relied more heavily on authority-based reasoning (Ethos).

3. Cross-Model Consistency:
Findings were largely consistent across GPT-4o, Claude-3.5-Sonnet, and LLaMA-4-Maverick. Qwen3-235B-A22B displayed notable differences, often shifting blame to others rather than the author, highlighting that specific model architectures may encode distinct biases.

Significance and Claims

The paper claims to be the first large-scale study to systematically examine the intersectionality of multiple persona dimensions in AI moral reasoning and persuasion. Its significance lies in:

1. Validating Psychological Theories: The results mirror established findings in human psychology and cultural studies, such as the link between political ideology and moral foundations, and the influence of personality traits on moral cognition. This suggests LLMs can internalize and express psychologically grounded persuasion strategies shaped by assigned personas.
2. Dynamic Reasoning Insights: By moving beyond static, single-turn judgments to multi-turn debates, the study reveals how personas influence not just what models decide, but how they argue, adapt, and reach (or fail to reach) consensus.
3. Evaluation Frameworks: The work underscores the necessity for persona-aware evaluation frameworks. It demonstrates that moral reasoning in LLMs is not neutral; assigned personas introduce systematic variations that can amplify biases or alter persuasive dynamics.
4. Ethical Implications: The authors note that these findings offer actionable insights for designing LLMs sensitive to diverse value systems. They also highlight risks, such as the potential for malicious actors to exploit high-influence personas to sway public opinion, advocating for transparency and guardrails in the deployment of persuasive conversational agents.

The authors maintain a modest stance, acknowledging limitations such as the dataset's origin (Reddit/WEIRD demographics), the use of English-only prompts despite diverse country personas, and the dyadic nature of the debates, which may not fully capture complex group dynamics.