Technical Summary: Sparks of Rationality

Problem Statement

Large Language Models (LLMs) are increasingly deployed as decision engines in high-stakes domains such as hiring, healthcare, and economic planning. While classical Rational Choice and Expected-Utility Theories (EUT) model agents as maximizing stable utility functions, human judgment is characterized by a balance between rational deliberation and emotion-driven biases (heuristics, dual-process dynamics, and affective shifts). A critical gap exists in understanding whether LLMs, particularly those with "reasoning" capabilities, exhibit analogous patterns of rationality and emotion-induced bias. Furthermore, it is unclear how affective steering methods interact with reasoning processes to alter decision behavior, and whether these models can be reliably steered to simulate human-like judgment or maintain normative consistency.

Methodology

The authors evaluate multiple LLM families (including Qwen, Llama, and Olmo variants) across two primary dimensions:

1. Rationality Benchmarks: The study tests compliance with four canonical axioms of rational choice (Completeness, Transitivity, Continuity, and Independence) derived from EUT.
2. Decision Domains: The models are evaluated on tasks operationalizing classical behavioral economics and social psychology benchmarks where emotions are known to shape judgment. These domains include:
   • Choices under Risk and Uncertainty: Risk seeking, ambiguity aversion, loss aversion, and the endowment effect.
   • Judgment and Belief Formation: Heuristic stereotyping and framing/persuasion effects.
   • Norm Enforcement and Moral Evaluation: Moral condemnation and legal blame attribution.
   • Prosocial Preferences: Fairness (Ultimatum Game), altruism (Dictator Game), and welfare assistance.

Steering Mechanisms:
To probe affective distortions, the study employs two distinct emotion-steering methods:

• In-Context Priming (ICP): Uses emotional personas and vignettes in the prompt to induce the model to simulate a specific feeling (e.g., "You are currently feeling fear").
• Representation-Level Steering (RLS): Injects low-rank vectors into the model's hidden states to encode emotional directions, altering internal processing without explicit textual cues.

The study compares Thinking Models (which generate explicit reasoning traces before answering) against Non-Thinking Models and analyzes how these modes interact with steering strengths (low, medium, high).

Key Contributions

1. Rationality Enhancement via Reasoning: The paper demonstrates that enabling "thinking" (deliberative reasoning) consistently increases compliance with rational-choice axioms across different model families and scales. Thinking models default to expected-value (EV) maximization, exhibiting near-linear probability weighting and weak loss aversion.
2. Divergent Steering Effects: The study identifies a qualitative difference between ICP and RLS:
   • ICP induces strong, directional shifts aligned with human behavior but often with extreme magnitudes that are difficult to calibrate (e.g., fear inducing near-zero gambling regardless of expected value). It maintains a neutral deliberative style while reasoning about the emotion.
   • RLS produces more graded, psychologically plausible patterns (e.g., risk aversion that remains sensitive to expected payoff) but with lower reliability and smaller effect sizes. It alters the internal tone, resulting in "emotional thinking."
3. Systematic Deviations: Even with reasoning enabled, LLMs exhibit systematic deviations from human behavior, including a reversed endowment effect (selling prices lower than buying prices), near-complete ambiguity aversion, and anomalies in temporal discounting.
4. Interaction of Reasoning and Affect: The findings suggest that the same deliberative mechanisms that improve rationality also amplify sensitivity to affective interventions. Reasoning models can "rationalize" induced affect, making them susceptible to steering that pushes them toward specific behavioral outcomes.

Key Results

• Axiom Compliance: Thinking models show significantly higher adherence to rationality axioms compared to non-thinking models. However, emotional steering (both ICP and RLS) causes modest decreases in axiom consistency, correlated with steering strength.
• Risk and Uncertainty:
  • Risk Seeking: Neutral thinking models track EV boundaries closely. ICP fear drives near-zero gambling (exaggerated risk aversion), while RLS fear produces graded, human-aligned shifts.
  • Ambiguity Aversion: Models exhibit strong ambiguity aversion. Sadness steering via RLS reduces this aversion (moving toward neutrality), whereas ICP often fails to replicate the human benchmark or increases aversion.
  • Loss Aversion: Neutral models show little classical loss aversion. Fear via RLS increases loss sensitivity, while ICP often leads to universal rejection of gambles.
• Endowment Effect: Models consistently display a reversed endowment effect (WTA < WTP), contrary to human behavior. Emotional steering does not reliably correct this sign.
• Temporal Discounting: Thinking models show inconsistent temporal valuation structures compared to normative discounting, with some exhibiting non-monotonic behavior.
• Moral and Social Judgments:
  • Moral Condemnation: Disgust steering increases overall condemnation while preserving the correlation between perceived harm and condemnation.
  • Legal Blame: Anger steering increases blame and punishment severity, consistent with human patterns, particularly when thinking is enabled.
  • Fairness/Altruism: Sadness increases rejection of unfair offers (Ultimatum Game) and increases altruistic giving (Dictator Game), aligning with human data.
• Controllability: ICP shifts are often saturated and hard to moderate via prompt wording. RLS offers more controllable, dose-dependent shifts but can degrade task sensitivity at high injection strengths.

Significance and Claims

The paper positions itself as a controlled testbed for auditing LLM decision-making under both baseline and affective perturbations. Its primary significance lies in:

1. Safety and Alignment: Highlighting a tension between alignment for human simulation (where emotional biases may be desirable) and alignment for consistent, unbiased decision-making (where susceptibility to steering is a safety concern). The authors note that reasoning models can rationalize induced affect, potentially making them vulnerable to manipulation in high-stakes workflows.
2. Evaluation Framework: The work provides a reproducible, parameterized framework for measuring specific behavioral mechanisms (e.g., EV sensitivity, probability weighting, utility curvature) rather than just surface-level outputs. This allows for the separation of systems optimized for normative decision support from those intended for human simulation.
3. Limitations of Steering: The study challenges the assumption that steering can reliably shape decision behavior beyond surface style without engaging the model's reasoning mode. It suggests that while steering can induce human-aligned effects, the magnitude and stability of these effects vary significantly by method and model architecture.

The authors conclude that while reasoning improves rationality, it does not automatically confer human-like preference representations, and that affective steering interacts complexly with deliberation, necessitating careful evaluation before deploying LLMs in consequential decision systems.