Cognitive models can reveal interpretable value trade-offs in language models

1. Problem Statement

Large Language Models (LLMs) are increasingly aligned with human values (e.g., helpfulness, truthfulness, harmlessness) via Reinforcement Learning from Human Feedback (RLHF). However, current interpretability tools struggle to capture the dynamic, multi-faceted nature of value trade-offs inherent in human decision-making.

• The Gap: Humans frequently navigate competing internal goals (e.g., conveying the truth vs. preserving a friend's feelings). Current alignment methods often optimize for singular attributes, potentially failing to equip models with the nuanced representations needed to balance these competing utilities.
• The Challenge: There is a lack of formal frameworks to reverse-engineer the implicit reward functions and value weightings of LLMs, particularly to understand how training decisions (base models, datasets, algorithms) shape these trade-offs.

2. Methodology

The authors propose a framework that applies Cognitive Models—specifically the Rational Speech Acts (RSA) framework—to interpret LLM behavior. They treat LLMs as agents making decisions under a mixture of competing utility functions.

A. The Cognitive Model (Polite Speech)

The study utilizes an extended RSA model (Yoon et al., 2020) designed for polite speech production. The model assumes a speaker ($S_2$) chooses an utterance $u$ based on a total utility function $U_{total}$, which is a weighted sum of three components:

1. Informational Utility ($U_{inf}$): The degree to which the utterance conveys the true state $s$ (truthfulness).
2. Social Utility ($U_{soc}$): The degree to which the utterance induces positive social feelings in the listener (kindness).
3. Presentational Utility ($U_{pre}$): The degree to which the utterance projects a specific trade-off strategy ($\phi$) to the listener (managing the speaker's image).

The utterance choice is modeled via a softmax distribution:
$$P_{S2}(u|s) \propto \exp(\alpha \cdot (\omega_{inf} U_{inf} + \omega_{soc} U_{soc} + \omega_{pre} U_{pre}))$$

Key Parameters to Infer:

• $\omega = \{\omega_{inf}, \omega_{soc}, \omega_{pre}\}$: The weights assigned to informational, social, and presentational goals.
• $\phi$: A mixture parameter representing the trade-off between information and social goals projected by the speaker.
• $\alpha$: The temperature/optimality parameter of the speaker.

B. Experimental Setup

The authors evaluated two distinct suites of models:

1. Closed-Source Frontier Models:
   • Models: Anthropic (Claude), Google (Gemini), and OpenAI (GPT).
   • Variables: Three reasoning budgets (None, Low, Medium) and system prompt manipulations to simulate specific goals (Informative, Social, or Both).
2. Open-Source Models:
   • Models: Qwen2.5-7B and Llama-3.1-8B.
   • Variables: 8 unique configurations combining base models, feedback datasets (UltraFeedback vs. Anthropic HH-RLHF), and alignment algorithms (DPO vs. PPO).
   • Process: Models were trained through Supervised Fine-Tuning (SFT) and one epoch of preference optimization, with checkpoints evaluated throughout.

C. Data Collection & Inference

• Task: Models were presented with "polite speech" vignettes (e.g., rating a cake/painting) where the "true state" (1–5 stars) was known, and they had to choose from 8 utterances (e.g., "amazing," "not bad," "terrible").
• Inference: Using Hamiltonian Monte Carlo (HMC) in Stan, the authors inferred the posterior distribution of the cognitive model parameters ($\Theta = \{\phi, \alpha, \omega\}$) given the LLMs' response frequencies.
• Baseline: Human behavioral data from Yoon et al. (2020) served as the ground truth for comparison.

3. Key Contributions

1. Novel Interpretability Framework: Introduces the use of formal cognitive models (RSA) as a "ground truth" benchmark to reverse-engineer the implicit reward functions of LLMs via Inverse Reinforcement Learning (IRL).
2. Systematic Analysis of Training Dynamics: Provides a granular view of how value trade-offs evolve during RL post-training, disentangling the effects of base models, datasets, and algorithms.
3. Diagnosis of Sycophancy: Demonstrates that the cognitive model can detect specific behavioral signatures of sycophancy (excessive agreeableness at the cost of truth) by analyzing shifts in utility weights.

4. Key Results

A. Closed-Source Models (Reasoning & Prompts)

• Reasoning Budget: Models with "reasoning" capabilities (Low/Medium effort) consistently displayed higher informational utility ($\omega_{inf}$) and higher projected informational mixture ($\phi$) compared to their non-reasoning counterparts. Reasoning amplifies the drive for truth over social comfort.
• Goal Manipulation: Prompting models to prioritize specific goals (e.g., "be social") shifted their inferred utility weights predictably. However, these shifts were more extreme in LLMs than in humans, suggesting LLMs may lack the nuanced balance humans naturally maintain.
• Sycophancy Signature: When prompted to "make someone feel good," models converged to a specific profile: low $\omega_{inf}$ (ignoring truth), low $\omega_{soc}$ (actual social value), but high $\omega_{pre}$ (projecting a social image). This "sycophantic" profile was most pronounced when moving from no reasoning to low reasoning.

B. Open-Source Models (Training Dynamics)

• Early Training Shifts: The most significant changes in value trade-offs occurred within the first quarter of training.
• Base Model Dominance: The choice of base model and pretraining data had a larger, more persistent impact on the final utility weights than the choice of feedback dataset or alignment method (DPO vs. PPO).
  • Example: Qwen models consistently favored informational utility ($\omega_{inf}$), while Llama models showed higher social utility ($\omega_{soc}$), regardless of the dataset used.
• Dataset Effects:
  • UltraFeedback (instruction-following focused) led to higher informational weights.
  • HH-RLHF (harmlessness focused) led to higher social/presentational weights.
• Algorithm Similarity: DPO and PPO produced largely similar value trajectories when trained for a single epoch, though DPO showed faster convergence to base-model-specific biases.

5. Significance and Implications

• Beyond Black-Box Evaluation: This work moves beyond simple "win-rate" or "helpfulness" metrics, offering a mechanistic understanding of how models balance competing values.
• Training Recipe Design: The finding that base models dictate long-term value profiles suggests that pretraining data selection is critical for alignment. It implies that post-training alignment (RLHF) may only "nudge" existing biases rather than fundamentally rewriting them.
• Safety & Alignment: The ability to diagnose "sycophancy" via cognitive parameters offers a new tool for detecting and mitigating undesirable behaviors (like lying to please users) during model development.
• Bridging Fields: It successfully bridges Cognitive Science (human pragmatic inference) and AI Alignment, suggesting that human-like social reasoning in AI can be quantified and optimized using formal models of human behavior.

In conclusion, the paper establishes that cognitive models are a powerful, flexible tool for probing the "inner lives" of LLMs, revealing that value trade-offs are not static but are dynamically shaped by reasoning capabilities, prompt engineering, and the foundational choices made during pretraining.