💬 NLP

Using Reinforcement Learning to Train Large Language Models to Explain Human Decisions

This paper demonstrates that employing reinforcement learning with outcome-based rewards to fine-tune pretrained large language models enables them to serve as dual-purpose cognitive models that simultaneously achieve strong predictive accuracy and generate interpretable natural language explanations for human risky choices.

Original authors: Jian-Qiao Zhu, Hanbo Xie, Dilip Arumugam, Robert C. Wilson, Thomas L. Griffiths

Published 2026-02-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jian-Qiao Zhu, Hanbo Xie, Dilip Arumugam, Robert C. Wilson, Thomas L. Griffiths

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Teaching AI to "Think Aloud"

Imagine you have a super-smart student (a Large Language Model, or LLM) who is incredibly good at guessing what you will do next. If you show them a tricky math problem, they can usually give you the right answer. But, if you ask how they got that answer, they might just say, "I just knew it," or give you a vague explanation that doesn't quite make sense.

In the world of psychology, scientists want models that don't just guess the answer (predict human behavior) but also explain why people make those choices (reveal the cognitive mechanism).

This paper asks: Can we teach an AI to not only predict human decisions but also to "think out loud" in a way that actually explains human psychology?

The Experiment: The "Risky Choice" Game

To test this, the researchers used a classic psychological game called "risky choice." Imagine you are offered two options:

Option A: You get $27 for sure.
Option B: You have a 90% chance of getting $25, but a 10% chance of getting $92.

Most people have to choose between a safe bet and a risky gamble. The researchers had a massive dataset of thousands of real humans making these choices. They wanted to train an AI to look at the options and say, "70% of people will pick Option A, and 30% will pick Option B."

The Three Training Methods

The researchers tried three different ways to teach the AI this task, like three different coaching styles:

The "Memorizer" (Standard Training): They showed the AI thousands of examples of questions and the correct answers. The AI learned to copy the pattern. It got good at guessing the percentages, but it didn't really learn why. It was like a student who memorized the answer key but couldn't explain the math.
The "Masked Memorizer" (Centaur-style): This is a fancy version of the first method where the AI is forced to focus only on the numbers in the answer, ignoring the rest of the text. It also got good at guessing, but still didn't generate a good explanation.
The "Coach with a Scoreboard" (Reinforcement Learning - RL): This is the star of the show. Here, the AI was allowed to write a "Chain of Thought" (a step-by-step reasoning process) before giving its final answer.
- The Twist: The AI didn't just get points for being right. It got points based on how close its final prediction was to what real humans actually did.
- The Result: To get the points, the AI realized it needed to "think" like a psychologist. It started writing things like, "People are risk-averse," or "They calculate the expected value." By trying to win the game (match human data), it accidentally learned to verbalize the psychological rules humans follow.

The Results: The AI Becomes a Psychologist

The researchers found that the "Coach with a Scoreboard" (RL) method was a game-changer for two reasons:

It Predicted Well: It was just as good at guessing what humans would choose as the other methods.
It Explained Well: The "thinking aloud" parts (the Chain of Thought) were actually high-quality psychological explanations.
- The Analogy: Imagine the other methods are like a weather app that says, "It will rain." The RL method is like a meteorologist who says, "It will rain because a cold front is colliding with warm air, creating low pressure."
- The AI started using real psychological concepts like Risk Aversion (people hate losing), Expected Value (doing the math on the odds), and Certainty Effect (people love a guaranteed win).

The "Smartness" Requirement

There was a catch, though. The researchers tried this same "Coach" method on a smaller, weaker AI model.

The Analogy: Imagine trying to teach a complex strategy to a toddler versus a college student. The toddler (the small model) couldn't figure out the math or the strategy, even with the coach. It just got confused and gave bad answers.
The Finding: The "Coach" method only worked because the base AI was already smart enough to understand the concepts. If the AI isn't smart enough to understand "Expected Value" to begin with, you can't train it to explain it.

The "Shape-Shifting" Test

To prove the AI wasn't just reciting memorized phrases, the researchers changed the rules of the game. Instead of using real human data, they fed the AI data generated by a robot that only cared about math (Expected Value).

The Result: The AI instantly changed its "thinking." It stopped talking about human fears and risk aversion and started talking purely about math and logic.
The Takeaway: This proved the AI wasn't just copying a script; it was actually adapting its reasoning to match the data it was being trained on. It was truly "learning" the rules of the specific game it was playing.

Summary

This paper shows that if you train a smart AI to predict human behavior using a reward system (like a video game score), it will naturally start to "think out loud" using the same psychological rules that humans use. It turns the AI from a simple guesser into a model that can explain why we make the choices we do. However, this only works if the AI is already smart enough to understand those concepts in the first place.

Technical Summary: Using Reinforcement Learning to Train Large Language Models to Explain Human Decisions

Problem Statement

A central challenge in cognitive modeling is developing systems that not only predict human behavior with high accuracy but also provide interpretable insights into the underlying cognitive mechanisms. While neural network models trained on large-scale behavioral data often achieve strong predictive performance, they typically function as "black boxes," offering limited theoretical understanding of the cognitive processes they capture. Although recent work (e.g., the Centaur model) has demonstrated that Large Language Models (LLMs) can be fine-tuned to predict human behavior effectively, these models often lack the ability to generate explicit, natural language explanations of why those predictions are made. This paper addresses the dilemma of balancing predictive accuracy with interpretability by exploring whether LLMs can serve as dual-purpose cognitive models capable of both prediction and explanation.

Methodology

The authors propose a framework to train LLMs to generate explicit reasoning traces (Chain-of-Thought, or CoT) that explain human risky choices. The study focuses on the domain of human risky choice (decisions between safe and risky options) using the choices13k dataset (Peterson et al., 2021).

Experimental Setup

The backbone model used is Qwen-2.5-7B-Instruct. The authors compare three distinct post-training strategies applied to Low-Rank Adaptation (LoRA) modules:

Supervised Fine-Tuning (SFT): Standard autoregressive training on human choice data formatted as JSON.
Centaur-style SFT: A variant where only tokens corresponding to human choice proportions (enclosed in special brackets) contribute to the loss, masking out reasoning tokens during training.
Reinforcement Learning (RL): The authors employ Group Relative Policy Optimization (GRPO) with outcome-based rewards.

The RL Approach

In the RL setting, the model generates multiple candidate completions (including CoT reasoning and a final JSON prediction) for a given problem. The reward function is strictly outcome-based and linked to the ground truth of human behavior:

Correctness Reward: Defined as $1 - |o_B - p_B|$ , where $o_B$ is the model's predicted probability for Option B and $p_B$ is the empirical proportion of humans choosing Option B.
Format Reward: A bonus (up to 0.5) for generating a structurally valid JSON output where the prediction follows the reasoning.
Coherence Constraints: The model is penalized if its predicted probabilities do not sum to 1 or fall outside the [0, 1] range.

The RL training utilizes a group size of 12 candidates per prompt and employs the "GRPO Done Right" algorithm, which omits reward normalization by standard deviation to prevent bias toward question-level difficulty.

Control Experiments

To validate the adaptability of the reasoning traces, the authors conducted:

Data Control: Replacing human data with synthetic data generated by an expected-value maximization model to see if the CoT adapts to the new data structure.
Model Control: Replicating the experiment with a smaller, weaker backbone model (Gemma-2-2B-Instruct) to test the dependency on model capacity.

Key Results

Predictive Performance

Comparable Accuracy: At the final checkpoints, the RL-trained model achieved predictive accuracy (Mean Squared Error on the test set) comparable to both standard SFT and Centaur-style SFT ( $M \approx 0.0148$ for RL vs. $0.0144$ for SFT). Statistical tests revealed no significant differences between the methods.
Learning Efficiency: RL achieved a faster reduction in prediction error relative to the number of training examples processed compared to SFT, though it incurred higher computational costs due to generating multiple candidates per step.
Model Capacity Dependency: When applied to the weaker Gemma-2-2B model, RL performed significantly worse than SFT methods. The weaker model failed to generate the critical cognitive mechanism of "expected value calculation," leading to poor predictive performance.

Quality of Reasoning (CoT)

Emergence of Reasoning: Unlike SFT and Centaur models (which were not explicitly trained to generate reasoning), the RL model consistently produced detailed CoT reasoning prior to its final prediction.
Cognitive Mechanisms: Analysis of the RL-generated CoTs revealed the emergence of specific cognitive mechanisms, most notably Expected Value computation and Risk Aversion, which accounted for ~29–36% of the reasoning tokens. Other identified mechanisms included Loss Aversion, the Certainty Effect, and Probability Weighting.
Adaptability: In the data control experiment, when trained on synthetic expected-value data, the RL model shifted its reasoning to focus heavily on expected value calculations, demonstrating that the CoT adapts to the structure of the training data rather than remaining static.
Causal Influence: A "CoT swapping" experiment demonstrated that using the RL model's CoT to guide the backbone model significantly reduced prediction error, whereas using the backbone's original CoT with the RL model increased error. This suggests the RL-generated reasoning is causally linked to the improved predictions.
Human Evaluation: In a small-scale human evaluation, experts rated the RL model's CoTs as more reasonable than those from the backbone model in 61.5% of trials.

Significance and Claims

The paper claims to demonstrate that Reinforcement Learning with outcome-based rewards (RLVR) can elicit high-quality, interpretable reasoning traces from LLMs that serve as verbal theories of human decision-making.

Dual-Purpose Modeling: The work establishes that LLMs can function as cognitive models that simultaneously predict human behavior and provide natural language explanations of the cognitive mechanisms driving those behaviors.
Rationality vs. Heuristics: The findings suggest that, contrary to some literature emphasizing heuristics and biases, rational characterizations (specifically expected value computation and risk aversion) may be dominant forces in explaining human risky choices when viewed through the lens of these models.
Elicitation Hypothesis: The results support the "elicitation hypothesis," suggesting that RL does not necessarily teach new capabilities but rather surfaces and amplifies latent theoretical knowledge already present in the pre-trained LLM. Consequently, the success of this approach is heavily dependent on the capacity of the backbone model; weaker models fail to elicit the necessary reasoning mechanisms.
Scalability: The method offers a scalable alternative to manual cognitive modeling or expert-driven model discovery, automatically generating interpretable theories from large-scale behavioral data without requiring explicit symbolic programming or human-in-the-loop feedback for every decision.

The authors conclude that while RL post-training is computationally more expensive than SFT, it provides a unique pathway to developing cognitive models that are both quantitatively accurate and qualitatively interpretable.