🤖 machine learning

Reinforcing Human Behavior Simulation via Verbal Feedback

This paper introduces DITTO, a model trained via reinforcement learning with verbal feedback to better simulate human behavior, alongside the SOUL benchmark suite, demonstrating significant performance improvements over base models and surpassing GPT-5.4 on multiple human-like simulation tasks.

Original authors: Weiwei Sun, Xuhui Zhou, Jiarui Liu, Weihua Du, Haojia Sun, Yiqing Xie, Qianou Ma, Sihao Chen, Mengting Wan, Longqi Yang, Pei Zhou, Sherry Wu, Sean Welleck, Graham Neubig, Yiming Yang, Maarten Sap

Published 2026-05-21

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Weiwei Sun, Xuhui Zhou, Jiarui Liu, Weihua Du, Haojia Sun, Yiqing Xie, Qianou Ma, Sihao Chen, Mengting Wan, Longqi Yang, Pei Zhou, Sherry Wu, Sean Welleck, Graham Neubig, Yiming Yang, Maarten Sap

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Teaching AI to "Act Human" by Talking to It

Imagine you are teaching a child how to behave at a dinner party.

The Old Way (Scalar Rewards): You give the child a score of "6 out of 10" after they finish eating. You don't tell them why they got a 6. Did they talk too much? Did they use the wrong fork? Did they spill soup? They have no idea how to improve, so they just guess next time.
The New Way (Verbal Feedback): You sit them down and say, "You did great with the napkin, but you talked over your grandmother, and that was rude. Next time, wait for her to finish speaking."

This paper argues that current AI models (LLMs) are being trained like the child getting a score of "6." They are good at math and coding because those things have clear right/wrong answers (like a test score). But when we ask AI to simulate human behavior—like acting as a patient, a student, or a friend—simple scores don't work. Humans learn social norms through words, explanations, and corrections, not numbers.

The authors built a new system called DITTO that teaches AI to act human by using this "verbal feedback" method.

How DITTO Works: The "Draft and Polish" Game

Think of DITTO as a creative writing workshop where the AI is both the student and the editor. Here is the process:

The First Draft (The Student): The AI tries to answer a prompt (e.g., "Act like a grumpy teacher"). It writes a response.
The Critique (The Judge): A powerful AI "judge" reads the draft and gives verbal feedback. It doesn't just say "Good job." It says, "You were too polite. A grumpy teacher would be more impatient. Also, you forgot to mention the homework."
The Second Draft (The Teacher): The AI takes that specific feedback and writes a new, improved version of the response.
The Lesson (The Magic): The system trains the AI to look at the original prompt and immediately generate the improved version, without needing the feedback text anymore. It's like the student reading the teacher's notes, fixing the essay, and then memorizing the feeling of how to write a good essay so they don't need the notes next time.

The Result: The AI learns to "internalize" the advice. When you talk to it later, it acts more human because it has learned why certain behaviors are better, not just that they are better.

The Playground: SOUL (Simulation Gym)

To test if this actually works, the researchers built a giant gym called SOUL (Simulation gym Of hUman-Like behavior).

Imagine a gym with 10 different stations, each testing a different type of "human" skill:

Theory of Mind: Can the AI understand that someone else knows something it doesn't? (Like a game of hide-and-seek).
Role Play: Can it act like a specific character from a book without breaking character?
Social Skills: Can it navigate a conversation to get a goal (like asking for a raise) without being rude?
User Simulation: Can it pretend to be a confused customer talking to a help desk?

They found that existing AI models often failed these tests. They sounded too robotic, too perfect, or too similar to each other.

The Results: DITTO Wins the Gym

The researchers tested their new model (DITTO) against the best existing models (including giant ones from big tech companies).

The Score: DITTO improved by 36% compared to the basic model.
The Showdown: DITTO beat the top-tier "GPT-5.4" model on 6 out of the 10 challenges.
Where it Shined: It was especially good at tasks that require nuance, like social skills and role-playing. It learned to keep secrets better and handle complex conversations without getting "stuck" or sounding fake.

Why This Matters (According to the Paper)

The paper claims that to make AI truly useful for simulating people (like in therapy bots, educational tutors, or customer service training), we have to stop treating them like math students who just need a score. We have to treat them like social beings who need explanations.

By using verbal feedback (words) instead of just scalar rewards (numbers), the AI learns the texture of human behavior. It learns that being "rude" isn't just a low score; it's a specific way of speaking that hurts feelings.

In short: DITTO is an AI that learned to act human by listening to a teacher's detailed critiques, practicing the corrections, and then forgetting the teacher's notes because it finally "got it."

Technical Summary: Reinforcing Human Behavior Simulation via Verbal Feedback

Problem Statement
Current Large Language Model (LLM) simulators for human behavior suffer from systematic "Sim2Real" gaps, exhibiting homogeneous outputs, misaligned cognitive biases, and superhuman behaviors that diverge from real users. A primary limitation lies in the training paradigm: standard Reinforcement Learning (RL) methods (e.g., RLHF, DPO, GRPO) compress all feedback into scalar rewards or binary preferences. While effective for domains with verifiable correctness (e.g., code, math), scalar rewards fail to capture the nuance required for human behavior simulation, which is subjective, multi-faceted, and context-dependent. Humans learn social norms not from scores, but from verbal feedback—explanations, corrections, and dialogue. Existing approaches that attempt to incorporate verbal feedback often rely on supervised fine-tuning (SFT) or inference-time reflection, lacking the exploration capacity of RL or failing to update model weights effectively from experience.

Methodology: DITTO
The authors propose DITTO, a model trained by treating verbal feedback as a first-class signal within a Reinforcement Learning framework. The core innovation is a training loop that generates feedback-conditioned improvements and jointly optimizes them with the base policy, allowing the model to internalize guidance without requiring feedback at test time.

The training process for a given prompt $x$ proceeds as follows:

Draft Rollout: The policy $\pi_\theta$ generates an initial response $y_0$ .
Feedback Generation: An LLM judge evaluates $y_0$ , producing a scalar reward $r_0$ and structured verbal feedback $h$ (including critiques and reflections on dimensions like believability, relationship maintenance, and goal achievement).
Refined Rollout: The policy generates a second response $y_1$ conditioned on the prompt and the feedback ( $x, h$ ).
Joint Optimization: Both $y_0$ $y_{0}$ and $y_1$ $y_{1}$ are optimized using Group Relative Policy Optimization (GRPO).
- A group-relative advantage is computed over the set of original and refined rollouts.
- A standard clipped GRPO objective ( $L_{group}$ ) is applied to the joint group to encourage the base policy to learn from the improvements.
- An additional objective ( $L_{fb}$ ) is applied specifically to the feedback-conditioned rollouts to improve the model's ability to respond to feedback directly.
- The final loss is $L = L_{group} + L_{fb}$ .

Crucially, the verbal feedback $h$ is treated as privileged information: it is available during training to guide the generation of $y_1$ and optimize the policy, but it is not required during inference. The goal is for the base policy to internalize the "teacher" behavior (the improved $y_1$ ) into its standard generation process.

SOUL Benchmark
To facilitate training and evaluation, the authors introduce SOUL (Simulation gym Of hUman-Like behavior), a unified benchmark and training data suite. SOUL addresses the bottleneck of lacking training data for human simulation by providing curated RL training splits for 10 tasks across six categories:

Theory of Mind: FanToM, HiToM, ToMi.
Character Role Play: CoSER, LifeChoices.
Social Skill: Sotopia.
Learner Simulation: Mistakes.
User Simulation: MirrorBench, UserLLM.
Persona Simulation: TwinVoice.

The suite covers discriminative (multiple-choice), generative (open-ended dialogue), and multi-turn/multi-agent interaction formats.

Key Results
Experiments were conducted using a Qwen3-8B base model.

Performance: DITTO achieved an average normalized score of 0.726 across SOUL tasks, representing a 36% improvement over the base model.
Comparison to State-of-the-Art: DITTO outperformed the frontier closed-source model GPT-5.4 on 6 out of 10 benchmarks, including all three Theory of Mind tasks and both User Simulation tasks. It also consistently surpassed specialized human-simulation baselines (e.g., Sotopia-RL, HER).
Ablation: The verbal feedback mechanism provided consistent gains over standard GRPO, with the largest improvements observed on generative and multi-turn tasks (e.g., +14 points on TwinVoice, +11 on ToMi).
Safety: Analysis showed that verbal feedback helped mitigate safety risks. While standard GRPO degraded on "secret-keeping" dimensions during optimization, DITTO better preserved the ability to keep secrets, likely due to the explicit guidance provided in the verbal feedback.

Significance and Claims
The paper claims that moving beyond scalar rewards toward richer, verbal feedback is a promising direction for training more human-like simulators. The significance of the work lies in three main contributions:

Verbal Feedback as a Signal: Demonstrating that verbal feedback provides actionable, dimension-specific guidance that improves learning speed and reduces safety risks (such as secret leakage) compared to scalar rewards.
DITTO Architecture: Introducing a method that distills verbal feedback into the base policy via feedback-conditioned rollouts, achieving significant performance gains with an 8B model that matches or exceeds larger, closed-source models.
SOUL Benchmark: Providing the first unified training and evaluation environment for human-like behavior across diverse task formats, addressing the critical lack of training data in this domain.

The authors conclude that their approach enables the base policy to internalize rich, actionable guidance, suggesting that RL with verbal feedback is a viable path to bridging the Sim2Real gap in human behavior simulation.