HumanLM: Simulating Users with State Alignment Beats Response Imitation

The paper proposes HumanLM, a novel training framework that improves user simulation by aligning psychologically grounded latent states with ground-truth responses via reinforcement learning, outperforming existing response-imitation methods on the comprehensive Humanual benchmark and in real-time human evaluations.

The Gist

Imagine you are trying to teach a robot to act like a specific person, let's call him "Bob." Bob is a grumpy, sarcastic, but deeply caring guy who loves politics and hates when people are treated unfairly.

The Old Way (Response Imitation):
Most AI models today try to learn how to be Bob by reading thousands of his old text messages and trying to copy his exact words. It's like a student trying to pass a test by memorizing the answer key.

• The Problem: If the teacher asks a new question, the student panics. They might use Bob's favorite slang words (like "ugh" or "lol") but get the point completely wrong. They might say something cheerful when Bob would actually be furious. They are copying the surface (the paint) but missing the soul (the canvas underneath).

The New Way (HUMANLM):
The researchers behind this paper, HUMANLM, realized that to truly simulate a person, you don't need to memorize their words. You need to understand their brain state before they speak.

Think of it like an actor preparing for a role.

• Old Method: The actor memorizes the script line-by-line.
• HUMANLM Method: The actor asks, "What is my character feeling right now? What do they believe? What is their goal?"

How HUMANLM Works (The "State Alignment" Secret Sauce)

The paper introduces a framework where the AI doesn't just guess the answer; it first writes down its "thought process" based on six psychological dimensions:

1. Belief: What does this person think is true? (e.g., "The government is lying.")
2. Goal: What are they trying to achieve? (e.g., "I want to shame the politician.")
3. Value: What matters to them? (e.g., "Honesty is more important than politeness.")
4. Stance: Do they agree or disagree? (e.g., "Strongly disagree.")
5. Emotion: How do they feel? (e.g., "Heartbroken but angry.")
6. Communication: How do they say it? (e.g., "Sarcastic and direct.")

The Training Process:
Imagine a coach (the AI Judge) watching the robot try to act like Bob.

1. The robot generates a "thought trace" (the 6 states above).
2. The coach checks: "Does this robot's anger match Bob's actual anger? Does its belief match Bob's?"
3. If the robot gets the feelings right but the words slightly off, the coach gives a high score.
4. If the robot uses the perfect words but gets the feelings wrong (e.g., sounding happy when Bob is sad), the coach gives a low score.

The robot learns to maximize these "State Alignment Scores." It learns that getting the internal state right is more important than getting the exact words right.

The "HUMANUAL" Benchmark

To prove this works, the team built a massive testing ground called HUMANUAL.

• Think of this as a giant "Turing Test" arena.
• They gathered 26,000 real people and 216,000 real responses from places like Reddit, Amazon reviews, and political blogs.
• They tested the AI on everything: news comments, book reviews, and even angry emails.

The Results: Why It Matters

When they put HUMANLM to the test against other AI models:

• The Imitators: The old models sounded like robots trying to be human. They used emojis and slang but often missed the point or sounded fake.
• HUMANLM: It sounded like a real human. In a study with 111 real people, 41.4% of participants said HUMANLM's response was "mostly similar" or "nearly identical" to what they would have written.
• The "Humanlikeness" Score: 76.6% of people thought HUMANLM sounded "quite natural" or "indistinguishable from a human."

The Big Picture

The paper argues that if we want AI to help us (like helping politicians understand voters, or helping companies design better products), we can't just train AI to mimic us. We have to train AI to understand us.

In a nutshell:

• Old AI: "I will say 'Wow, that's terrible!' because Bob said that 50 times."
• HUMANLM: "Bob is feeling heartbroken about the fire, he believes the government is negligent, and he wants to express his anger sarcastically. Therefore, I will say: 'Oh, great plan, let's cut the fire budget while people lose their homes. Brilliant.'"

By focusing on the internal state (the "why" and "how" of a person's mind) rather than just the output (the "what"), HUMANLM creates a simulation that feels genuinely human, not just a cheap copy.

Technical Summary

Here is a detailed technical summary of the paper "HUMANLM: Simulating Users with State Alignment Beats Response Imitation."

1. Problem Formulation

The paper addresses the limitations of current Large Language Model (LLM) user simulators. Existing approaches primarily rely on Supervised Fine-Tuning (SFT) to imitate surface-level language patterns (e.g., tone, emoji usage, vocabulary) of real users.

• The Core Issue: Surface-level imitation fails to capture the underlying latent states (beliefs, emotions, values, stances) that drive human behavior. Consequently, these simulators often produce responses that sound stylistically similar but miss the core intent, logic, or emotional nuance of a real user, especially in unseen contexts.
• The Goal: To develop a user simulator that accurately reflects real users by aligning with their internal psychological states rather than just copying their output text.

2. Methodology: HUMANLM

The authors propose HUMANLM, a training framework that shifts the objective from imitating responses to aligning with latent states. The method is built on the insight that user responses are outcomes of specific psychological dimensions.

A. State Dimensions

The model operates on six psychologically grounded state dimensions derived from cognitive science and sociolinguistics:

1. Belief: Foundational assumptions about the world.
2. Goal: What the user aims to achieve with the response.
3. Value: What the user prioritizes (normative aspect).
4. Stance: Position on a specific topic or claim.
5. Emotion: Affective state and intensity.
6. Communication: Structural approach (e.g., direct vs. indirect, assertive vs. questioning).

B. Training Framework: Ad-hoc Alignment via RL

Instead of training the model to predict the next token of a ground-truth response, HUMANLM uses a two-stage process reinforced by Group Relative Policy Optimization (GRPO):

1. Generating Aligned Latent States (Ad-hoc):

   • The model is prompted to generate explicit natural-language latent states for a specific state dimension (e.g., "What is the user's stance?").
   • Reward Mechanism: An LLM judge evaluates the generated latent states against the ground-truth response. Crucially, the judge scores a batch of rollouts comparatively (rather than individually) to reduce bias and provide precise rewards.
   • This forces the model to learn the correct internal state before generating text.

2. Synthesizing Responses:

   • Once aligned latent states are generated, the model creates a reasoning trace that integrates these states.
   • The final response is synthesized based on this reasoning trace, ensuring the output is consistent with the aligned internal states.
   • The model is further optimized using response alignment scores to ensure the final text matches the ground truth.

C. Key Distinction from Baselines

• SFT: Learns to mimic text patterns (often leading to hallucinations or opposite stances).
• Standard RL (GRPO): Optimizes directly on response similarity, which can be ambiguous (a response might score high but for the wrong reasons).
• HUMANLM: Explicitly decouples state generation from response generation, ensuring the model "thinks" like the user before "speaking."

3. Benchmark: HUMANUAL

To rigorously evaluate user simulation, the authors introduce HUMANUAL, a comprehensive benchmark comprising six diverse datasets from public sources:

• Humanual-News: YouTube comments on BBC/CNN videos.
• Humanual-Book: Amazon book reviews.
• Humanual-Opinion: Reddit threads on personal issues (e.g., AITA).
• Humanual-Politics: Medium political blog responses.
• Humanual-Chat: Multi-turn conversations with LLM assistants.
• Humanual-Email: Enron email threads.

Statistics: 26,000 unique users, 216,000 responses, and 66,000 topics. User profiles are automatically generated from historical data using an LLM.

4. Key Results

The paper evaluates HUMANLM against strong baselines (Qwen3-8b, SFT, SFT-think, GRPO, GRPO-think, UserLM).

• Benchmark Performance:
  • HUMANLM achieves the highest Response Alignment Scores across all six datasets.
  • It shows an average relative improvement of 16.3% over the best baseline (GRPO-think).
  • It significantly outperforms SFT, which often fails to capture correct stances despite mimicking tone.
• State Alignment:
  • HUMANLM achieves the highest scores on 80% of latent state dimensions (e.g., correctly identifying "Stance" and "Emotion").
  • Training dynamics analysis shows HUMANLM explores a broader range of states, avoiding the "local optima" where models get stuck on specific text patterns.
• Human Evaluation (Real-time Simulation):
  • In a study with 111 participants, users compared their own responses to simulated ones.
  • HUMANLM achieved a 41.4% win rate on overall similarity (vs. ~28-30% for baselines).
  • 68.6% of participants rated HUMANLM responses as "mostly similar" or "nearly identical" to their own.
  • HUMANLM was rated as more "human-like" (76.6% rated as "quite natural" or "indistinguishable") and less redundant than baselines.

5. Significance and Contributions

1. Paradigm Shift: Moves user simulation from text imitation to state alignment. This ensures models capture the why behind a response, not just the how.
2. Novel Training Framework: Introduces a method to explicitly generate and align latent states using reinforcement learning, resolving the ambiguity of reward signals in standard RLHF.
3. Comprehensive Benchmark: HUMANUAL provides the first large-scale, real-world dataset for evaluating user simulators across diverse domains (politics, commerce, social interaction), moving beyond synthetic or chat-only benchmarks.
4. Practical Impact: Enables more reliable "human-in-the-loop" applications, such as:
   • Testing policy changes or product features with simulated user feedback.
   • Improving AI assistants by understanding user intent and emotional state.
   • Generating high-quality synthetic data for training other models.

Conclusion

HUMANLM demonstrates that simulating users requires modeling their internal psychological states. By explicitly aligning latent states (belief, emotion, stance) before generating text, the model produces responses that are not only linguistically plausible but also psychologically accurate, significantly outperforming traditional imitation-based approaches.