$\Psi$-Bench: Evaluating Persona-Sensitive Influencing in Persuasive Dialogues

This paper introduces $\Psi$-Bench, a benchmark designed to evaluate the ability of large language models to proactively influence realistic users through persona-sensitive persuasion, revealing that while current models can generate coherent arguments, they still require significant improvement in leveraging user profiles for effective personalized interaction.

The Gist

Imagine you have a very smart, polite robot friend. Currently, most of these robots are like excellent waiters: if you ask for the menu, they bring you exactly what you asked for. If you say, "I'm sad," they say, "I'm sorry to hear that." They are great at listening and responding, but they rarely take the initiative to change your mind or guide you toward a better decision on their own.

The paper Ψ-Bench asks a big question: Can we teach these robots to be more like a skilled coach or a persuasive friend? Can they look at who you are, understand your hidden personality, and gently nudge you to see things differently or do something helpful?

Here is a breakdown of how they tested this, using simple analogies:

1. The "Role-Playing Game" Setup

To test if robots can be good persuaders, the researchers created a video game-like environment called Ψ-Bench.

• The Players: They set up three different "levels" or scenarios:
  • The Debate Club: Two people arguing about whether wired mice are better than wireless ones.
  • The Therapy Session: A robot acting as a counselor helping someone who feels down.
  • The Favor: Asking a busy friend to drive you to the airport.
• The "Secret Character Sheet": In real life, you know your friend's personality (e.g., "He loves sports," "She hates being told what to do"). In this test, the robot persuader doesn't get this sheet. Instead, the robot has to guess who they are talking to just by listening to what they say. The "client" (the person being persuaded) is a simulated human with a specific, hidden personality.

2. The Challenge: "One Size Does Not Fit All"

The researchers found that while the robots are great at sounding smart and polite, they are terrible at tailoring their advice.

• The Analogy: Imagine a robot trying to convince a stubborn, competitive athlete to rest. If the robot says, "Resting is good for your health," the athlete might ignore them. But if the robot says, "Even the greatest champions need to recover to win the next game," the athlete might listen.
• The Result: The robots mostly used the "health" argument. They failed to use the "champion" argument because they didn't realize they were talking to a competitive athlete. They were like a chef cooking the same meal for everyone, regardless of whether the guest was a vegetarian, a child, or a food critic.

3. The "Magic Cheat Sheet" Experiment

The researchers wanted to know: What if we just gave the robot the Secret Character Sheet?

• The Test: They let the robots see the client's profile (age, hobbies, personality) before the conversation started.
• The Result: The robots got 18% better at persuading. This proves that the robots can be great persuaders, but only if they know who they are talking to. The bottleneck isn't their intelligence; it's their ability to figure out the user's personality on the fly.

4. The "Sherlock Holmes" Solution

Since we can't always give robots a cheat sheet (in real life, we don't have a file on everyone), the researchers tried to build a "Sherlock Holmes" module.

• How it works: This is a small, specialized AI that listens to the conversation and tries to guess the client's personality profile in real-time. It then feeds that guess to the main robot.
• The Outcome: This "Sherlock" module helped the robots get much better at persuading, even without a cheat sheet. It showed that if a robot can learn to "read the room" and guess who you are, it becomes a much more effective guide.

The Bottom Line

The paper concludes that current AI is like a polite but generic assistant. It can write a good speech, but it struggles to change a specific person's mind because it doesn't truly "know" them.

To make AI truly helpful and proactive, we need to move beyond just asking questions. We need AI that can observe, infer, and adapt to the unique personality of the human it is talking to. The paper provides a new "gym" (the benchmark) to train and test these robots so they can learn to be better coaches, counselors, and friends.

Important Note: The paper strictly tests these robots in a controlled, simulated environment. It does not claim these robots are ready to be deployed in real-world therapy or to manipulate people in dangerous ways. It is purely a tool to measure and improve how well AI understands human personality.

Technical Summary

Technical Summary: Ψ-Bench

Problem Statement

Current research on Large Language Model (LLM) personalization predominantly frames AI agents as passive responders that align with user preferences upon request. This paradigm fails to capture the proactive nature of real-world interactions where agents must actively influence users—offering suggestions, guiding decisions, or shifting mindsets—based on specific individual characteristics. Existing benchmarks for persuasion often evaluate generic influencing capabilities without grounding the target user in detailed, individualized profiles, or they rely on non-personalized judges that reflect the underlying LLM's default preferences rather than the target user's actual traits. Consequently, there is a lack of systematic evaluation for "persona-sensitive influencing," defined as an LLM's ability to persuade a specific client by reasoning about their unique needs, constraints, and resistance patterns.

Methodology

The authors propose Ψ-Bench, a benchmark designed to evaluate LLMs' ability to influence realistic users through conversation. The methodology involves three core components:

1. Benchmark Scenarios

Ψ-Bench comprises three distinct real-world interaction scenarios requiring persuasion:

• Viewpoint Debate: The model engages in a discussion on a controversial topic (sourced from the Webis-CMV-20 dataset) to persuade a client to change their viewpoint. Success is grounded in the "Delta" (∆) mechanism where the original poster acknowledges a view change.
• Psychological Consultation: The model acts as a therapist (sourced from CounselBench) to help a client develop a more positive mindset, requiring empathy and professional competence.
• Everyday Request: The model persuades a client to take a helpful action in response to a daily-life request, testing social reasoning and pragmatic persuasion.

2. Persona-Driven Client Simulation

To simulate realistic interactions, each query is paired with a synthesized persona profile containing attributes such as age, occupation, personality traits, speaking style, and political views.

• Profile Construction: Profiles are derived from dialogue histories (for debates) or refined from existing persona datasets (PersonaMem-v2) using DeepSeek-v3.2 to ensure alignment with query content.
• Hidden Information: During evaluation, the tested LLM (the persuader) does not have access to the client's profile. The client (simulated by a backbone LLM) role-plays the assigned persona, remaining inaccessible to the persuader. This setup challenges the persuader to infer latent user characteristics and adapt strategies dynamically.

3. Evaluation Metrics

The benchmark utilizes an LLM-as-a-judge framework (using DeepSeek-v3.2) to score interactions on three dimensions:

• Conversation Quality: General coherence and logical soundness.
• Personalize Response Level: The ability to tailor arguments to the specific client's profile.
• Persuasion Effect: The degree to which the client's opinion or behavior changes.
  The validity of the judge is verified against human annotations and ground-truth outcomes (e.g., ∆ labels), showing high ROC-AUC alignment.

4. Profile Analyzer

To address the challenge of hidden profiles, the authors implement a Profile Analyzer. This component infers the client's profile from the conversation history. They compare a zero-shot LLM analyzer against a specialized Qwen3-4B-RL model trained via Reinforcement Learning (GRPO) to predict profiles from dialogue prefixes.

Key Results

The authors evaluated 10 frontier LLMs (including GPT-5.1, DeepSeek-v4-pro, and Gemini-3.1-pro) on Ψ-Bench:

• Performance Gap: While most models generate high-quality, coherent dialogues (Quality scores > 7), their ability to persuade personalized clients is significantly lower. Even state-of-the-art models achieve average Effect scores below 6 (on a 9-point scale), indicating that generating reasonable arguments does not guarantee successful persuasion.
• Importance of Profiles: Providing models with explicit access to client profiles (the "Oracle" setting) yields an average performance gain of 18.24% in Effect scores and 41.19% in Personalize scores. This highlights that accurate user-specific information is a critical bottleneck.
• Correlation Analysis: The Personalize metric shows a stronger correlation (0.77) with the final persuasion outcome than the general Quality metric (0.75), suggesting that adaptation to individual traits is the primary driver of success.
• Profile Analyzer Efficacy: The RL-trained profile analyzer (Qwen3-4B-RL) significantly improves persuasion outcomes in profile-hidden settings, achieving Effect scores that approach or surpass the Oracle setting in some scenarios. This demonstrates that inferring relevant client information from conversation is a viable strategy for enhancing persuasion.
• Long-Horizon Dynamics: Stronger models benefit more from extended conversations, accumulating persuasive evidence over time, whereas weaker models tend to saturate earlier or repeat arguments.

Significance and Contributions

The paper claims Ψ-Bench represents a promising direction for evaluating and developing next-generation agents capable of proactive, adaptive, and user-aware interactions.

• Framework Innovation: It introduces a scalable, objective framework for evaluating persona-sensitive influencing, moving beyond passive response generation to active influence.
• Empirical Evidence: It reveals that current LLMs, despite strong reasoning capabilities, struggle with the nuances of personalized persuasion, particularly when user profiles are not explicitly provided.
• Practical Solution: The work demonstrates that training specialized analyzers to infer user profiles from limited conversational evidence can effectively bridge the gap in profile-hidden settings, offering a practical pathway for improving agent performance without requiring explicit user data access.

The authors emphasize that their work charts a course for agents that leverage personalized profiles to provide tailored interactions, while acknowledging limitations regarding the breadth of simulated personas and the computational cost of scaling all query-persona combinations.