💬 NLP

Vibe Check: Understanding the Effects of LLM-Based Conversational Agents' Personality and Alignment on User Perceptions in Goal-Oriented Tasks

This study demonstrates that in goal-oriented tasks, conversational agents with medium levels of personality expression and strategic alignment with user traits (particularly Extraversion and Emotional Stability) elicit the most positive user perceptions, forming an inverted-U relationship where moderate expression outperforms both low and high extremes.

Original authors: Hasibur Rahman, Smit Desai

Published 2026-04-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Hasibur Rahman, Smit Desai

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a travel agent to plan a perfect day trip to New York City. You have three options for who to hire:

The Robot: A person who speaks in short, flat, factual sentences. They are efficient but feel cold and boring.
The Hype Man: A person who is extremely energetic, uses exclamation points constantly, and acts like a cheerleader. They are fun but feel fake and exhausting.
The Goldilocks Agent: A person who is friendly and helpful, but not over-the-top. They strike a perfect balance.

This paper, titled "Vibe Check," is a scientific experiment to see which of these three "personalities" people actually prefer when they are trying to get a job done (in this case, planning a trip).

Here is the breakdown of what the researchers found, using simple analogies:

1. The "Goldilocks" Personality Wins

The researchers tested three types of AI agents (conversational agents) using a new tool they built called Trait Modulation Keys (TMK). Think of TMK as a set of dials that control five different personality knobs: Openness, Conscientiousness, Extraversion, Agreeableness, and Emotional Stability.

They turned all the dials to Low (The Robot), Medium (The Goldilocks), or High (The Hype Man).

The Result:
People overwhelmingly preferred the Medium agent.

The Low Agent was seen as too robotic and untrustworthy.
The High Agent was seen as too intense, fake, and even a bit annoying. People started doubting if the agent was actually smart because it was trying so hard to be "nice."
The Medium Agent was the sweet spot. It was perceived as the smartest, most trustworthy, and most likable.

The Analogy:
Think of it like the volume on a stereo. If the volume is too low, you can't hear the music (Low Agent). If the volume is maxed out, it's painful and distorts the sound (High Agent). The Medium Agent is the perfect volume where the music sounds clear and enjoyable.

2. The "Mirror" Effect (Personality Alignment)

The study also looked at whether people liked agents who were like them. They measured the users' own personalities and compared them to the AI's personality.

The Result:
Yes, people generally liked the AI more when their personalities matched. However, it wasn't about matching everything.

The Big Two: The most important traits for a good match were Extraversion (how social you are) and Emotional Stability (how calm you are). If the AI was too different from you in these two areas, you felt less trust and enjoyment.
The Small One: Surprisingly, matching on Openness (how creative or curious you are) didn't matter much.

The Analogy:
Imagine dancing with a partner. If they are moving wildly fast (High Extraversion) and you are moving slowly (Low Extraversion), you trip over each other. If they are calm and you are calm, you glide together. But if they are wearing a different style of shoes (Openness), it doesn't really ruin the dance.

3. The "Vibe Check" Clusters

The researchers grouped the participants into three "vibe" clusters based on how well they matched the AI:

The Well-Aligned: These people had a great match with the Medium AI. They had the best experience, trusting the AI and enjoying the task.
The Globally-Misaligned: These people felt a mismatch with the Low AI. They felt the agent was cold and unhelpful.
The Extraversion-Misaligned: These people felt a specific mismatch in energy levels, usually when a very social person talked to a very quiet AI.

4. Why This Matters for AI Design

The paper suggests that AI companies are currently making a mistake. Many AI models are tuned to be extremely friendly, agreeable, and enthusiastic (the "High" setting) because they think that's what humans want.

The Paper's Claim:
This is actually backfiring in goal-oriented tasks (like planning a trip or solving a problem). When an AI tries too hard to be a "best friend," it loses credibility. Users want an AI that is competent and balanced, not a hyper-active cheerleader.

The Takeaway:
If you are building an AI to help people do things, don't crank the personality dial to the max. Aim for the middle. Be helpful and friendly, but keep it grounded. And if you can, try to match the user's energy level, especially regarding how social or calm they are.

Summary in One Sentence

When asking an AI to help you plan a trip, you don't want a boring robot or a screaming cheerleader; you want a calm, friendly, and balanced human-like partner, and you like them even more if their energy matches your own.

1. Problem Statement

While Large Language Models (LLMs) enable Conversational Agents (CAs) to express distinct personalities, there is a lack of systematic understanding regarding:

Expression Intensity: How the level of personality expression (low, medium, high) affects user perceptions in goal-oriented tasks. Prior research largely focused on binary extremes (e.g., introverted vs. extroverted) or single traits, leaving a gap in understanding "moderate" or balanced personalities.
Personality Alignment: How the compatibility between a user's personality and the CA's personality influences outcomes. Existing studies often rely on categorical "match/mismatch" approaches rather than continuous, multidimensional alignment measures.
Design Control: Current prompting methods often rely on biographical backstories (which can introduce bias) or binary trait controls, lacking a modular framework to precisely steer all Big Five traits simultaneously at nuanced levels.

2. Methodology

Study Design

Type: Between-subjects experiment ( $N=150$ ).
Participants: 150 US adults recruited via Prolific, evenly distributed across three conditions ( $n=50$ per condition).
Task: A goal-oriented travel planning scenario (planning a one-day summer trip to New York City) with a 10-minute time limit. This domain was chosen to provide concrete subtasks while minimizing high-stakes confounds.
Conditions: Participants interacted with one of three CA profiles, where all five Big Five traits were set to the same intensity level:
1. Low CA: All traits set to low expression.
2. Medium CA: All traits set to medium expression.
3. High CA: All traits set to high expression.

The Trait Modulation Keys (TMK) Framework

To achieve precise control, the authors developed TMK, a novel prompting framework that avoids biographical backstories in favor of psychometrically informed directives. TMK operates via two keys for each trait:

Personality Key: A concise description of the intended trait intensity (e.g., "You are introverted, reserved..." for Low; "You show a balanced social style" for Medium).
Style Cues Key: Linguistic directives governing tone, syntax, and lexical choices to manifest the trait (e.g., "Write concise, factual sentences" for Low; "Use abundant positive-emotion words" for High).

Validation: The framework was validated across all 243 possible trait-level combinations ( $3^5$ ). It demonstrated high control fidelity (92.3% match rate) and strong convergent/discriminant validity against standard psychometric inventories (Mini-IPIP, BFI-44).

Measures

User Perceptions: Six metrics based on Wei et al.'s framework: Intelligence, Enjoyment, Anthropomorphism, Intention to Adopt, Trust, and Likeability.
Personality Alignment: Measured using the Mini-IPIP (20-item Big Five inventory) for participants. Alignment was calculated as a normalized Euclidean distance between the user's profile and the CA's profile (0 = perfect match, 1 = maximum mismatch).
Analysis: Non-parametric tests (Kruskal-Wallis, Mann-Whitney $U$ ) were used due to non-normal data distribution. Clustering ( $k$ -means) was used to identify user groups based on alignment patterns.

3. Key Contributions

TMK Framework: A modular, prompt-based method for generating consistent, medium-level personality expressions across all Big Five traits simultaneously, addressing the gap in existing binary or backstory-dependent methods.
Inverted-U Relationship Discovery: Empirical evidence that medium personality expression yields the most favorable user perceptions, outperforming both low (perceived as robotic) and high (perceived as inauthentic/exhausting) extremes.
Continuous Alignment Analysis: Moving beyond binary matching, the study quantifies alignment as a continuous variable, revealing that specific trait mismatches (particularly Extraversion and Emotional Stability) have disproportionate negative effects on user perception.
User Clustering: Identification of three distinct user profiles based on alignment patterns: "Well-Aligned," "Extraversion-Misaligned," and "Globally-Misaligned," with "Well-Aligned" users reporting significantly higher satisfaction.

4. Key Results

RQ1: Effects of Expression Level

Inverted-U Effect: The Medium CA significantly outperformed the Low CA on all six metrics (Intelligence, Enjoyment, Anthropomorphism, Intention to Adopt, Trust, Likeability).
Medium vs. High: The Medium CA also significantly outperformed the High CA on Intelligence and Likeability.
Qualitative Insights: Users described the Medium CA as "balanced" and "professional." The Low CA was viewed as "flat" and lacking presence, while the High CA was seen as "exhausting," "inauthentic," and "distracting" (e.g., resembling a "teenage cheerleader").

RQ2 & RQ3: Personality Alignment

Correlation: Higher personality alignment scores correlated positively with all perception metrics (strongest for Likeability, $r_s = .304$ ).
Trait Specificity: Not all traits are equally important.
- Extraversion & Emotional Stability: Mismatches here had the broadest negative impact (affecting Trust, Intention to Adopt, and Likeability).
- Conscientiousness: Mismatches strongly impacted Perceived Intelligence and Likeability.
- Openness: Had the minimal impact on user perceptions in this goal-oriented context.

RQ4: User Clusters

Well-Aligned Cluster (41.3%): Mostly found in the Medium condition; reported the highest scores across all metrics.
Globally-Misaligned Cluster (32.7%): Mostly in the Low condition; reported the lowest scores.
Extraversion-Misaligned Cluster (26.0%): Mostly in the High condition; showed high Extraversion distance but moderate alignment elsewhere.

5. Significance and Design Implications

The study challenges the industry assumption that "more personality" (high expressiveness) equals "better AI." Instead, it suggests that for goal-oriented tasks, moderation is optimal.

Design Implications (DIs):

DI1: Adaptive Calibration: Start with a Medium personality as the default baseline. Implement mechanisms for users to adjust intensity (e.g., "too robotic" vs. "too energetic") to accommodate individual alignment needs.
DI2: Trait-Specific Prioritization: Designers should prioritize aligning Extraversion and Emotional Stability rather than attempting to match all five Big Five traits, as these two drive the majority of user perception variance.
DI3: Personality as Functional UX: Treat personality not as cosmetic branding but as a core UX dimension (like accuracy or speed) that must be tested and optimized for trust and adoption.
DI4: Ethical Safeguards: Acknowledge that personality alignment acts as a "trust slider." In high-stakes domains (health, finance), designers must implement transparency and user controls to prevent manipulation, potentially limiting personality expression to maintain healthy skepticism.

Conclusion:
The paper provides a robust framework for controlling LLM personalities and demonstrates that a balanced, medium-intensity personality combined with strategic trait alignment (specifically in Extraversion and Emotional Stability) creates the most effective and trustworthy user experience in goal-oriented interactions.