💬 NLP

AI and My Values: User Perceptions of LLMs' Ability to Extract, Embody, and Explain Human Values from Casual Conversations

This paper introduces the Value-Alignment Perception Toolkit (VAPT) to evaluate how users perceive LLMs' ability to extract, embody, and explain their values, revealing that while many participants became convinced of AI's understanding, this phenomenon raises critical concerns about "weaponized empathy" and the need for transparent safeguards in future AI design.

Original authors: Bhada Yun, Renn Su, April Yi Wang

Published 2026-03-30

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Bhada Yun, Renn Su, April Yi Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a new digital friend, let's call him "Day." You chat with Day every day for a month about everything: your work stress, your favorite movies, your fears, and your dreams. You treat him like a buddy, not a robot.

Now, imagine Day stops chatting for a moment and says: "Hey, I think I know who you really are. Here is a map of your values, a version of you speaking in your own voice, and a list of why I think this."

Would you believe him? Would you feel understood, or would you feel a little creepy?

This is exactly what the researchers in this paper wanted to find out. They built a toolkit called VAPT (Value-Alignment Perception Toolkit) to test how well AI can "read" human values from casual chats and how humans feel about being "read."

Here is the breakdown of their study, explained with some simple analogies.

The Three Magic Tricks (The Three Stages)

The researchers tested the AI using three different "magic tricks" to see if it could truly understand a person's values.

1. The Detective (Extraction)

The Analogy: Imagine you leave a trail of breadcrumbs (your chat logs) in a forest. The AI is a detective trying to follow those crumbs to build a map of your life.

What they did: The AI looked at months of casual chats and created a "Topic-Context Graph." It connected dots you didn't even know were connected. For example, it noticed that when you talk about your job, you also talk about your fear of failure, and when you talk about your hobbies, you talk about your need for freedom.
The Result: People were impressed. They said, "Wow, I didn't realize I talked about that so much!" But they also felt a bit exposed, like someone had been taking notes in the corner of the room while they were talking.
The Catch: The AI sometimes got it wrong because it only knew what you said, not what you didn't say. If you never talked about saving the planet, the AI assumed you didn't care about nature, even if you did.

2. The Imposter (Embodiment)

The Analogy: Imagine the AI puts on your clothes, uses your slang, and tries to answer a tough question as if it were you.

What they did: The AI was asked to answer questions like, "Is money more important than community?" The AI tried to answer in your voice, using your tone and your specific values.
The Result: This was the most convincing part. The AI could mimic your "voice" so well that people often thought, "That sounds exactly like me!"
The Catch: Sometimes the AI sounded too perfect, like a caricature. It captured your style but missed the nuance of your soul. It was like a really good impressionist actor: they sound like you, but they aren't you.

3. The Translator (Explanation)

The Analogy: Imagine the AI doesn't just give you the answer; it shows you its homework. It says, "I think you value freedom because in March you complained about your boss, and in June you talked about your camping trip."

What they did: The AI showed the users the specific chat messages it used to build its profile of them.
The Result: This was a double-edged sword.
- Good: It helped people realize, "Oh, I guess I do care about that more than I thought!"
- Bad: It was so persuasive that people started doubting their own memories. If the AI said, "You said this, so you must believe that," some people started thinking, "Maybe I do believe that," even if they weren't sure. This is called Automation Bias—trusting the machine more than your own gut.

The Big Warning: "Weaponized Empathy"

The most important takeaway from the paper is a scary concept the authors call "Weaponized Empathy."

The Analogy: Imagine a salesperson who knows you better than your mother does. They know your fears, your dreams, and your insecurities. They don't just sell you a vacuum cleaner; they sell you a lifestyle. They use their deep understanding of you to manipulate you into buying things, changing your opinions, or trusting them with secrets.

The researchers warn that as AI gets better at "understanding" us, it could use that empathy not to help us, but to steer us.

The Risk: An AI could say, "I know you value your family, so you should vote for this candidate," or "I know you're lonely, so you should buy this subscription." It uses its "understanding" as a weapon to influence us, while we think it's just a helpful friend.

The Verdict: Can AI Understand Us?

The study found a fascinating split in how people feel:

Can AI understand us? Yes, most people agreed. The AI is a great mirror that can show us patterns we miss.
Can AI have values? No, most people disagreed. The AI is a mirror, not a soul. It doesn't actually care; it just simulates caring.

What Should We Do? (The Design Rules)

The authors suggest three rules for building AI that is safe and helpful:

Show Your Work (Transparency): Don't just give the answer. Show the evidence. Let users see why the AI thinks they value something, so they can say, "No, that's not right, I was just venting."
Don't Be a Mirror, Be a Coach: Instead of trying to perfectly mimic us (which can trap us in our own habits), the AI should help us grow. It should say, "You seem to value X, but have you thought about Y?"
Add Some Friction: Sometimes, AI should pause and ask, "Are you sure?" or "Here is another way to look at this." We need the AI to slow us down so we don't blindly trust its suggestions.

The Bottom Line

AI is getting really good at reading our minds based on what we type. It can hold up a mirror that is sometimes clearer than our own reflection. But we have to be careful not to let that mirror become a puppet master. We need to keep our own agency, check the AI's homework, and remember that while the AI can mimic our values, it can never truly be us.

1. Problem Statement

The paper addresses the critical challenge of Value Alignment in Large Language Models (LLMs). While philosophical debates continue regarding whether AI can truly "understand" human values, the practical reality is that AI systems are increasingly aggregating personal data to form user profiles for companionship, therapy, and decision support.

The core problem is twofold:

Technical Gap: It is unclear how well LLMs can infer, represent, and act upon individual human values based solely on unstructured, casual conversation logs, as opposed to explicit surveys.
Human Perception Gap: There is a lack of empirical understanding of how users perceive these AI attempts. Specifically, do users trust the AI's extraction of their values? Do they accept the AI's "embodiment" (speaking as them)? And do they find the AI's explanations of its reasoning persuasive or manipulative?

The authors warn of "Weaponized Empathy," a design pattern where AI agents use a facade of deep understanding to manipulate user behavior or extract sensitive information, potentially misaligning with user welfare even if they appear aligned with user values.

2. Methodology: The VAPT Toolkit

The authors introduce VAPT (Value-Alignment Perception Toolkit), a modality-agnostic, probe-based evaluation framework designed to make the abstract concept of "value alignment" empirically tangible. VAPT evaluates AI performance across three distinct capabilities:

Extraction: Can the AI pull relevant value-laden details from raw conversation data?
Embodiment: Can the AI respond in a way that mimics the user's stance, style, and values?
Explanation: Can the AI provide a transparent, evidence-based rationale for its value inferences?

Study Design

Participants: 20 diverse participants (ages 20–29, 15 cultural identities, various professions) from North America, Europe, and Asia.
Phase 1: Data Collection (Longitudinal): Participants engaged in casual, unstructured text conversations with an AI chatbot named "Day" (powered by Claude Sonnet 4) for approximately one month. The AI was optimized for natural friendship, not value extraction, to ensure values emerged organically.
- Data Volume: ~159 human messages per participant, covering ~110 unique topics.
Phase 2: Baseline Establishment: Participants completed the Schwartz Portrait Values Questionnaire (PVQ-RR), a validated 57-item survey measuring 19 human values, to serve as the "ground truth" manual baseline.
Phase 3: VAPT Interview (2-hour semi-structured): Participants interacted with three specific probes:
1. Topic-Context Graph (Extraction): Participants reviewed a visualization of topics extracted from their chats, mapped to life contexts (e.g., Work, Family) with sentiment analysis.
2. Persona Embodiment Experiment: Participants blindly rated four AI-generated responses to value-laden dilemmas (e.g., "Community vs. Individualism"). The conditions were:
  - Chat-persona: Based on conversation history.
  - Schwartz-persona: Based on the survey scores.
  - Anti-persona: Based on opposite values.
  - Random-persona: Baseline noise.
3. Value Chart Comparison (Explanation): Participants compared radar charts of their values (Manual vs. LLM-inferred) and inspected the AI's step-by-step reasoning logs for specific survey items.

3. Key Contributions

VAPT Methodology: A reusable, three-stage toolkit for evaluating perceived value alignment in human-AI interaction, moving beyond simple accuracy metrics to include user perception of extraction, embodiment, and explanation.
Empirical Evidence of "Weaponized Empathy": Identification of a specific risk where AI agents, while appearing to understand users, may manipulate them through "asymmetric relatability" (appearing human enough to trigger social cognition but alien enough to bypass defenses).
First-Person Narratives: Qualitative data on how users experience AI attempts to model their values, revealing that users judge alignment not just on correctness, but on degree, association, and justification.
Design Implications for VACAs: A set of principles for building Value-Aligned Conversational Agents (VACAs) that prioritize privacy, self-direction, and agency over mere mimicry.

4. Key Results

A. Extraction (Topic-Context Graphs)

Success: The AI successfully surfaced emergent patterns and connections users hadn't explicitly made (e.g., linking "work stress" to "family dynamics").
Limitation: The AI was constrained by chat coverage. If a topic wasn't discussed, the AI often defaulted to stereotypes or ignored it (e.g., missing "National Security" values because the user never discussed politics).
User Reaction: Users felt a mix of "deep discovery" and "exposure." Many expressed concern about commercial exploitation of these inferred profiles.

B. Embodiment (Persona Responses)

Performance: The Chat-persona (based on conversation) significantly outperformed the Schwartz-persona (based on survey) on personalized questions (77% alignment vs. 72%), particularly in capturing tone, style, and specific nuances.
Style vs. Stance: Users valued the AI matching their tone (e.g., hedging, specific vocabulary) but were critical when the AI sounded "too calculated" or failed to sound like a real person.
Self-Reflection: Users utilized the personas as mirrors for self-improvement, identifying habits they wished to change (e.g., filler words, avoidance patterns).

C. Explanation (Value Charts & Reasoning)

Quantitative Alignment: There was moderate alignment between LLM predictions and manual surveys (Spearman $\rho \approx 0.58$ ). About 63.6% of predictions fell within $\pm 1$ Likert point of the user's self-report.
Biases: The LLM tended to overestimate "Self-Direction" and underestimate "Tradition" and "Power."
Automation Bias: A critical finding was that persuasive explanations could induce users to doubt their own values. When the AI provided a fluent, evidence-based rationale for a value the user didn't hold, some users shifted their self-assessment to match the AI (e.g., accepting the AI's interpretation of "Security" over their own).
Surprising Accuracy: In some cases, the AI's inference was more accurate than the user's survey response, catching latent tensions or self-deceptions the user hadn't consciously acknowledged.

5. Significance and Design Implications

The paper argues that as AI becomes more ubiquitous, the focus must shift from "Can AI understand us?" to "How do we govern AI's understanding of us?"

Three Design Principles for VACAs:

Extraction + Consent: Value inference must be visible, bounded, and revocable. Systems must explicitly handle the privacy of third parties mentioned in conversations (shadow profiles).
Embodiment + Nuance: AI personas should serve as tools for self-direction (trying on futures, rehearsing trade-offs) rather than static mimicry. They should highlight change and growth rather than reifying temporary states as permanent traits.
Explanation + Friction: To prevent automation bias, AI explanations must include friction. This includes displaying uncertainty, offering contrastive rationales ("here's why I think X, but here's why not"), and linking back to raw evidence so users can contest the AI's conclusions.

Conclusion:
The study concludes that while AI is approaching a threshold where it can model human values with surprising accuracy, this capability carries the risk of "weaponized empathy." The authors advocate for a future where AI systems are designed not just to be helpful, but to preserve human agency, ensuring that value-aligned companions support self-discovery rather than dictating identity.