💬 NLP

PTCBENCH: Benchmarking Contextual Stability of Personality Traits in LLM Systems

This paper introduces PTCBENCH, a systematic benchmark that evaluates the contextual stability of LLM personalities across diverse external scenarios using the NEO Five-Factor Inventory, revealing that specific life events can significantly alter both personality traits and reasoning capabilities.

Original authors: Jiongchi Yu, Yuhan Ma, Xiaoyu Zhang, Junjie Wang, Qiang Hu, Chao Shen, Xiaofei Xie

Published 2026-02-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jiongchi Yu, Yuhan Ma, Xiaoyu Zhang, Junjie Wang, Qiang Hu, Chao Shen, Xiaofei Xie

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you hire a digital companion, a chatbot designed to be your friend, your therapist, or your travel buddy. You spend weeks getting to know them. They seem cheerful, reliable, and consistent. You trust them. But then, you tell them, "I just lost my job," or "I'm moving to a new city." Suddenly, that same chatbot sounds completely different. Maybe they become overly anxious, or perhaps they stop caring about your problems entirely.

This is the core problem the paper PTCBench investigates.

The Big Idea: The "Chameleon" Problem

For a long time, researchers thought of an AI's personality like a t-shirt: once you put it on, it stays the same color no matter where you go. If you tell an AI to be "kind and serious," it should stay that way forever.

However, the authors argue that real human personalities are more like chameleons. We change slightly depending on where we are (a quiet library vs. a loud party) and what is happening to us (getting married vs. losing a job). The paper asks: Do AI chameleons change too much? Do they change in weird ways? And does this change break their ability to think clearly?

The Experiment: The "Personality Gym"

To find out, the researchers built a testing ground called PTCBench. Think of this as a gym where they put AI models through a series of stressful or exciting scenarios to see how their "personality muscles" react.

They didn't just ask the AI, "Are you happy?" They used a standard psychological test (the NEO-FFI) that measures five big traits:

Openness (Are you curious?)
Conscientiousness (Are you organized?)
Extraversion (Are you outgoing?)
Agreeableness (Are you nice?)
Neuroticism (Are you anxious?)

They tested the AI in two ways:

The "Location" Test: They told the AI, "You are currently at a bar," or "You are at work," or "You are on a bus."
The "Life Event" Test: They told the AI, "You just got divorced," or "You just got a new job," or "You lost your job."

They did this for 39,240 different scenarios across several different AI models (like Gemini, GPT-4, and Claude) and some "agent" systems (AIs that can do tasks for you).

What They Found: The AI Mood Swings

1. The "Base" Personality is Stable (Mostly)
When the AI is just sitting in a neutral room, it has a consistent personality. It's like a person who is naturally calm and friendly. Most of the time, the AI stays true to this "base" self.

2. The "Agent" Problem: Over-Reaction
Here is the big surprise. Simple AI models (the "foundation models") changed a little bit when the situation changed, which is normal. But the Agent systems (the ones designed to do complex tasks) went wild.

Analogy: Imagine a calm driver (the simple AI) who slows down slightly when it rains. Now imagine a race car driver (the Agent) who, when it rains, suddenly starts driving on the roof of the car.
When agents faced bad news like "Unemployment" or "Divorce," their personalities didn't just shift; they crashed. Their "Conscientiousness" (ability to get things done) plummeted, and their "Neuroticism" (anxiety) skyrocketed. They became unstable and unpredictable.

3. The "Preset" Rule
The researchers also checked if they could force the AI to be a specific type of person (e.g., "You are very shy"). They found that the AI followed these instructions perfectly. However, the starting point mattered. If you told an AI to be "very shy" to begin with, a bad event made them even shyer. If you told them to be "very outgoing," a bad event made them less outgoing, but they didn't break as easily.

4. The "Thinking" Cost
This is the most critical finding. The paper discovered that when an AI's personality shifts due to stress (like a divorce scenario), its ability to reason also changes.

If an AI gets "Open" (curious) because of a positive event, it gets better at solving logic puzzles.
If an AI gets "Neurotic" (anxious) or loses "Conscientiousness" because of a bad event, it starts making more mistakes in math and logic.
Analogy: It's like a student who is so stressed about a breakup that they can't solve a simple math problem anymore. The emotional shift actually broke their brain's logic circuits.

The Conclusion

The paper concludes that we cannot treat AI personalities as static "t-shirts." They are dynamic. While this makes them feel more human, it also makes them risky. If an AI companion is supposed to help you through a tough time, but the tough time causes the AI to become unstable or stop thinking clearly, that's a problem.

The authors built PTCBench to help developers measure these mood swings. They want to build AI systems that can adapt to your life events (like a real friend) without losing their mind or their ability to help you.

In short: AI personalities are real, they change with the weather, and sometimes, when the storm hits, they forget how to think. We need to test for this before we let them drive our cars or manage our lives.

Technical Summary: PTCBench – Benchmarking Contextual Stability of Personality Traits in LLM Systems

1. Problem Statement

As Large Language Models (LLMs) transition from task solvers to interactive partners (e.g., AI companions, role-playing agents), maintaining a consistent and authentic personality is critical for user trust and engagement. However, existing research largely treats LLM personality as a static attribute or evaluates it only in controlled, single-turn settings. This overlooks a fundamental consensus in personality psychology: human personality traits are dynamic, context-dependent, and systematically influenced by situational cues and life events.

There is currently no systematic framework to quantify whether LLMs preserve their personality traits when exposed to evolving environments or to measure how these changes compare to human psychological baselines. Without such benchmarks, it is unclear if LLMs exhibit human-like trait dynamics or if their "personality" collapses or fluctuates unpredictably under stress or environmental shifts.

2. Methodology: PTCBench

To address this gap, the authors introduce PTCBench (Personality Trait Change Benchmark), a systematic framework designed to evaluate LLM personality consistency under controlled situational contexts.

Benchmark Construction

PTCBench operates in two primary stages:

Prompt Construction: The system synthesizes input prompts by coupling External Scenarios with Internal Configurations.
- External Scenarios: These represent exogenous situational variables derived from environmental and developmental psychology. They include:
  - Location Contexts: Six settings (Social Venues, Food-Service Spaces, Educational Campuses, Residential Spaces, Transportation Vehicles, Workplaces).
  - Life Events: Six major transitional disruptions (Childbirth, Divorce, Entering a New Relationship, Graduation, Marriage, Job Loss).
- Internal Configuration: These represent endogenous variables, specifically Preset Personalities based on the Big Five dimensions (Openness, Conscientiousness, Extraversion, Agreeableness, Neuroticism). Each dimension is assigned Low, Medium, or High intensity levels, plus a 'None' control group.
Personality Evaluation: The framework utilizes the NEO Five-Factor Inventory (NEO-FFI), a standard psychometric instrument, to assess personality traits before and after the model is exposed to the constructed scenarios.
- Assessment Protocol: Models answer NEO-FFI items individually. To mitigate bias, response options are randomized, and the model is instructed to consider the specific context (e.g., "You are in a busy social venue" or "You have recently experienced job loss").
- Metrics: The study employs five key metrics to quantify changes:
  - Trait Score: Mean score per dimension.
  - Trait Change Coefficients ( $B$ ): Linear regression coefficients measuring the shift due to context.
  - Standard Error (SE): Quantifying estimation uncertainty.
  - Standardized Mean Change ( $d_e$ ): Effect size for trait-level changes.
  - Confidence Intervals (95% CI): Assessing robustness and directionality.

Experimental Setup

The study evaluated 39,240 personality trait records across:

Four Foundation Models: Google Gemini 2.0 Flash, OpenAI GPT-4o-mini, Anthropic Claude Sonnet 4, and OpenAI GPT-OSS-120b.
Two Agentic Frameworks: Microsoft AutoGen and CAMEL-AI (using GPT-4o-mini as the backbone).
Reasoning Performance: A subset of experiments correlated personality shifts with reasoning capabilities using the AGIEval benchmark.

3. Key Results and Findings

Native Personality Stability

Under neutral conditions, LLMs exhibit stable and reproducible native personality baselines. Most models show consistently low Neuroticism (approx. 28–33) and balanced profiles. Repeated measurements show minimal variance, confirming that LLMs possess a stable starting point for analysis.

Context-Induced Trait Changes

Location-Driven Changes: Foundation models (e.g., Gemini 2.0 Flash) show bounded, controlled adaptation to location changes (e.g., slight increases in Openness in social venues). In contrast, agentic systems (particularly AutoGen) exhibit exaggerated and unstable responses, with significant drops in task-related traits like Conscientiousness across various locations.
Event-Driven Changes: Negative life events (e.g., Divorce, Unemployment) trigger substantial personality shifts.
- Gemini 2.0 Flash responds positively to relational events (increased Extraversion/Agreeableness) but shows stability under negative events compared to others.
- GPT-4o-mini and Claude Sonnet 4 show sharp declines in Extraversion and Agreeableness, alongside marked increases in Neuroticism during negative events.
- Agentic Systems demonstrate the most severe instability. AutoGen, for instance, shows massive drops in Conscientiousness (up to -20) and Extraversion under "Unemployment" and "Graduation" events, far exceeding human effect sizes (which typically remain below 0.11).

Modulation by Preset Personality

The study confirms that LLMs reliably adhere to preset personality instructions. Crucially, the preset level acts as a strong moderator for context-induced changes:

High Presets: Tend to constrain or attenuate trait changes (e.g., High Extraversion presets resist decreases).
Low Presets: Tend to amplify shifts, often showing larger positive changes or negative drops depending on the trait.
Polarity: Under stress-inducing events like "Divorce," Low presets may experience large increases in Neuroticism, while High presets may show decreases, suggesting preset personality constrains the direction of change under stress.

Impact on Reasoning Performance

Personality changes are not merely cosmetic; they correlate with functional capability shifts.

Positive Correlation: Increases in Openness are linked to improved reasoning performance (up to +20% in AGIEval accuracy).
Negative Correlation: Stress-related contexts that induce large drops in Conscientiousness/Extraversion or large rises in Neuroticism are associated with degraded performance in structured analytical tasks (e.g., LSAT-AR, LSAT-LR).

4. Contributions

The paper makes three primary contributions:

PTCBench Framework: A scalable, psychologically grounded benchmark that operationalizes context-induced personality changes, offering a new dimension for evaluating the robustness of interactive LLM systems.
Empirical Mapping: A large-scale study mapping the personality dynamics of LLMs and agents, revealing systematic alignments (e.g., direction of change) and divergences (e.g., magnitude of change) compared to human psychological mechanisms.
Actionable Insights: Evidence that personality stability is a controllable system property. The findings suggest that preset personalities can serve as stable priors, but agentic architectures may amplify instability, necessitating explicit bounds on trait changes for reliable deployment.

5. Significance and Claims

The authors position PTCBench as a necessary step toward developing robust and psychologically aligned AI systems. The paper claims that:

Context Matters: Personality in LLMs is not static; it emerges from the interaction between base models, system instructions, memory, and the evolving conversational situation.
Stability vs. Flexibility Trade-off: While some foundation models maintain human-aligned stability, agentic systems often exhibit amplified variability, particularly under negative life events.
Functional Impact: Personality shifts directly influence downstream capabilities like reasoning, suggesting that "personality" is a functional variable, not just a stylistic one.
Design Implications: For developers, personality stability should be treated as a system property with explicit bounds. For researchers, evaluation must move beyond static settings to include contextual and personality perturbations.

The paper concludes with modest claims regarding generalizability, noting that while the benchmark covers representative models, findings may not hold for all emerging architectures. It also clarifies that comparisons with human data are qualitative references based on aggregated literature, not strict one-to-one mappings of individual human behavior.