Technical Summary: Evaluation Drift in LLM Personality Induction

Problem Statement

The paper investigates the reliability of Large Language Models (LLMs) in expressing stable, human-like personalities. While prior work has attempted to induce personality traits into LLMs for purposes such as narrative coherence, trust, and controllability, a critical gap remains: evaluation drift. Previous studies often rely on prompt-based methods or single-dimension assessments that suffer from high statistical variance and surface-level mimicry. The authors question whether LLMs possess a stable underlying personality profile or merely mimic surface cues, noting that existing evaluation protocols often move the "goalpost" by assessing individual Big Five traits (Openness, Conscientiousness, Extraversion, Agreeableness, Neuroticism) in isolation, which ignores the multi-dimensional nature of personality.

Methodology

The study employs a two-phase methodology to evaluate personality induction across five distinct models (ranging from 2B to 175B parameters, including Gemma, LLaMA, and GPT-3.5):

1. Dataset: The authors utilize the Essays Dataset (Pennebaker and King, 1999), comprising 2,467 long-form essays (approx. 1.9 million words). This dataset was chosen over shorter social media texts (e.g., Twitter, Reddit) to capture stable personality traits rather than transient moods. Each essay is associated with binary labels for the five OCEAN traits.
2. Prompt Stability Assessment: To address evaluation fragility, the authors first tested three prompt variations (S1: Numeric, S2: String, S3: Alphabet) on pre-trained and fine-tuned models. They measured the standard deviation of responses to the IPIP-NEO questionnaire to determine if fine-tuning reduces sensitivity to prompt rephrasing.
3. Fine-Tuning Strategies:
   • Supervised Fine-Tuning (SFT): Models were trained on essays paired with target personality labels. Two variations were tested: (1) essays only, and (2) essays augmented with portions of the IPIP-NEO questionnaire to explicitly link text to trait scores.
   • Preferential Fine-Tuning (DPO/ORPO): The authors constructed parallel datasets where "chosen" and "rejected" responses were generated based on mismatched OCEAN profiles (e.g., contrasting a sample with its inverse or near-inverse) to simulate human preference signals.
4. Evaluation Protocol: The primary metric was the exact match rate of the full 5-dimensional OCEAN vector (32 possible binary combinations) against the ground truth. This was compared against a random baseline (3.125%). The study also compared censored (safety-aligned) vs. uncensored model variants to assess the impact of security alignment.

Key Contributions and Results

1. Reduction of Evaluation Variance (RQ1)

The study demonstrates that fine-tuning (both SFT and preferential methods) consistently reduces the statistical variance in questionnaire responses across all tested models.

• Finding: Fine-tuned models exhibited a 15% to 33% reduction in standard deviation compared to pre-trained models when subjected to prompt rephrasings.
• Implication: This confirms that supervised signals stabilize the model's response to psychological assessments, mitigating the "evaluation fragility" reported in pre-trained models.

2. Failure of Full-Vector Induction (RQ2)

Despite improved stability, the models failed to accurately induce the target multi-dimensional personality profiles from unguided essays.

• Finding: The accuracy for the full 5-dimensional OCEAN vector remained near the random baseline (~3.13% to 9.38%), even when single-trait metrics appeared to improve. The best result achieved was 3/32 cases (9.38%).
• Analysis: The authors argue that prior work reporting high accuracy on single traits is misleading because it ignores the joint space of personality. A model can be correct on 4/5 traits but fail the holistic profile due to a single systematic error (e.g., misinterpreting "aspirational sociality" as Extraversion or negative task affect as low Conscientiousness).
• Conclusion: Unguided essays lack the necessary cues for faithful, multi-dimensional personality expression.

3. Impact of Security Alignment (RQ3)

The study compared uncensored versions of models (LLaMA 3.1-8B, Gemma-2-2B, LLaMA 3.2-3B) against their safety-aligned counterparts.

• Finding: No significant performance gains were observed in uncensored models regarding personality induction accuracy.
• Conclusion: Security alignment is not a primary bottleneck for personality induction efficacy in this context.

Significance and Claims

The paper claims that the current paradigm of evaluating LLM personality induction is fundamentally flawed if it relies on single-dimension metrics or assumes that surface-level mimicry equates to a stable profile.

• Reframing the Goal: The authors argue that for LLMs to be considered as having a "personality," they must reproduce the full multi-dimensional vector consistent with human testing protocols (like IPIP-NEO), not just isolated traits.
• Limitations of Current Data: The results suggest that unguided text (essays) is insufficient for learning complex personality vectors. The authors propose that future work requires scenario-grounded datasets or interactive elicitation methods that accumulate evidence over time to reliably assess personality.
• Caution on Metrics: The paper warns that reported successes in single-trait induction may be artifacts of the evaluation method rather than genuine personality modeling, as simple bag-of-words approaches can achieve similar single-trait accuracy without true semantic understanding.

In summary, while fine-tuning stabilizes LLM responses to psychological questionnaires, it does not currently enable the reliable induction of coherent, multi-dimensional human personalities from unguided text. The authors advocate for a shift toward more rigorous, multi-dimensional evaluation standards and richer, context-aware data sources.