Can LLMs Assess Personality? Validating Conversational AI for Trait Profiling

1. Problem Statement

Traditional personality assessment relies heavily on static, self-report questionnaires (e.g., IPIP-50). While established, these methods suffer from significant limitations:

• Lack of Nuance: They fail to capture the fluidity, context, and dynamic nature of human behavior.
• Bias: They are prone to social desirability bias and response fatigue.
• Static Nature: They do not allow for adaptive probing when responses are ambiguous.

While Large Language Models (LLMs) have shown promise in inferring personality from static text corpora (e.g., social media posts), there is a lack of validation regarding real-time, guided conversational assessment against gold-standard psychometric instruments, particularly with explicit measures of user-perceived accuracy.

2. Methodology

The study employed a within-subjects experimental design ($N=33$) to compare two assessment methods for the Big Five personality traits (Openness, Conscientiousness, Extraversion, Agreeableness, Neuroticism).

A. Participants

• Sample: 33 participants recruited via social media and university channels.
• Demographics: Predominantly young, educated, and tech-savvy (limiting generalizability but ensuring baseline AI literacy).
• Procedure: Participants completed the study via a custom web application involving informed consent, LLM conversations, the IPIP-50 questionnaire, and accuracy ratings.

B. Assessment Methods

1. Gold Standard: The IPIP-50 questionnaire, a validated 50-item self-report inventory (10 items per trait) using a 5-point Likert scale.
2. Experimental Condition (LLM): A guided conversational assessment using Gemini 2.5 Flash.
   • Question Battery: 20 open-ended questions (4 per trait) designed to elicit behavioral descriptions rather than hypothetical self-evaluations.
   • Prompt Engineering:
     • Conversational LLM: Received dynamic system prompts to guide the dialogue, probe deeper into ambiguous answers, and terminate the conversation once sufficient behavioral indicators were gathered (max 10 user messages).
     • Scoring LLM: A separate LLM instance analyzed all 20 transcripts collectively to generate trait scores on a 0–120 scale (normalized to 0–100). This separation prevented anchoring effects.
   • Design Principle: Questions focused on specific past behaviors (e.g., "Tell me about the last time...") rather than hypothetical scenarios to minimize social desirability bias.

C. Analysis Plan

• Statistical Tests: Paired-samples t-tests to compare mean scores between methods; Pearson correlations to assess convergent validity.
• Advanced Analysis: Principal Component Analysis (PCA) and Hierarchical Clustering to examine the structural relationship between the two methods and participant subgroups.
• User Perception: Likert-scale ratings (1–5) for the perceived accuracy of each trait and overall method preference.

3. Key Results

A. Convergent Validity (Statistical Correlation)

The study found moderate convergent validity between the LLM and IPIP-50 scores ($r = 0.38 – 0.58$).

• Strongest Correlation: Extraversion ($r=0.58$).
• Weakest Correlation: Agreeableness ($r=0.38$).
• Statistical Equivalence: For Conscientiousness ($p=0.084$), Openness ($p=0.485$), and Neuroticism ($p=0.441$), there were no statistically significant differences between the mean scores of the two methods.
• Significant Differences: Agreeableness ($p=0.002$) and Extraversion ($p=0.036$) showed statistically significant differences, suggesting these specific traits require further calibration in conversational models.

B. User-Perceived Accuracy

• Participants rated both methods as equally accurate (Mean $\approx$ 4/5).
• There were no statistically significant differences in perceived accuracy between the LLM and IPIP-50 for any trait ($p > 0.05$).
• However, the LLM ratings showed wider interquartile ranges, indicating higher variability in how users judged the AI's accuracy compared to the questionnaire.

C. Structural Analysis

• Clustering: Hierarchical clustering showed that for most traits, IPIP and Chat scores clustered together. However, Chat-derived Conscientiousness scores clustered closer to Extraversion than to IPIP Conscientiousness, indicating method-specific variance in how the trait is captured.
• Demographics: No strong dependency was found between age and trait scores, though older participants were slightly more critical in their accuracy ratings.

4. Key Contributions

1. Validation of Real-Time Conversational AI: The study provides empirical evidence that LLMs can extract Big Five personality traits from guided, dynamic conversations with accuracy comparable to traditional questionnaires, outperforming previous static-text inference studies (which typically reported $r \approx 0.25–0.33$).
2. User Acceptance Framework: It demonstrates that users perceive conversational AI profiles as equally valid as traditional psychometric tools, a critical factor for adoption in consumer applications (hiring, dating, therapy).
3. Methodological Innovation: The introduction of a dual-LLM architecture (one for interaction, one for scoring) and a behavioral-focused question battery that avoids direct trait labeling to reduce bias.
4. Identification of Calibration Needs: The study highlights that while Openness, Conscientiousness, and Neuroticism are well-captured, Agreeableness and Extraversion require specific prompt engineering adjustments to align with self-report standards.

5. Significance and Limitations

Significance:
This research suggests a paradigm shift in psychometrics from static self-reporting to dynamic, naturalistic interaction. Conversational AI offers a more engaging, context-aware alternative that can potentially capture behavioral nuances missed by rigid questionnaires. The high user acceptance indicates a viable path for integrating AI into consumer-facing psychological profiling.

Limitations:

• Sample Size: $N=33$ is small; the study had power only for medium-to-large effects, potentially missing subtle differences (e.g., the non-significant $p=0.084$ for Conscientiousness).
• Population Bias: The sample was young and tech-savvy, limiting generalizability to broader demographics.
• Order Effects: All participants completed the Chat assessment before the IPIP-50, which may have influenced responses.
• Ground Truth: The validation relies on IPIP-50 (a self-report measure), which is itself subject to bias; true personality remains unobservable.
• Model Specificity: Results are tied to a specific LLM configuration (Gemini 2.5 Flash) and may vary across different architectures.

Future Work:
The authors recommend larger, diverse samples, counter-balanced study designs, test-retest reliability assessments, and comparative studies across different LLM architectures (e.g., GPT-5, Claude) to establish robust generalizability.