Technical Summary: LEX-EC Framework for Zero-Shot LLM Personality Classification

Problem Statement

Large Language Models (LLMs) can assign personality labels (specifically Big Five Inventory traits) to authors based on text, yet the validity and interpretability of these inferences remain open problems. A plausible label does not guarantee a valid one; models may rely on label prevalence, demographic priors, or topical content shortcuts rather than genuine trait signals. This issue is exacerbated in black-box settings where standard interpretability methods (requiring access to weights, gradients, or activations) are inapplicable. Furthermore, prior work often aggregates multiple texts per author, leaving the performance of zero-shot inference on limited-length or single-text conditions poorly characterized.

Methodology: The LEX-EC Framework

To address these gaps, the authors introduce LEX-EC (Lexical Evidence-Channel Audit), a reusable black-box audit framework designed to distinguish marginal-distribution effects from trait-associated signals recoverable under restricted evidence. The framework operates through five integrated stages:

1. Prevalence and Agreement Diagnostics: Evaluates classification prevalence, item-level association, and chance-corrected agreement (Cohen's $\kappa$) to separate genuine signal from prediction-prevalence bias.
2. Controlled Lexical Ablation: A core component where topical and demographic content is masked. The framework retains a "preserved lexical layer" consisting of:
   • Function words (POS classes).
   • Affective terms (from Empath and NRC-EmoLex).
   • Cognitive-style vocabulary.
   • Structural elements (punctuation, whitespace).
   • Numerals are masked to prevent encoding of age or counts.
     This tests whether associations persist when content-bearing vocabulary is removed.
3. Prompt Sensitivity Analysis: Compares model outputs under different prompt framings (Basic, Basic+Explanation, and Linguistic+Explanation) to assess how instruction shifts affect self-explanations and predictions.
4. Cross-Condition Evaluation: The framework jointly evaluates classification performance across text-length gradients (micro, short, and middle-length), different genres, and multiple model snapshots.
5. Self-Explanation Audit: Manually codes model-generated justifications to determine if linguistic prompting shifts rationales toward stylistic/affective evidence versus demographic/topical content.

Experimental Setup

• Datasets: Three distinct genres were used to span text-length gradients:
  • Facebook Statuses: ~17 tokens (micro-text), balanced for trait distribution.
  • Graduate Student Introductions: ~89 tokens (short-text), containing demographic/biographical details.
  • Pennebaker-King Essays: ~672 tokens (middle-length), stream-of-consciousness writing.
• Models: Five closed, proprietary LLMs were evaluated (GPT-4o-mini, GPT-4o, o3-mini, GPT-5.4-mini, and Claude Haiku 4.5) to ensure findings are not artifacts of a single model snapshot.
• Task: Zero-shot prediction of binary high/low labels for the Big Five traits (Openness, Conscientiousness, Extraversion, Agreeableness, Neuroticism).

Key Results

1. Text Length and Genre Dependence

• Essays (Long-form): Produced the broadest associations across all five traits. However, even in this condition, chance-corrected agreement ($\kappa$) remained low (e.g., $\kappa \approx 0.12-0.22$), indicating that statistical detectability did not translate to reliable individual-level classification.
• Student Introductions (Short-form): Showed a narrower pattern, with Extraversion being the only trait with statistically detectable pre-ablation associations in a majority of conditions.
• Facebook Statuses (Micro-text): Yielded little stable evidence. Even in a trait-balanced sample, most associations were weak or non-existent, suggesting a lower bound for content or length required for reliable inference.

2. Impact of Lexical Ablation (Masking)

Masking topical and demographic content revealed heterogeneous dependencies on lexical evidence:

• Extraversion: In student introductions, the association weakened significantly after masking (e.g., $\rho$ dropped from 0.226 to 0.062 in one condition), suggesting a dependence on explicit lexical cues often found in self-presentation.
• Conscientiousness: In the essay corpus, this trait showed the largest attenuation after masking, indicating a strong reliance on content-bearing vocabulary.
• Openness and Agreeableness: These traits remained comparatively stable in the essay condition after masking, suggesting their signals may be recoverable from function words, affective terms, or cognitive-style vocabulary.
• Facebook: Pre-ablation associations largely disappeared after masking, consistent with a dependence on sparse lexical evidence that was removed.

3. Prompt Sensitivity

• Self-Explanations: Linguistic prompting successfully shifted the distribution of model self-explanations. Under a linguistic prompt, explanations citing linguistic/affective evidence increased (from 50% to 67.7%), while those citing demographic or topical reasoning decreased.
• Predictive Performance: While the content of explanations shifted, the predictive performance (correlation and agreement) did not show a consistent advantage for the linguistic prompt over the basic prompt across models.
• Faithfulness: The study notes that fluent self-explanations may not faithfully reflect the computation producing the prediction, serving only as aggregate behavioral outputs.

Significance and Claims

The paper claims that LEX-EC provides a novel application of lexical methods to black-box interpretability in personality labeling. Its primary contributions are:

1. Differentiation of Signal Sources: The framework successfully distinguishes between signals derived from broad topical content versus those recoverable from restricted psycholinguistic channels (function words, affect, cognition).
2. Evidence of Heterogeneity: It demonstrates that "personality inference" is not a unitary capability; evidential profiles vary significantly by genre, text length, and specific trait.
3. Audit Utility: By jointly evaluating prevalence, item-level agreement, lexical-channel persistence, and explanation content, LEX-EC reveals information invisible to conventional statistical measures alone.
4. Practical Implications: The results suggest that personality-labeling systems should be used with caution, particularly for short texts or single-status inputs, as they may rely on demographic shortcuts or topical cues rather than stable trait signals. The framework aims to narrow causal hypotheses and inform decisions regarding the conditions under which such systems should be deployed.

The authors remain modest in their claims, noting that the study does not prove the models are "thinking" in specific ways, but rather characterizes the behavioral outputs and the conditions under which specific lexical evidence channels support or fail to support trait associations.