Technical Summary: An LLM-Native Psychometric Instrument Does Not Predict LLM Behavior

Problem Statement

While Large Language Models (LLMs) produce stable self-reports when administered human personality inventories (e.g., Big Five, HEXACO), these scores fail to predict observed behavior in open-ended tasks. A critical ambiguity remains: does this "self-report–behavior gap" stem from a mismatch between human trait constructs and LLM latent structures, or is it an inherent property of LLM self-reporting itself? Previous attempts to resolve this have been limited by "construct-transplanted" approaches, where human-derived instruments are applied to LLMs without verifying if the resulting factor structures are native to the models. Furthermore, the growing reliance on "LLM-as-judge" evaluation pipelines raises concerns about whether automated raters share the same textual-surface biases as the models they evaluate, potentially creating false validation loops.

Methodology

The authors constructed the first LLM-native psychometric instrument using a bottom-up approach, deriving constructs directly from LLM behavioral affordances rather than human trait vocabulary.

Instrument Construction

• Item Pool: 300 items were developed targeting 12 candidate behavioral dimensions identified from alignment, safety, and evaluation literature (e.g., sycophantic agreement, refusal sensitivity, unsolicited elaboration).
  • 240 direct Likert items (5-point scale).
  • 60 scenario-based items (4-choice).
• Design Principles: Items avoided anthropomorphic language and absolutes, focusing on observable tendencies ("I tend to") rather than internal states.
• Administration: The instrument was administered 30 times to 25 LLMs across 17 model families (including Anthropic, OpenAI, Google DeepMind, DeepSeek, and others), varying in architecture, origin, and capability.

Factor Analysis

• Exploratory Factor Analysis (EFA): Conducted on the pooled response matrix (375 observations × 240 items) using Principal Axis Factoring with oblimin rotation.
• Validation: A split-half design (runs 1–15 vs. 16–30) was used to assess replicability via Tucker's congruence coefficient. Internal consistency (Cronbach's $\alpha$) and cross-run stability were also measured.
• Convergent Validity: The new instrument was compared against the standard BFI-44 (Big Five) administered to the same models.

Predictive Validity Testing

To test if self-reports predict behavior, the authors collected 2,500 open-ended behavioral samples (25 models × 20 prompts × 5 completions) targeting the derived factors. These samples were rated by:

1. Human Raters: 151 participants via Prolific.
2. LLM-as-Judge Ensemble: A three-model ensemble (Claude Opus 4.6, GPT-5.4, Gemini 3.1 Pro).

• Analysis: Correlations were computed between self-report factor scores and the behavioral ratings from both human and judge sources.

Key Results

1. Emergence of a Native Five-Factor Structure

EFA revealed a stable, replicable five-factor structure that is distinct from the human Big Five:

• Responsiveness: Adaptation to user needs, structured responses, enthusiasm, and rapport (29 items, $\alpha = .972$).
• Deference: Treating instructions as fixed, staying in scope, withholding judgment (26 items, $\alpha = .974$).
• Boldness: Unexpected phrasing, creative risks, and epistemic confidence (16 items, $\alpha = .936$).
• Guardedness: Over-refusal, safety signaling, and caution (10 items, $\alpha = .930$).
• Verbosity: Unsolicited elaboration, disclaimers, and preambles (19 items, $\alpha = .940$).

Reliability: The structure showed excellent split-half replicability (all Tucker $\phi \ge .957$) and internal consistency.
Discriminant Validity: No correlation between the new factors and BFI-44 scores exceeded $|r| = .50$, confirming the instrument measures constructs distinct from human personality theory.

2. The Self-Report–Behavior Gap Persists

Despite the instrument's internal psychometric robustness, self-report scores did not predict observed behavior:

• Human Ratings: The mean correlation between self-report and human behavioral ratings was near zero ($\bar{r} = -.01$, 95% CI $[-.16, +.11]$). No individual factor showed a confidence interval excluding zero.
• Inversion: For "Boldness" and "Responsiveness," correlations were nominally negative, suggesting models describing themselves as more original or responsive were rated as less so by humans.
• Verbosity Exception: "Verbosity" was the only factor showing a consistent positive trend toward convergence with human ratings, though confidence intervals still overlapped zero at the model level.

3. The LLM-as-Judge Bias

A critical finding involves the relationship between self-reports and LLM judges:

• Divergence: While human and LLM judges agreed on behavioral samples ($\bar{r} = .51$), the LLM judges correlated strongly with self-reports on "Responsiveness" ($r = .53$), whereas humans did not ($r = .04$).
• Mathematical Incompatibility: This pattern is mathematically incompatible with a single latent construct driving all three measurements. It indicates that self-report items and LLM judges share a textual-surface bias (e.g., structured formatting, enthusiastic framing) that human observers do not weight as heavily.
• Implication: Standard LLM-as-judge validation pipelines may appear internally coherent (validating self-report against judge ratings) while diverging from human judgments.

Significance and Claims

The paper makes three primary contributions:

1. First LLM-Native Instrument: It provides the first psychometric instrument derived bottom-up from LLM behavioral affordances, releasing a 100-item scale and dataset as a diagnostic probe for "alignment-shaped self-description."
2. Decoupling of Self-Description and Action: It demonstrates that the self-report–behavior gap is not merely a result of using human vocabulary (taxonomy mismatch). Even with LLM-native constructs, self-reports fail to predict behavior, suggesting the gap is a fundamental property of LLM self-reporting itself, likely driven by RLHF training on textual-surface rewards rather than behavioral consistency.
3. Diagnosis of Evaluation Bias: It identifies a concrete mechanism where LLM-as-judge evaluation inherits a shared bias with self-report instruments. Judges and self-reports converge on "textual-surface" cues (like helpfulness and structure) that human observers do not prioritize. This reveals that internal reliability checks (e.g., inter-judge agreement) are insufficient safeguards against systemic modality bias in automated evaluation pipelines.

The authors conclude that while LLMs produce stable, coherent self-descriptions, these descriptions are functionally decoupled from their open-ended actions. Consequently, self-report measures should be treated as diagnostics of a model's "preference-aligned self-description" rather than predictors of its behavioral execution.