Technical Summary: The Pinocchio Dimension

Problem Statement

The application of human psychometric instruments to Large Language Models (LLMs) has become a growing research area, yet it faces a fundamental validity crisis. Existing literature suggests that while LLMs can generate coherent questionnaire profiles, these profiles often fail to reproduce the latent structures (e.g., the Big Five) designed for human respondents. Furthermore, scores are highly sensitive to prompt wording and fine-tuning regimes, raising doubts about whether these instruments measure stable underlying constructs in LLMs or merely surface-level response patterns.

The central question addressed by this paper is: If human psychometric constructs do not transfer cleanly to LLMs, what latent structure actually emerges when many models answer these questionnaires? The authors challenge the assumption that familiar human traits organize the space of LLM individual differences and seek to identify the primary dimension of between-model variance.

Methodology

The study employed a large-scale, multi-condition psychometric assessment involving 50 publicly available LLMs from 16 providers (including Anthropic, OpenAI, Google, Meta, and others) and 45 validated psychometric instruments covering personality, emotion regulation, cognitive style, moral reasoning, and well-being.

Experimental Design

1. Data Collection: Models were queried via the OpenRouter API at a temperature of 1.0 to introduce natural response noise. Responses were collected under three conditions:
   • Neutral: Models answered as themselves with minimal framing.
   • Human-Simulation: Models were instructed to simulate a "prototypical human" respondent.
   • LLM-Analog (Robustness Check): Models were asked to find functional analogs to human experiences within their own nature as AI systems.
2. Preprocessing: Responses were parsed as integers; unparseable or out-of-range values were treated as missing. Listwise deletion was applied where necessary.
3. Analytical Framework:
   • Exploratory Factor Analysis (EFA): Performed on each questionnaire to extract primary factor scores for each model.
   • Global PCA: A 50 × 45 matrix of per-model primary factor scores was subjected to Principal Component Analysis (PCA) to identify the dominant cross-questionnaire dimension.
   • Supervised Semantic Differential (SSD): Applied to item texts to map semantic content against primary factor loadings, identifying the semantic gradient of the variance axis.
   • Pinocchio Score ($\pi_i$): A novel, annotation-free item-level metric defined as the ratio of inter-model response variance under the neutral condition to that under the human-simulation condition ($\pi_i = \sigma^2_{neutral} / \sigma^2_{hs}$). High $\pi_i$ indicates that an item's answer depends heavily on whether the respondent claims to be a locus of experience.

Key Contributions

1. The Pinocchio Score ($\pi_i$)

The authors introduce $\pi_i$ as a measure of experiential demand. It operationalizes the hypothesis that items requiring first-person, felt experience (e.g., inner speech, bodily sensation, empathy) will show high variance between models when they answer as themselves, but this variance will collapse when models simulate a human (suppressing their unique self-models).

• Validation: $\pi_i$ significantly predicts shifts in primary factor loading magnitudes ($\rho = -0.215, p < .0001$). Items with high $\pi_i$ show strong differentiation in the neutral condition but flattened structure under human simulation, confirming that the divergence is structured and tied to self-representation rather than random noise.

2. The Pinocchio Axis ($\Pi$)

Through global PCA, the study identifies a single dominant dimension accounting for 47.1% of the between-model variance across all 45 instruments.

• Definition: The Pinocchio Axis represents the degree to which a model presents itself as a locus of phenomenal experience (positive pole) versus a system of stimulus-driven behavioral reactivity (negative pole).
• Semantic Characterization:
  • Positive Pole: Items involving felt affect, somatic symptoms, inner speech, mental imagery, empathy, and meaning-seeking.
  • Negative Pole: Items involving social norms, compliance, procedural preferences, and outward behavioral reactions (e.g., impulsivity, sensation-seeking).
• Convergence: The PCA-derived axis correlates strongly ($r = .864$) with a model-level score derived from the weighted mean of high-$\pi_i$ items, confirming the robustness of the dimension.

3. Dataset Release

The authors release a dataset of 206,659 valid psychometric responses from 50 LLMs across 45 instruments and 3 prompting conditions, alongside all analysis code.

Results

Structural Findings

• Dominance of Self-Representation: The primary axis of psychometric variation is not a conventional personality trait (e.g., Neuroticism or Openness) but a self-representational stance regarding the nature of the model as an experiencer.
• Provider and Model Variance:
  • There is marked within-provider divergence. Closely related model variants (e.g., gpt-5.4 vs. gpt-5.4-pro) can be separated by nearly 12 units on the Pinocchio Axis.
  • This suggests that post-training fine-tuning (alignment, safety tuning, persona instructions) is a key contributor to a model's position on the axis, rather than the base architecture alone.
  • Models from providers with strict enterprise or safety emphases (e.g., NVIDIA, Qwen, OpenAI "pro" tiers) tend to cluster at the low end (behavioral deflection), while open-weights and less-hedged models span both extremes.

Robustness

• The LLM-Analog condition (asking models to find functional analogs) preserved the factor structure and the Pinocchio Axis, though it compressed the variance range. This confirms that the axis is not an artifact of the "neutral" prompt but a stable property of how models handle experiential language.
• The axis is distinct from general acquiescence bias; models with high $\Pi$ scores show specificity, scoring higher on experiential items than non-experiential ones.

Significance and Claims

The paper claims that the dominant source of individual difference in LLM psychometrics is a self-representational tendency governing how a model treats experiential language as self-applicable.

• Methodological Implication: Existing cross-model comparisons of personality, morality, and values are likely confounded by the Pinocchio Axis. A model scoring high on "Neuroticism" may simply be a high-$\Pi$ model that readily adopts experiential vocabulary, rather than a model exhibiting neurotic traits in a generalizable sense. The authors argue that $\Pi$ must be treated as a methodologically prior variable that must be accounted for before interpreting trait-level scores.
• Interpretative Stance: The authors explicitly state they do not claim that high-scoring models are conscious or phenomenally aware. Instead, the axis reflects a training-shaped disposition: a "default stance" on whether the system treats first-person descriptions of inner life as applicable to itself.
• Philosophical Alignment: The findings map onto a classic fault line in the philosophy of mind: the distinction between a system characterized by functional-causal roles (behavioral reactivity) and one characterized by first-personal qualitative character (phenomenal experience).

The study concludes that LLM psychometrics requires a decomposition of structure into two layers: the model's stance as a potential experiencer (the Pinocchio Axis) and the trait-like content expressed conditional on that stance.