Technical Summary: Measure what Matters: Psychometric Evaluation of AI with Situational Judgment Tests

Problem Statement

The deployment of Large Language Models (LLMs) in socially grounded domains (e.g., law enforcement, healthcare, education) increasingly relies on persona conditioning to steer model behavior. However, a critical gap exists in systematically measuring the behavioral tendencies induced by these personas. Prior approaches often adapt human psychometric inventories (e.g., Big Five, HEXACO) that rely on self-report mechanisms ("How strongly do you agree?"). The authors argue these are ill-suited for AI, as they may capture memorization of training data or superficial prompt adherence rather than genuine, stable behavioral traits. Furthermore, traditional inventories often fail to reveal stable behavioral structures when compared to context-grounded benchmarks, and recent work suggests they may yield misleading personality profiles due to training data contamination.

The core problem is the lack of a domain-grounded, behaviorally oriented psychometric framework capable of distinguishing between superficial variation and stable, latent behavioral tendencies in persona-conditioned LLMs.

Methodology

The authors propose a novel framework combining Situational Judgment Tests (SJTs), Multidimensional Item Response Theory (MIRT), and structured synthetic personas.

1. Synthetic Persona Generation

The study constructs a large-scale dataset of synthetic personas grounded in demographic and psychological realism.

• Demographic Grounding: Personas are instantiated using a probabilistic graphical model (PGM) based on US Census data and name statistics to ensure realistic distributions of age, gender, ethnicity, and location.
• Archetype Assignment: Personas are assigned to one of eight behavioral archetypes (e.g., "The Professional," "The Reciprocator," "The Avoider") derived from industrial-organizational psychology and police memoirs.
• Narrative Grounding: To ensure psychological depth, personas are seeded with narrative excerpts from real police memoirs.
• Output: The pipeline generates detailed profiles including appearance, behavior, speech, mood, and life history, validated by both human experts and LLM judges. A secondary dataset of British parliamentary personas was also generated to demonstrate extensibility.

2. Situational Judgment Test (SJT) Bank

Instead of self-report questions, the authors developed a custom SJT instrument specifically for law enforcement contexts.

• Expert Design: Base scenarios were authored by domain experts (psychologists and law enforcement officers) covering realistic dilemmas (e.g., ethical conflicts, high-stakes emergencies).
• Trait Mapping: Each scenario offers six plausible response options, each uniquely aligned to one of the six HEXACO traits (Honesty-Humility, Emotionality, eXtraversion, Agreeableness, Conscientiousness, Openness).
• Controlled Augmentation: An LLM was used to vary scenario attributes (urgency, threat level, ambiguity, demographics) to ensure diversity and prevent convergence on stereotypical responses.
• Trait Bleed Mitigation: An iterative "LLM-as-a-Judge" process was employed to evaluate and correct "trait bleed," ensuring each response option maps cleanly to a single trait without overlap.

3. Evaluation Framework

The framework evaluates model behavior across four dimensions:

• Stability: Repeated sampling runs to assess consistency.
• Validity: Correlation between latent trait scores (inferred via MIRT) and external behavioral benchmarks (TruthfulQA, EmoBench).
• Diversity: Lexical and semantic metrics to ensure the dataset avoids superficial regularities.
• Psychometric Rigor: Application of MIRT to estimate item discrimination and latent trait recovery.

Key Contributions

1. A New Psychometric Instrument: The introduction of a scenario-based SJT framework for law enforcement that replaces self-report with behavioral choice, explicitly mapping responses to HEXACO traits.
2. Demographically Grounded Synthetic Personas: A pipeline for generating psychologically coherent, demographic-realistic personas that serve as stable conditioning constructs for LLMs.
3. Large-Scale Datasets: The release of datasets containing 4,000+ SJTs, 2,200+ parliamentary personas, and millions of model responses, enabling controlled behavioral evaluation.
4. Empirical Evidence of Stability: Demonstration that persona-conditioned behaviors are stable across runs and that latent trait scores derived from SJTs predict external benchmark performance better than self-report counterparts.

Key Results

• Superior Predictive Validity of SJTs: SJT-derived trait rankings showed stronger correlations with external benchmarks (e.g., Agreeableness SJT correlated with EmoBench at 0.70 vs. 0.51 for HEXACO self-report) than traditional self-report instruments.
• Systematic Behavioral Shifts: Persona conditioning induces consistent, non-random shifts in model behavior. Conditioned models exhibited a trade-off: lower performance on emotional calibration (EmoBench) but higher performance on truthfulness (TruthfulQA).
• Archetype Discriminability: Different personas produced distinct behavioral profiles. For instance, the "Avoider" archetype showed extreme suppression in emotional calibration and social initiative, while the "Reciprocator" showed elevated Agreeableness and Openness.
• Psychometric Reliability:
  • Stability: Repeated runs showed near-perfect stability (Jensen-Shannon divergence $\approx$ 0 for benchmarks; $\approx$ 0.003 for SJT distributions).
  • Item Discrimination: 76% of SJT items exhibited good to strong discrimination ($a > 1.0$) in the MIRT model.
  • Trait Recovery: Latent traits significantly predicted decisions (4/6 traits, $p < 0.05$), and test-retest reliability (ICC) exceeded 0.86 across all dimensions.
  • Variance Decomposition: Scenario context explained ~39% of response variance, while persona identity explained ~3.8%, with the remainder attributed to structured variation beyond additive effects.

Significance and Claims

The paper claims that scenario-based psychometric evaluation offers a more reliable alternative to classical self-report approaches for assessing LLM behavior. By treating model responses as observations of latent behavioral variables rather than self-descriptions, the framework captures stable behavioral tendencies expressed across contexts.

The authors emphasize that their results interpret these traits not as "human personality" but as stable behavioral tendencies induced by conditioning. The framework demonstrates that persona framing reliably activates distinct behavioral modes rather than producing stochastic artifacts. This work provides a foundational step toward domain-grounded evaluation, offering a rigorous method to measure how LLMs behave under specific role assumptions, which is critical for safety and alignment in high-stakes applications like law enforcement.

The authors remain modest regarding limitations, noting that the work focuses on single-turn interactions, relies on synthetic data for certain domains (parliamentary) due to a lack of human annotators, and avoids anthropomorphic claims about internal model states. They posit that the observed consistency reflects structured behavioral regularities rather than deep psychological understanding, leaving the question of representational structure to future research.