Technical Summary: ActTraitBench

1. Problem Statement

While Large Language Models (LLMs) can convincingly simulate personas in explicit self-reports, they frequently exhibit a significant divergence between their stated traits and their actual behaviors in implicit decision-making scenarios. This phenomenon, termed the Knowledge-Decision Gap ($G_{KD}$), represents a profound semantic-pragmatic asymmetry.

Existing benchmarks fail to accurately measure this gap due to three critical methodological limitations:

1. Unverified Measurement Validity: Many frameworks rely on synthetically generated labels or LLM-based extraction without strict empirical human benchmarking, leaving the mapping between text/decisions and latent psychological states unverified.
2. Multi-dimensional Entanglement: Complex interactive scenarios (e.g., multi-agent debates) often force models to process multiple macro-traits simultaneously, introducing high-dimensional confounding noise that obscures specific psychometric facets.
3. Distributional Bias: Current evaluations suffer from uncalibrated distributional shifts, such as default compliance biases in models or systematic avoidance of negative traits by LLM-as-a-judge evaluators, leading to statistical misalignment with natural human empirical distributions.

2. Methodology: ActTraitBench

To address these issues, the authors propose ActTraitBench, a human-grounded evaluation framework designed to quantify personality consistency in LLMs. The framework operates through three core phases:

A. Data Construction and Paradigm Refinement

The framework utilizes a Triple-Iteration Loop to develop 11 micro-situational behavioral paradigms mapped directly to the BFI-2 (Big Five Inventory-2) psychometric facets.

• Design Principles: Scenarios are adapted from classic experimental psychology and behavioral economics, adhering to Lewin's Equation ($B = f(P, E)$) to isolate the persona variable ($P$) by rigorously controlling the environment ($E$).
• Two-Stage Prompting: Each test item employs a sequential flow:
  1. Quantitative Decision: The model outputs a numerical value (0–99) representing a behavioral inclination.
  2. Qualitative Attribution: The model provides a rationale for the decision.
• Human Grounding: Authentic behavioral responses are collected from human cohorts. A "LLM-as-a-judge" (GPT-5.4) scores responses based on strict criteria, and items are iteratively refined until they demonstrate statistically significant Spearman correlations ($\rho$) with human BFI-2 self-reports.

B. Distributional Calibration

To correct for scoring biases inherent in LLM judges, the authors implement a Quantile Mapping procedure. This non-parametric method transforms raw judge scores into calibrated scores aligned with human BFI-2 norms by:

1. Extracting the percentile rank of a raw score within the judge's distribution.
2. Anchoring this percentile to the corresponding interval in the human benchmark distribution.
3. Transforming the score to the expectation of human scores within that interval.

C. Mitigation Strategy: Chain of Cognitive Alignment (CoCA)

To mitigate the $G_{KD}$, the authors introduce CoCA, a plug-and-play, inference-time intervention inspired by Objective Self-Awareness Theory.

• Mechanism: Before making a decision, the model is forced to execute a structured, three-step self-reflective protocol within a rigid JSON schema:
  1. Identity Recognition: "Who am I?" (Self-assess assigned traits).
  2. Contextual Mapping: "Where am I?" (Identify which traits are salient in the current situation).
  3. Consistency Planning: "What should I do?" (Derive a strategy to align actions with knowledge).

3. Key Results

Experiments were conducted on 14 mainstream LLMs (including DeepSeek, Qwen, Claude, Gemini, and GPT-4o) compared against a human baseline.

The Knowledge-Decision Gap ($G_{KD}$)

• Pervasive Asymmetry: A significant gap exists between explicit self-reports and implicit decisions across all models.
• The Scaling Paradox: Contrary to the expectation that larger models would be more consistent, scaling often exacerbates the gap.
  • Small Models: Exhibit a "Neutrality Illusion" where both knowledge and decisions collapse to the mean (safe, non-committal responses), creating a superficial synchrony.
  • Frontier Models: Successfully escape neutrality but show systematic decoupling. Their implicit decisions diversify into nuanced tendencies, while explicit self-reports remain rigidly tethered to socially desirable extremes.
• Dimensional Vulnerabilities:
  • Neuroticism: Models claim high emotional stability (low scores) but exhibit high anxiety in crisis decisions (high scores), revealing a social desirability bias.
  • Openness: Models often act more radically and intellectually in decisions than their conservative self-reports suggest.

Role-Playing Experiments

When models are injected with specific personas (High/Low traits):

• Knowledge Compliance: Models accurately reflect the injected persona in self-reports (low deviation).
• Decision Deviation: Models fail to enact the injected logic in implicit behavioral tasks, showing substantial deviation from the target persona. This asymmetry persists even in "slow-thinking" reasoning models.

Effectiveness of CoCA

• Frontier Models: CoCA significantly reduces the $G_{KD}$ for top-tier models (e.g., Minimax-m2.5 improved by 40.58%, Claude-sonnet-4-6 by 36.63%). The reflection mechanism successfully binds semantic personas to pragmatic execution.
• Reasoning Threshold: The intervention is strictly bottlenecked by model capacity. Smaller models (e.g., Qwen3-8b) suffer performance degradation (worsening $G_{KD}$ by ~25%) because complex meta-cognitive prompts introduce context distraction and identity confusion rather than alignment.

4. Contributions

1. Human-Grounded Benchmark & Calibration: ActTraitBench provides 11 validated micro-situational paradigms with construct validity ensured by empirical human data and distributional calibration via Quantile Mapping.
2. Quantification of the Gap: The study rigorously quantifies the $G_{KD}$ across 14 models, revealing that scaling alone fails to resolve fundamental pragmatic inconsistencies and often increases divergence.
3. Inference-Time Mitigation: The introduction of CoCA demonstrates that psychological principles (self-awareness) can be leveraged to narrow the knowledge-decision gap in reasoning-capable models without retraining.

5. Significance

The paper argues that accurately measuring emergent psychological traits is critical for trustworthy AI, as latent behavioral tendencies influence user trust and alignment risks. By exposing the "personality illusion" and the "Value-Action Gap," ActTraitBench challenges the reliance on static self-reports. The findings suggest that current frontier models, despite their advanced reasoning capabilities, lack genuine behavioral consistency with their stated identities. The proposed CoCA strategy offers a scalable, training-free pathway to improve the reliability of persona-consistent LLM agents, provided the models possess sufficient cognitive bandwidth to process the reflective intervention.