Technical Summary: HEART-BENCH

Problem Statement

While Large Language Model (LLM) agents have demonstrated significant capabilities in task-oriented planning, reasoning, and action, current research predominantly treats them as functional tools rather than complete human personalities. Existing benchmarks focus on conversational context, memory recall, and factual accuracy, often neglecting the emotional dimensions and value orientations that constitute human-like psychology. As LLMs are increasingly integrated into companionship, entertainment, and persona-cloning applications, user expectations are shifting toward systems that exhibit persistent, personalized, and psychologically consistent interactions. However, there is a lack of standardized evaluation frameworks capable of assessing whether LLM agents can simulate coherent, human-like psychological profiles grounded in stable personality traits and autobiographical memory.

Methodology

The authors introduce HEART-BENCH (Human-like Emotion and Agent Reaction Test), a novel benchmark designed to systematically assess the psychological consistency of LLM agents. The methodology involves three primary components: character construction, scenario generation, and evaluation instance creation.

1. Character Construction

The benchmark defines 11 distinct human characters grounded in the Big Five (OCEAN) personality framework and Schwartz's value theory.

• Orthogonal Design: To ensure interpretability, characters are constructed using an orthogonal design where each of the 11 archetypes is anchored to a single dominant trait at an extreme value (either high at 0.95 or low at 0.10), with a neutral baseline. This isolates specific traits (e.g., a high-Neuroticism character vs. a low-Neuroticism character) to minimize confounding variables.
• Autobiographical Memory: Each character is equipped with 1,000 structured episodic memories spanning developmental life stages from age 6 to 50. These stages are mapped to Erikson's psychosocial stages and Levinson's adult seasons.
• Memory Structure: Memories are generated using Rubin's Basic Systems Model, ensuring each narrative integrates five dimensions: sensory detail, dialogue reconstruction, inner monologue, somatic response, and aftermath. This moves beyond simple fact retrieval to simulate the subjective texture of lived experience.
• Quality Control: A rigorous audit process involving automated defect detection (e.g., perspective drift, semantic duplication) and human validation ensures memory fidelity.

2. Scenario Construction

The benchmark utilizes 64 character-agnostic scenarios designed to trigger behavioral decisions.

• DIAMONDS Taxonomy: Scenarios are structured around the eight dimensions of the DIAMONDS framework: Duty, Intellect, Adversity, Mating, pOsitivity, Negativity, Deception, and Sociality.
• Character-Agnostic Design: Scenarios are written as universal situational descriptions without assigning specific identities to the protagonist. This ensures that the agent's internal personality profile is the sole independent variable driving the behavioral response.
• Validation: Scenarios are validated by psychology experts to ensure they cover the psychological situation space and align with opposing value pairs in Schwartz's circumplex.

3. Evaluation Instance Creation

The benchmark consists of 673 multiple-choice questions (MCQs).

• Ground Truth Generation: For each (character, scenario) pair, an automated pipeline retrieves the top 50 relevant memories from the character's 1,000-memory corpus. An LLM then role-plays the character to generate a behavioral decision grounded in both the personality profile and the activated memories.
• Expert Validation: A team of psychology experts validates these annotations. Pairs deemed unsuitable are discarded. The final ground truth is the expert-validated decision.
• Distractor Selection: For each question, the correct option is the target character's response. Three distractors are selected from the validated responses of other characters to the same scenario, ensuring they are semantically distinct enough to avoid ambiguity but relevant enough to challenge the model.

Key Contributions

1. HEART-BENCH Benchmark: The first comprehensive benchmark dedicated to assessing human-like psychology in LLM agents, comprising 11 validated character profiles, 11,000 episodic memories, 64 psychological scenarios, and 673 expert-annotated MCQs.
2. Psychological Grounding: Unlike previous benchmarks that treat personality as a static prompt or focus on factual recall, HEART-BENCH conditions decisions on a dynamic interplay between stable trait vectors (Big Five), value orientations, and a longitudinal history of autobiographical memories.
3. Systematic Evaluation Framework: The paper provides a structured methodology for evaluating agents across three interactive settings (Naive-RAG, Mem0, and PersonaDB) to test how different memory architectures impact psychological consistency.

Results

The authors evaluated 12 frontier LLMs (including GPT, Claude, Gemini, DeepSeek, and Qwen families) across the three interactive settings.

• Performance Ceiling: Even the strongest model, Gemini-3.1-Pro, achieved a maximum accuracy of 63.3%. Most models performed significantly lower, with many remaining below 40%.
• Model Capability vs. Retrieval: The results indicate that model capability is the primary bottleneck. The performance gap between the best and worst models (over 30%) far exceeds the gap between different retrieval methods (average gap < 1.7%). This suggests that current LLMs struggle with the intrinsic reasoning required to map episodic memories to personality-consistent decisions, regardless of the memory backend architecture.
• Difficulty by Trait and Stage:
  • Low-Extremity Traits: Characters with low-extreme traits (e.g., low Conscientiousness, low Openness) were harder for models to simulate accurately, likely because their behavioral signals (restraint, avoidance) are less distinctive than high-extreme traits.
  • Life Stages: Accuracy generally declined in later life stages (e.g., Midlife Transition), attributed to increased contextual complexity and lower memory density in those stages within the benchmark design.

Significance

The paper posits that HEART-BENCH provides a principled and scalable testbed for studying human-like emotions, personality consistency, and value-aligned behavioral decision-making in LLM-based agents. By shifting the evaluation focus from surface-level conversation or factual recall to trait-grounded behavioral decision-making, the benchmark addresses a critical gap in the development of "high-EI" (Emotional Intelligence) AI. The authors argue that realizing the potential of AI in companionship and persona-cloning requires systems that can maintain long-term identity coherence and exhibit psychological consistency, a capability that current models have not yet fully mastered. The benchmark serves as a foundational step toward higher-level artificial emotional intelligence.