🤖 AI

Setoka: A Benchmark for Hierarchical User Understanding in Personalized Agents over Heterogeneous Data

This paper introduces Setoka, a novel benchmark grounded in cognitive and personality psychology that evaluates personalized agents' ability to perform hierarchical user understanding—from semantic and episodic memory to behavior patterns and personality traits—across heterogeneous data, revealing that current systems struggle significantly with tasks requiring the integration and abstraction of fragmented long-term information beyond simple fact retrieval.

Original authors: Lingyang Zeng, Guangze Chen, Kaichen Yu, Zhicheng Pan, Siyang Weng, Zirui Hu, Xiangyun Du, Hailin He, Rong Zhang, Chengcheng Yang, Kai Huang, Xuan Zhou

Published 2026-07-30

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Lingyang Zeng, Guangze Chen, Kaichen Yu, Zhicheng Pan, Siyang Weng, Zirui Hu, Xiangyun Du, Hailin He, Rong Zhang, Chengcheng Yang, Kai Huang, Xuan Zhou

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to get to know a new friend. You could just ask them, "What is your phone number?" or "What time is your meeting?" and write down the answer. That's easy; it's just grabbing a fact. But real friendship is deeper. It's noticing that your friend always laughs at the same type of jokes, or that they seem to get grumpy when it rains, or that they are the kind of person who plans their whole week in advance. These aren't facts you can just look up; they are patterns you have to piece together from many different conversations, texts, and moments over time.

In the world of artificial intelligence, we are building "agents" that act like personal assistants. Right now, these AI helpers are getting pretty good at the easy stuff: remembering your phone number or finding a meeting time in your calendar. But scientists are worried that these agents are terrible at the "deep" stuff. They struggle to understand who you really are—your habits, your personality, and your long-term behavior. To fix this, researchers need a way to test if an AI can actually "get" a human, not just read a file. This is where a new study comes in, trying to build a better test to see if AI can move from being a simple note-taker to becoming a true understanding companion.

The Problem: AI Has Amnesia for "Who You Are"

Think of your digital life as a giant, messy attic filled with different kinds of boxes. Some boxes are labeled "Messages," some are "Calendar Events," and others are "Social Connections." Right now, most AI assistants are like a librarian who can only pull out one specific box when you ask for it. If you ask, "What is Alice's phone number?" the AI finds the "Contacts" box and reads the number. Easy peasy.

But what if you ask, "Am I an extroverted person?" or "How often do I usually hang out with friends on weekends?" The AI can't just open one box. It has to look at your calendar to see group events, check your messages to see how active you are in chats, and scan your social connections to see how many friends you have. It has to connect the dots across different types of data to figure out a pattern. The problem is that current AI systems are really bad at this. They are great at finding facts but terrible at understanding the story behind the facts.

Enter Setoka: The Ultimate "Get to Know You" Test

To solve this, a team of researchers created a new benchmark called Setoka. You can think of Setoka as a giant, super-organized simulation lab. Instead of using real people's private data (which would be a privacy nightmare), the researchers built a "psychology factory."

Here is how their factory works:

The Blueprint: First, they use real psychological science to create 10 fake people. They don't just guess what these people are like; they use a special math model to make sure the personalities make sense. For example, if a fake person is very "outgoing," the math ensures they aren't also secretly "shy" in a way that real humans rarely are.
The Behavior: Next, they translate these personalities into daily habits. If a person is "outgoing," the system generates a realistic schedule where they talk to friends every other day, play multiplayer games daily, and watch sitcoms a few times a week.
The Messy Data: Finally, they turn those habits into a massive pile of digital clutter. They create thousands of fake text messages, calendar entries, and social network logs that span two months. This is the "heterogeneous data"—a mix of different file types that an AI would have to sift through.

Once the factory is running, they ask the AI 1,426 questions about these fake people. The questions are divided into four levels of difficulty, like a video game with increasing boss battles:

Level 1 (Semantic Memory): "What is Alice's phone number?" (Just find the fact).
Level 2 (Episodic Memory): "What was the user doing on the evening of April 20?" (Combine a calendar entry, a text message, and a location log to reconstruct one specific event).
Level 3 (Behavior Pattern): "How many times per week does the user socialize?" (Look at weeks of data and count the total).
Level 4 (Personality Trait): "How extroverted is the user?" (Look at all the social data, the work habits, and the leisure activities to guess a deep personality trait).

What They Found: The "Fact-Finder" vs. The "Mind-Reader"

The researchers tested 3 different AI brains (ranging from a powerful cloud model to a smaller one that could run on a phone) paired with 5 different memory systems (some that organize data like a graph, some like a simple list).

The results were a bit of a wake-up call for the AI world:

The Easy Stuff is Solved: When the question was just about finding a fact (Level 1), the AI did great. In fact, sometimes the AI didn't even need a special memory system; it could just look up the raw data like a database.
The Middle Ground is Hard: As soon as the AI had to combine a few pieces of information to remember a specific event (Level 2), the scores dropped.
The Deep Stuff is Broken: When the AI had to find patterns or guess personality traits (Levels 3 and 4), the performance crashed. The best systems only got about 24% of the personality questions right.

The study suggests that simply making AI "remember" more facts isn't enough. The current systems are like students who can memorize a textbook perfectly but fail the essay question because they can't connect the ideas. The researchers found that to understand a user, an AI needs to do more than just retrieve data; it needs to link different sources, aggregate them over time, and generalize to find hidden traits.

The Takeaway

Setoka shows us that we are still far from having AI that truly "knows" us. While our digital assistants are getting better at remembering our phone numbers and meeting times, they are still struggling to understand our habits and personalities. The study suggests that the next big breakthrough won't come from just giving AI more memory, but from teaching it how to weave together the messy, scattered pieces of our digital lives into a coherent picture of who we really are. Until then, our AI friends might know what we did yesterday, but they still don't really know us.

Technical Summary: Setoka – A Benchmark for Hierarchical User Understanding in Personalized Agents

1. Problem Statement

Personalized agents based on Large Language Models (LLMs) are increasingly deployed to assist users, requiring the ability to retrieve explicit facts and infer abstract personal characteristics. However, existing memory benchmarks primarily evaluate an agent's capacity to retrieve information explicitly stated in conversational histories. They fail to assess hierarchical user understanding, specifically the ability to integrate implicit evidence across heterogeneous data sources (e.g., unstructured text, structured records, graphs) to infer deeper user attributes like behavior patterns and personality traits.

Constructing a benchmark for this task faces four primary challenges:

C1 (Hierarchical User Understanding): Reasoning requirements vary from simple fact retrieval to complex personality inference, necessitating a framework that distinguishes these levels.
C2 (Realistic Persona Sampling): Synthetic personas must reflect coherent individuals where trait combinations follow real-world psychological correlations, rather than random bundles of traits.
C3 (Diverse Trait-Behavior Mapping): A single personality trait can manifest as diverse behaviors across different contexts; generation methods must avoid stereotypical mappings.
C4 (Scalable and Consistent Generation): Generating long-term, heterogeneous user data requires maintaining consistency across records and preserving behavioral constraints over time without context overflow.

2. Methodology

2.1 The Setoka Framework

Setoka defines a four-level hierarchy of user understanding, grounded in cognitive and personality psychology:

Semantic Memory (SM): Retrieval of explicit facts from single records (e.g., a phone number).
Episodic Memory (EM): Integration of records describing a specific event (e.g., "What happened on April 20?").
Behavior Patterns (BP): Aggregation of events over time to identify regularities (e.g., "How often does the user socialize?").
Personality Traits (PT): Generalization across multiple behavior patterns and event categories to infer latent dispositions (e.g., "How extraverted is the user?").

The framework formalizes the transition between levels using four operators: selection ( $\sigma$ ), linking ( $\lambda$ ), aggregation ( $\gamma$ ), and generalization ( $\rho$ ). Evidence sets form a nested chain ( $E_{SM} \subseteq E_{EM} \subseteq E_{BP} \subseteq E_{PT}$ ), where higher levels require broader evidence scopes and additional transformations.

2.2 Psychometrics-Based Generation Pipeline

To address the challenges of realistic and consistent data generation, Setoka employs a top-down pipeline:

Correlation-Aware Trait Sampling: Instead of sampling traits independently, the pipeline draws personality vectors (Big Five) from a multivariate Gaussian distribution. The covariance matrix is derived from meta-analytic psychometric correlations, ensuring that synthetic personas reflect realistic trait interdependencies.
Psychological-Scale-Guided Behavior Generation: To avoid stereotypical LLM generation, sampled trait vectors are mapped to behavior patterns using validated psychological scales (specifically the BFI-2). This converts trait scores into item-level responses, which are then translated into concrete, diverse behavior patterns.
Event-Grounded Generation Tree: To ensure long-term consistency, behavior patterns are expanded into a temporal hierarchy of events. Child events are generated sequentially to preserve continuity, while details are expanded in parallel. All heterogeneous records (logs, relations, messages) are derived from a single "source of truth" (the event) to maintain cross-schema consistency.
Query Construction: Testing instances $(q, a, E)$ are generated automatically based on the ground truth of the generation process. Queries are designed to isolate the specific reasoning operator required at each level (e.g., partial-cue recovery for EM, statistical aggregation for BP).

2.3 Experimental Setup

The authors instantiated Setoka with 10 synthetic user profiles containing approximately 1,600 heterogeneous records across 23 schemas and three data models (relational, graph, unstructured).

Models Evaluated: Three LLMs (DeepSeek-V4-Flash, Ministral 3 14B, Gemma 3 4B-IT).
Memory Systems: Five systems representing different paradigms: Graph-structured (Cognee, HippoRAG 2), Flat fact-based (Mem0), and Hierarchical (MemMachine, Letta).
Baseline: A DBQuery baseline where the LLM interacts directly with raw databases without intermediate memory construction.
Metrics: LLM-judge similarity scores for SM, EM, and BP; Kendall's rank correlation coefficient for PT (evaluating the correct relative ordering of users).

3. Key Results

Performance Decline with Abstraction: Accuracy decreases monotonically as the level of abstraction increases. The best scores dropped from 0.85 (SM) to 0.46 (EM), 0.28 (BP), and 0.24 (PT). Average scores followed a similar trend, falling from 0.37 to 0.07.
Memory Construction vs. Direct Retrieval:
- DBQuery outperformed memory systems on SM (0.85) by retrieving exact details from single records but performed poorly on EM (0.27) where multi-record integration was required.
- MemMachine (hierarchical) achieved the best EM score (0.46), attributed to its preservation of fine-grained details combined with temporal context retrieval.
- Graph-structured systems (Cognee, HippoRAG 2) generally outperformed others on BP and PT tasks, suggesting that linking related records aids in aggregating recurring behaviors into higher-level traits.
Answer Rate vs. Competence: A counter-intuitive finding was that answer rates increased with abstraction (72% for SM to 93% for PT) while accuracy decreased. This indicates that models are more likely to "guess" abstract traits when evidence is missing, whereas they abstain from answering concrete factual questions if evidence is absent.

4. Contributions and Significance

The paper claims the following contributions:

Identification of a Limitation: It highlights that existing benchmarks inadequately evaluate the multi-level user understanding required for personalized agents, particularly regarding heterogeneous data integration.
First Psychology-Grounded Framework: Setoka introduces the first framework spanning concrete memories to abstract characteristics, explicitly defining the evidence scope and reasoning operations for each level.
Psychometrics-Based Pipeline: It develops a scalable generation pipeline that produces coherent, traceable heterogeneous user data grounded in psychological theory, enabling level-specific queries with verifiable evidence.
Comprehensive Evaluation: Through the evaluation of 3 models and 5 memory systems, the paper demonstrates that simple fact retrieval is insufficient for deep user understanding.

Significance:
The authors conclude that current memory mechanisms are insufficient for tasks requiring cross-source integration and abstraction over long-term behavior. The findings motivate the design of memory systems capable of linking, aggregating, and generalizing user evidence beyond literal recall. Setoka serves as a testbed to drive progress in these areas, showing that while retrieval is improving, the ability to infer latent attributes from fragmented, heterogeneous data remains a significant unsolved challenge.