💻 computer science

Dynamic User-State Modeling for LLM-Assisted Cultural Heritage Exploration: A Secondary Analysis of Semantic, Behavioral, and Psychometric Signals

This study proposes a dynamic user-state modeling framework that integrates semantic, behavioral, and psychometric signals to predict user experience metrics in LLM-assisted cultural heritage exploration, demonstrating that adaptive systems can effectively tailor conversational guidance to individual curiosity and interaction patterns.

Original authors: Hongjiang Li

Published 2026-09-30

📖 8 min read🧠 Deep dive

CC BY 4.0

Original authors: Hongjiang Li

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine walking through a vast digital museum where the exhibits are not just static images but living stories waiting to be explored. In this modern version of a cultural site, visitors can wander through collections, ask questions, and build their own paths through history. For years, researchers have tried to make these digital experiences better by adding artificial intelligence that can talk back, answering questions and suggesting what to look at next. The goal has always been to help people connect with the past, but a fundamental challenge remains: not everyone explores the same way. Some visitors are deeply curious and ask complex questions, while others prefer to be guided gently or may feel overwhelmed by too much freedom. A one-size-fits-all approach, where the AI speaks to everyone in the same tone and offers the same level of help, often misses the mark. It fails to notice when a visitor is drifting away from their interests or when they are struggling to find the next step.

This is where a new study from researchers at Xiamen University Tan Kah Kee College steps in. They asked a simple but difficult question: can we teach an AI to understand the changing state of a visitor's mind as they explore? Instead of just looking at the final score of how much a user liked the experience, the team wanted to see if they could predict a user's satisfaction and trust by watching how they interacted in real time. They focused on three specific things: the actual words the user typed and the answers the AI gave, the physical actions the user took like clicking and scrolling, and the user's own personality traits, specifically their natural level of curiosity. By combining these three streams of information, they built a model that treats every visitor as a unique individual with a shifting state of mind, rather than just a number in a survey.

To test this idea, the researchers analyzed data from 44 people who had previously visited a digital heritage site. These visitors had spent time exploring different historical themes, asking questions, and reading answers generated by a large language model. The researchers took this existing data and broke it down into tiny, meaningful pieces. They looked at how closely a question matched the specific exhibit the user was viewing at that moment, how much the conversation drifted away from the main theme, and whether the answers provided by the AI felt relevant and coherent. They also tracked how long the user spent on the site, how many exhibits they visited, and how quickly they moved from one action to the next. Crucially, they also included a measure of the user's curiosity, taken from a standard psychological inventory, to see if a person's natural desire to learn changed how they responded to the AI's help.

The results revealed that the way people interact with cultural AI is far more complex than simply choosing between a guided tour or a free exploration. The study found that the AI could successfully predict how much a user would enjoy the experience and how much they would trust the system, even before the session was over. This prediction worked not because the AI knew the user's name or age, but because it was watching the flow of the conversation and the user's behavior. For instance, the study showed that a user's physical behavior, such as how long they waited between clicks or how deeply they scrolled, was a strong indicator of how usable they found the system. In contrast, how well the AI's answers stayed on topic and followed the user's questions was a better predictor of whether the user felt the system was performing well.

One of the most interesting discoveries concerned the AI's habit of suggesting questions. In some versions of the site, the AI offered a list of recommended questions for the user to click on. The researchers found that when users chose these suggested questions, the conversation tended to stay more focused on the current exhibit and flowed more smoothly. The users felt more connected to the cultural content, and the answers provided by the AI were more consistent with the topic. However, the study also found that these suggestions did not necessarily make the exploration more novel or creative. Users who wrote their own questions often ventured into more unique territory, even if the conversation was sometimes less coherent. This suggests that recommended questions act like a safety rail, keeping the visitor on a clear path, but they do not necessarily encourage the kind of wild, creative discovery that some curious minds might seek.

The researchers also discovered that the level of help a user needs depends heavily on their natural curiosity. People who scored lower on curiosity scales tended to benefit more from the AI's suggestions and guidance. They found the pre-written questions helpful for getting started. However, for users with high levels of curiosity, the extra guidance did not make much of a difference. These users were already good at formulating their own questions and exploring on their own. This finding challenges the idea that every visitor needs the same amount of assistance. Instead, it suggests that the most effective AI systems would be those that can sense when a user is struggling to find a topic and offer help, but then step back when the user shows signs of deep, self-directed engagement.

Perhaps the most practical finding for the future of digital museums is that the AI does not need to wait until a session is over to understand the user. The study showed that by the middle of an interaction, the system had already gathered enough information about the user's behavior and the flow of conversation to make a reasonably accurate prediction of their experience. This means that in a real-world setting, an AI could potentially adjust its behavior while the user is still exploring. It could decide to offer more suggestions if it senses the user is lost, or it could stop offering help if it senses the user is in a flow state. The system does not need to know the user's entire history to make these adjustments; it only needs to watch the current conversation unfold.

The study also highlighted the importance of looking at the whole picture. If the researchers had only looked at the text of the conversation, they would have missed important clues about usability. If they had only looked at the user's personality, they would have missed the specific ways they interacted with the interface. It was the combination of the words, the actions, and the personality that allowed the model to see the full picture of the user's experience. This approach moves beyond simply asking users if they liked the site at the end. Instead, it treats the experience as a dynamic story that is being written in real time, with the AI acting as a co-author that can adapt its style to fit the writer.

While the findings are promising, the researchers are careful to note that this is a secondary analysis of a relatively small group of 44 people. The results are not a final proof that this method will work for everyone, but they provide strong evidence that it is possible to model these complex interactions. The study also points out that the AI's ability to stay on topic does not guarantee that the facts it presents are historically accurate; the model measures how well the words fit together, not whether they are true. Future systems would need to combine this understanding of user state with reliable fact-checking to be truly effective.

Ultimately, this research suggests a shift in how we think about artificial intelligence in cultural heritage. Instead of building a single, perfect guide for everyone, we can build systems that are sensitive to the individual. These systems would recognize that a visitor's needs change from moment to moment. They would understand that a moment of confusion is an opportunity to offer help, while a moment of deep focus is a signal to step back. By paying attention to the subtle signals of curiosity, behavior, and conversation, these digital guides could become more than just information machines. They could become partners in exploration, helping each visitor find their own unique connection to the past. The path forward is not about making the AI smarter in a general sense, but about making it more aware of the human sitting on the other side of the screen.

Technical Summary: Dynamic User-State Modeling for LLM-Assisted Cultural Heritage Exploration

1. Problem Statement

While Large Language Models (LLMs) enable open-ended information exploration in digital cultural heritage environments, current implementations often provide uniform conversational support. This "one-size-fits-all" approach fails to accommodate the heterogeneity of user behaviors, curiosity levels, and exploration strategies. Existing evaluations of AI in cultural heritage largely focus on aggregate outcomes (e.g., mean usability scores), which obscure why users interacting with the same AI assistance report divergent experiences. Furthermore, traditional semantic analysis often compresses evolving interactions into static similarity values, failing to distinguish between stable exploration, gradual semantic drift, or coherent expansion.

This study reframes LLM-assisted cultural heritage interaction as a dynamic user-state modeling problem. The core challenge is to determine whether the trajectory of user questions, LLM responses, observable interaction behaviors, and continuous psychometric characteristics can jointly predict individual differences in System Usability Scale (SUS) scores and Trust of Automated Systems Test Performance (TOAST-P) scores, independent of average interface differences.

2. Methodology

Data Source and Design
The study is a secondary computational analysis of an anonymized dataset from a prior user study [1] involving 44 participants (20 women, 24 men) who interacted with three interface conditions: a curated website, a free-text LLM interface, and a guided LLM interface with context-dependent question suggestions.

Primary Dataset: 88 LLM-assisted sessions (2 per participant) used for predictive modeling.
Secondary Dataset: 132 total sessions (including the curated condition) used for cross-interface analysis.
Outcomes: Predicted SUS (0–100) and TOAST-P (1–7) scores.

Feature Engineering
The framework integrates three complementary information sources:

Dynamic Semantic Interaction Representation (DSIR): A 28-dimensional vector derived from multilingual embeddings (BGE-M3) of user questions, LLM answers, Point of Interest (POI) content, and thematic paths. It captures seven interpretable descriptors:
- Local Contextual Relevance: Semantic alignment between question and current POI.
- Thematic Drift: Departure from the broader thematic path.
- Answer–Context Semantic Alignment (ACSA): Consistency between answer and POI.
- Question–Answer Alignment: Semantic correspondence between query and response.
- Semantic Novelty: Uniqueness of the question relative to previous turns.
- Follow-up Continuity: Development of the next question based on the previous answer.
- Culturally Constrained Semantic Extension (SES): Answers that are novel yet thematically consistent and non-redundant.
- Aggregation: Each descriptor is summarized by mean, standard deviation, maximum, and temporal slope.
Behavioral Signals: Seven features including session duration, number of explored POIs, click count, normalized scroll depth, total questions submitted, mean action interval, and exploration depth.
Psychometric/Profile Data: Continuous Curiosity and Exploration Inventory-II (CEI-II) scores, LLM familiarity, and general computer familiarity. Demographic variables (age, gender, education) were excluded from the primary model to avoid sparse categorical instability.

Model Architecture
A low-capacity adaptive gated architecture was employed to prevent overfitting on the small sample size (44 independent participants):

Source Projection: Semantic, behavioral, and profile vectors are projected into a shared 16-dimensional latent space.
Adaptive Gating: A routing mechanism computes normalized weights ( $\alpha_s, \alpha_b, \alpha_p$ ) for each source, allowing the model to dynamically allocate attention based on the session.
Multi-Task Learning: A shared hidden representation feeds into separate regression heads for SUS and TOAST-P, optimized jointly with L2 regularization.
Validation: Strict participant-grouped cross-validation (5-fold) ensures no data leakage, as all sessions from a single participant are kept entirely within either the training or test set.

3. Key Contributions

Dynamic Semantic Representation: Introduction of DSIR, which models cultural exploration as a temporally evolving semantic trajectory rather than a static topic overlap, capturing local relevance, drift, and continuity.
Leakage-Controlled Multi-Source Fusion: A framework that integrates semantic, behavioral, and continuous psychometric data while strictly preventing participant-level data leakage.
Differentiated Predictive Patterns: Demonstration that usability (SUS) and perceived system performance (TOAST-P) are driven by distinct interaction signals.
Early-Session Prediction: Evidence that substantial predictive information is available by the midpoint of an interaction session.

4. Results

Predictive Performance

The Adaptive Fusion model achieved the lowest errors: MAE = 5.74 for SUS and MAE = 0.49 for TOAST-P.
Compared to a "condition-only" baseline, the model reduced MAE by 21.6% (SUS) and 23.4% (TOAST-P), proving that individual variation exists beyond average interface differences.
Condition-Residual Analysis: Even after removing average condition means, the model explained 30% of within-condition SUS variance and 26% of TOAST-P variance.

Ablation and Source Analysis

Semantic vs. Behavioral: Removing behavioral features increased SUS MAE by 11.0%, while removing semantic features increased TOAST-P MAE by 16.3%.
- Conclusion: Behavioral patterns (action intervals, exploration depth) are primary predictors of usability, whereas semantic coherence (ACSA, thematic drift) is primary for perceived performance.
Dynamic vs. Static: Replacing static topic distributions with BGE-M3 improved results, but adding the DSIR dynamic features provided further gains, confirming that temporal structure adds value beyond encoder quality.
Fusion Strategy: Adaptive gating outperformed fixed fusion (Early/Late/Equal), though the margin was modest, suggesting the primary gain comes from the multi-source representation itself.

Specific Findings on Interaction Dynamics

Question Recommendations: In the guided condition, selected recommended questions showed higher local relevance and lower thematic drift compared to manually written questions. However, they did not significantly increase semantic novelty or semantic extension. Recommendations appear to scaffold coherence rather than uniformly boost exploration diversity.
Curiosity Interaction: The benefit of guided interfaces (vs. free-text) decreased as continuous curiosity (CEI-II) increased. However, this relationship was continuous and non-deterministic; high-curiosity users still showed variability in their response to guidance.
Early Prediction: Using only the first 50% of interaction events yielded performance close to full-session prediction (MAE increase of ~8–10%), indicating that user state can be inferred before session completion.

5. Significance and Claims

The paper claims to provide empirical support for shifting cultural AI systems from uniform conversational assistance to user-state-aware adaptation.

Individual Heterogeneity: The study demonstrates that users interacting with identical AI assistance exhibit systematic, predictable differences in experience based on their interaction trajectories and curiosity levels.
Methodological Shift: It argues for treating user experience as an outcome of an evolving interaction state rather than solely a function of interface membership.
Design Implications: The findings suggest that adaptive systems should monitor semantic drift and behavioral effort to tailor guidance. Recommendations may be most effective for users struggling to initiate or sustain exploration, while autonomous exploration may be preferable for highly curious users.
Modesty of Claims: The author explicitly states that these are predictive associations, not causal mechanisms. They acknowledge limitations including the small sample size (44 participants), the secondary analysis nature of the data, and the inability to fully disentangle thematic path content from interface conditions. They do not claim the gating mechanism is a decisive algorithmic advance, but rather that the domain-informed representation is the primary driver of performance.

The study concludes that while personalization holds promise, it requires careful representation of evolving user states, conservative statistical interpretation, and future prospective validation before deployment.