Adaptive Virtual Reality Museum: A Closed-Loop Framewor for Engagement-Aware Cultural Heritage

1. Problem Statement

Current Virtual Reality (VR) cultural heritage applications typically rely on static content delivery, where exhibit descriptions are pre-authored and presented identically to all users. This approach fails to account for real-time variations in user attention and cognitive load, leading to two primary issues:

• Cognitive Overload: Complex explanations are presented when users are moving rapidly or scanning the environment.
• Under-stimulation: Brief descriptions fail to satisfy users who are deeply focused and seeking detailed information.

While consumer VR headsets generate rich behavioral data (gaze, head motion, locomotion), these signals are rarely utilized to dynamically adjust content. Furthermore, integrating Large Language Models (LLMs) for real-time content generation in latency-sensitive VR environments remains an underexplored challenge.

2. Methodology

The authors propose a multimodal closed-loop adaptive framework that infers user engagement in real-time to modulate the complexity and length of textual explanations generated by an LLM.

A. System Architecture

The system operates on a continuous cycle: Sensing $\rightarrow$ Inference $\rightarrow$ Adaptation $\rightarrow$ Re-interaction.

• Hardware/Software: Implemented in Unity 6000.2.8f1, deployed on a Meta Quest 3 (Snapdragon XR2 Gen 2), with all core inference loops running locally for low latency.
• Content Generation: Uses GPT-4o via an asynchronous pipeline. Prompts are dynamically adjusted based on the inferred engagement state (e.g., "scholarly style with rich detail" for high engagement vs. "2–3 short bullet points" for disengagement).

B. Engagement Modeling (The Core Algorithm)

The system estimates "engagement" specifically as attentional stability and reading readiness using a three-stage pipeline:

1. Signal Pre-processing (90 Hz sampling):

   • Head Stability ($S_{head}$): Measures angular velocity. A threshold of 30°/s distinguishes scanning from focused fixation.
   • Gaze Dwell ($S_{gaze}$): Tracks raycast stability on exhibits. A 1.0-second dwell threshold distinguishes intentional focus from glances.
   • Locomotion Velocity ($S_{loco}$): Inversely maps speed to engagement. 1.2 m/s is the baseline; faster speeds reduce the engagement score.

2. Sensor Fusion:

   • Signals are normalized to [0, 1] and aggregated into a raw score ($E_{raw}$) using a weighted formula:
     $$E_{raw} = (0.75 \times S_{phys}) + (0.25 \times S_{read})$$
   • $S_{phys}$: Weighted average of Head Stability (35%), Gaze (30%), and Locomotion (35%).
   • $S_{read}$: Binary signal triggered if text is dwelled on for >2.0s.
   • Temporal Smoothing: Linear interpolation ($\alpha = 0.35$) is applied to model "affective inertia" and prevent rapid state oscillations.

3. Safety Gates (Logic Constraints):
   To prevent false positives during high-entropy states, the system enforces hard limits:

   • Locomotion Gate ($v > 1.2$ m/s): Caps inference at Neutral (users cannot read dense text while walking fast).
   • Run Gate ($v > 2.0$ m/s): Forces Disengaged state (running is incompatible with reading).

C. Experimental Design

• Context: A high-fidelity virtual reconstruction of the Berat Ethnographic Museum.
• Participants: $N=16$ (8 Adaptive, 8 Control), aged 13–34.
• Conditions:
  • Adaptive: Content depth changes in real-time based on the engagement classifier.
  • Control (Static): Medium-length, fixed content for all exhibits.
• Duration: 15-minute free exploration sessions.

3. Key Contributions

1. Novel Framework: A multimodal closed-loop system that fuses implicit behavioral signals (gaze, kinematics, locomotion) to infer user engagement in real-time without explicit input or calibration.
2. LLM Integration Pipeline: A proof-of-concept demonstrating how LLMs can be integrated into VR to dynamically adjust text complexity and length based on user attention, operating within consumer hardware latency constraints.
3. Empirical Insights: Initial data showing that implicit, pacing-aware personalization significantly increases behavioral engagement and content consumption in virtual museums.

4. Results

A between-subjects study (Mann–Whitney U tests) yielded the following findings:

• Behavioral Engagement:
  • Session Duration: Adaptive users stayed 2x longer (889s vs. 451s, $p=.003$).
  • Reading Time: Adaptive users spent 2.9x more time reading (473s vs. 164s, $p=.015$).
  • Content Consumption: Adaptive users were exposed to 2x more words (1515 vs. 745, $p=.007$).
  • Reading Events: Adaptive users triggered significantly more reading interactions (67 vs. 50, $p=.050$).
• Engagement State Distribution:
  • Highly Engaged Time: Nearly doubled in the adaptive group (24.3% vs. 13.6%).
  • Disengagement: Reduced by 37% in the adaptive group (6.2% vs. 9.9%).
  • Neutral/Passive Time: Reduced from 28.4% to 22.2%.
• Usability:
  • System Usability Scale (SUS) score was 84.3 (Excellent), indicating the adaptation did not disrupt the user experience.
  • Perceived presence and motivation to visit the physical museum trended higher in the adaptive group, though not statistically significant due to sample size.

5. Significance and Future Directions

• Feasibility of Implicit Adaptation: The study proves that consumer VR hardware can support sub-millisecond engagement inference and real-time content adaptation without breaking immersion.
• Flow State Maintenance: By matching content complexity to the user's current cognitive capacity (e.g., simplifying text when moving fast, expanding it when stationary), the system helps maintain the user in a "flow" state, reducing cognitive friction.
• Ethical Considerations: The paper raises questions about AI transparency in heritage contexts. While users did not question the AI-generated content, future work must address the need for disclosure and safeguards against hallucinations (e.g., using Retrieval-Augmented Generation).
• Future Work: The authors plan to investigate long-term learning effects, validate engagement proxies against ground-truth measures, and explore more complex multimodal foundation models for collaborative museum agents.

Conclusion: The paper demonstrates that an engagement-aware, closed-loop VR museum system significantly outperforms static systems in terms of user exploration time, reading depth, and active engagement, offering a scalable path toward personalized cultural heritage experiences.