From Head Yaw to HMD Pose: A Reproducible Benchmark of Subjective VR Outcome Recoverability

Technical Summary: From Head Yaw to HMD Pose: A Reproducible Benchmark of Subjective VR Outcome Recoverability

Problem Statement
Virtual Reality (VR) evaluation relies heavily on retrospective post-experience questionnaires (e.g., SUS, NASA-TLX, SPES, SSQ) to measure constructs such as usability, workload, spatial presence, and cybersickness. While standard and interpretable, these methods are limited by their retrospective nature, potential to interrupt the user experience, and inability to capture behavioral states as they unfold. Although prior research (e.g., Holzwarth et al.) established that head-yaw movements could serve as a proof-of-concept for estimating subjective states, it remains unclear whether richer Head-Mounted Display (HMD) pose representations can provide robust, reproducible behavioral signals for multiple standardized outcomes. Furthermore, existing studies often lack structured frameworks, open benchmark datasets, or rigorous validation protocols that prevent data leakage between participants.

Methodology
This study proposes a reproducible benchmark to evaluate the extent to which HMD-derived head-motion descriptors contain recoverable information about standardized subjective VR outcomes.

• Dataset: The research utilizes the open Full-scale Assembly Simulation Testbed (FAST) dataset, comprising 108 participants performing two VR assembly tasks (Structure A and Structure B). The dataset includes continuous 6DoF HMD tracking data and standardized questionnaire responses (SUS, NASA-TLX, SPES, SSQ).
• Feature Engineering: Instead of relying solely on head yaw, the authors formalized a framework extracting 46 HMD-derived descriptors organized into six feature families:
  1. Session Metadata: Duration, sampling rate.
  2. Translation: Spatial displacement and speed.
  3. Rotation: Orientation dynamics (yaw, pitch, roll).
  4. Exploration: Scanpath length, turning events, and entropy.
  5. Smoothness/Stability: Acceleration, jerk, and sway variability.
  6. Posture: Sustained head orientation and pitch ratios.
  • A single non-kinematic variable (prior VR experience) was also included.
• Prediction Targets: Questionnaire scores were converted into classification targets to ensure operational interpretability and stable class distributions under participant-grouped validation:
  • SUS: Acceptable (>70) vs. Not acceptable (≤70).
  • NASA-TLX: Low workload (<33) vs. Not low (≥33).
  • SPES: High spatial presence (≥4) vs. Not high (<4).
  • SSQ: Three classes (Negligible, Intermediate, Very problematic).
• Experimental Protocol:
  • Validation: A participant-grouped seven-fold cross-validation was employed to ensure that data from the same participant (both Build A and Build B) remained within the same partition (training, calibration, or testing) to prevent data leakage.
  • Modeling: The protocol included preprocessing, Pearson-based descriptor reduction, handling of class imbalance, and model selection (e.g., SVM, LightGBM).
  • Comparisons: The full HMD-pose framework was compared against baselines (majority class, duration-only) and reduced feature sets (yaw-only, yaw-dependent).
  • Generalization: Cross-build validation was performed to test if models trained on one assembly structure could generalize to the other.

Key Results
The study found that HMD head motion contains useful but construct-dependent behavioral information.

• Recoverability by Construct:
  • Spatial Presence (SPES): This was the most consistently recoverable outcome, achieving a best positive-class F1 of 0.556. It showed the most stable transfer across cross-build validation.
  • Cybersickness (SSQ): Showed modest three-class recoverability with a macro-F1 of 0.345. Performance was limited by the sparsity of the "very problematic" class and poor generalization in cross-build settings.
  • Usability (SUS) and Workload (NASA-TLX): These constructs showed substantially lower recoverability, with positive-class F1 scores of 0.259 and 0.234, respectively. The results for NASA-TLX were particularly unstable, with wide confidence intervals and poor cross-build transfer (dropping to near-zero in some directions).
• Feature Contribution: The full HMD-pose framework consistently outperformed duration-only, yaw-only, and yaw-dependent descriptor sets across all targets. However, no single feature family fully explained the outcomes; for SPES, performance relied on a combination of translation, rotation, exploration, and posture.
• Baseline Performance: Simple baselines (e.g., duration-only) captured some signal, particularly for SPES, but the full framework provided significant gains, confirming that broader motion descriptors add value beyond simple task duration or yaw rotation.

Significance and Claims
The authors position this work not as a replacement for standardized questionnaires, but as a complementary behavioral layer for signal-enhanced VR evaluation.

• Reproducibility: The paper establishes a reproducible benchmark using an open dataset (FAST) and a structured, deterministic feature extraction pipeline, addressing the lack of comparability in prior behavioral estimation studies.
• Construct-Dependency: The findings explicitly challenge the notion that all subjective states are equally observable from head motion. The study demonstrates that while spatial presence has a strong behavioral signature in HMD pose, usability and workload are less recoverable from aggregate head motion alone, likely due to their dependence on interaction quality, task success, and cognitive strategies not captured by head pose.
• Practical Implication: The proposed framework supports the use of HMD tracking as a non-intrusive, low-burden signal to enrich the interpretation of subjective outcomes, particularly for detecting spatial presence, rather than as a standalone automated assessment tool. The study concludes that while head motion provides valuable behavioral data, it does not currently possess the robustness to fully replace self-report instruments for all VR evaluation constructs.