💻 computer science

Behavioral Engagement in VR-Based Sign Language Learning: Visual Attention as a Predictor of Performance and Temporal Dynamics

This study demonstrates that in a VR-based sign language learning application, sustained visual attention and post-playback viewing time are significant predictors of learning performance, while temporal analysis reveals distinct engagement patterns aligned with information density and learning phases.

Original authors: Davide Traini, José Manuel Alcalde-Llergo, Mariana Buenestado-Fernández, Domenico Ursino, Enrique Yeguas-Bolívar

Published 2026-03-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Davide Traini, José Manuel Alcalde-Llergo, Mariana Buenestado-Fernández, Domenico Ursino, Enrique Yeguas-Bolívar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are learning a new dance, but instead of a human teacher, you are learning from a robot in a virtual reality (VR) world. You put on a headset, and suddenly you are in a university classroom. Your "teacher" is a deaf avatar named Essie, and she is teaching you how to say "Hello," "Nice to meet you," and other basic phrases using sign language.

This paper is like a detective story. The researchers wanted to solve a mystery: How can we tell if a student is actually learning just by watching their digital footprints?

In the real world, a teacher might see a student nodding, leaning forward, or looking confused. In VR, the computer can't see your face, but it can see where your head is pointing and what you are clicking. The researchers used these digital clues to guess who would do well on a quiz at the end.

Here is the breakdown of their investigation, explained simply:

The Three "Clues" (Engagement Indicators)

The researchers looked at three specific behaviors to see which ones predicted a good grade:

Visual Attention (The "Laser Beam"): This measures how much time you spent staring directly at the signing avatar.
- The Metaphor: Imagine your attention is a laser pointer. If you keep the laser steady on the teacher's hands, you are "locked in." If your laser wanders around the room looking at the virtual blackboard or your own hands, you are "drifting."
Post-Playback Viewing Time (The "After-Image"): This measures how long you kept the video on the screen after it finished playing.
- The Metaphor: Imagine watching a magic trick. When the magician finishes, do you immediately look away, or do you stare at the empty stage for a few seconds, trying to figure out how they did it? That extra staring time is the "After-Image." It suggests your brain is still working on the information.
Video Replay Frequency (The "Rewind Button"): This counts how many times you hit the "rewind" button to watch a video again.
- The Metaphor: This is like pressing the "repeat" button on a song. The researchers wondered: "Does hitting repeat 10 times mean you are a super-learner, or just a confused one?"

The Big Discovery

The researchers tested 117 university students. Here is what they found:

The "Laser Beam" (Visual Attention) was the Superstar: There was a huge connection between students who kept their gaze locked on the teacher and those who got high quiz scores. If you watched the hands, you learned the signs.
The "After-Image" (Post-Playback Time) was the Runner-Up: Students who lingered on the screen after the video stopped also did very well. It seems that taking a moment to let the information "sink in" is a powerful learning strategy.
The "Rewind Button" (Replay Frequency) was a Dud: Surprisingly, hitting the replay button a lot did not predict a good grade. In fact, it didn't matter at all.
- The Lesson: It's not about how many times you watch a video; it's about how well you watch it the first time. Quality beats quantity. If you are staring intently, you don't need to hit rewind. If you are staring blankly, hitting rewind won't help.

The "Heartbeat" of Learning

The researchers also looked at when students paid attention. They created a graph that looked like a heartbeat monitor.

The Warm-up: At the very start, attention was low as students got used to the VR headset.
The Peaks: Every time a new sign was taught, the "heartbeat" spiked. Students focused intensely on the tricky parts (like a specific hand shape).
The Valleys: Between the peaks, attention dipped. This wasn't boredom! The researchers think this was the students' brains taking a "breather" to process what they just saw.
The Quiz Spike: When the test started, attention spiked again, showing that students were focused on recalling what they learned.

Why Does This Matter?

This study is like finding a secret code for better learning.

For Students: It tells us that we should focus on staying present rather than just mindlessly re-watching things. If you catch your mind wandering, pull your "laser beam" back to the teacher.
For Teachers and App Developers: Imagine a VR app that could say, "Hey, I noticed your attention is drifting. Let's slow down," or "You stared at that sign for a long time; do you want to try it again?" This could help create "smart" classrooms that adapt to how you learn in real-time.

In a nutshell: Learning sign language in VR isn't about how many times you hit "repeat." It's about keeping your eyes on the prize and giving your brain a moment to digest the information. The computer can see this focus, and that focus is the key to success.

1. Problem Statement

While Virtual Reality (VR) has shown promise in sign language education, a critical gap exists in understanding how objective behavioral engagement indicators extracted from learner interactions correlate with actual learning outcomes. Most existing studies focus on general effectiveness or usability but lack systematic analysis of how specific behavioral traces (such as gaze patterns or interaction logs) predict performance. Furthermore, there is limited research on the temporal dynamics of engagement—how attention evolves moment-to-moment during immersive learning sessions. This study addresses these gaps by investigating whether automatically derived behavioral metrics can predict learning success in a VR sign language environment and how engagement patterns change over time.

2. Methodology

Experimental Setup & Application

Application: The study utilized SONAR, a VR application developed within the ISENSE project. It simulates a university classroom where a hearing student interacts with a deaf avatar (Essie) to learn basic sign language phrases.
Phases: The experience consists of two phases:
1. Training: Self-paced viewing of 12 instructional videos. Users can replay videos and observe signs from a first-person perspective.
2. Validation: A quiz where users must select the correct sign video corresponding to a phrase, testing retention.
Hardware: Meta Quest headsets (6DoF tracking) using controller-free hand tracking to maximize immersion and ensure continuous hand availability for signing practice.
Participants: 117 university students (86% with no prior sign language experience; 56.1% with no prior VR experience).

Behavioral Indicators (Features)

The study defined three automatically derived engagement metrics:

Visual Attention (VA): The proportion of time the user's head pose (used as a proxy for gaze due to lack of eye-tracking in consumer VR) is oriented toward the video screen displaying the signer.
Video Replay Frequency (VRF): The number of times a training video is replayed, normalized by total session time.
Post-Playback Viewing Time (PPVT): The duration a video remains visible on the screen after playback ends but before the user moves to the next video or replays it. This serves as a proxy for post-presentation cognitive processing (e.g., mental rehearsal).

Analytical Approach

Correlation Analysis: Pearson correlation coefficients were calculated between the three behavioral indicators and quiz performance (number of correct answers).
Regression Modeling: A Binomial Generalized Linear Model (GLM) with a logit link function was employed to predict the proportion of correct answers using standardized VA, PPVT, and VRF as predictors.
Temporal Analysis: Moment-to-moment VA traces were aggregated across all participants to characterize the evolution of engagement throughout the training and validation phases.

3. Key Contributions

Predictive Validity of Behavioral Traces: The study establishes that specific, automatically captured behavioral metrics (VA and PPVT) are strong predictors of learning performance in VR-based sign language training.
Quality vs. Quantity of Engagement: It challenges the assumption that frequent repetition (replaying videos) equates to better learning, demonstrating that the quality of attention (sustained focus and post-playback processing) is more critical than the quantity of replays.
Temporal Engagement Dynamics: The paper provides a granular temporal profile of user engagement, revealing distinct phases of acclimatization, oscillatory attention cycles during learning, and specific attentional peaks during assessment.
Scalable Analytics Framework: By utilizing head-pose data instead of specialized eye-tracking hardware, the study proposes a scalable method for engagement analytics in consumer-grade VR educational applications.

4. Results

Correlation and Regression Findings (RQ1)

Visual Attention (VA): Showed the strongest positive correlation with quiz performance ( $r = 0.76, p < 0.001$ ). In the GLM, VA was a highly significant predictor (Odds Ratio $\approx 2.13$ ), indicating that a one standard deviation increase in VA more than doubles the odds of a correct answer.
Post-Playback Viewing Time (PPVT): Also showed a strong positive correlation ( $r = 0.66, p < 0.001$ ) and was a significant predictor in the GLM (Odds Ratio $\approx 1.55$ ). This suggests that time spent reviewing content after playback is crucial for consolidation.
Video Replay Frequency (VRF): Showed negligible and non-significant correlation with performance ( $r = 0.02, p = 0.811$ ). The GLM coefficient was near zero, indicating that replaying videos does not independently predict success once VA and PPVT are accounted for.
Model Performance: The binomial GLM achieved a Cox-Snell pseudo- $R^2$ of 0.83, indicating that the combination of VA and PPVT explains a substantial proportion of the variance in learning outcomes.

Temporal Dynamics (RQ2)

Acclimatization: VA showed a gradual ramp-up at the start of the session as users familiarized themselves with the VR environment.
Oscillatory Patterns: During the training phase, VA exhibited 12 distinct peaks corresponding to the 12 instructional videos. Between peaks, VA dipped, suggesting a pattern of high focus during information-dense segments followed by periods of internal processing or reflection.
Validation Phase: A transition to a lower baseline occurred before the validation phase, followed by a second set of 12 peaks corresponding to quiz items, indicating renewed, task-driven attention during assessment.
Termination: A sharp collapse in VA occurred immediately after the session ended, confirming that the observed attention patterns were task-driven and stimulus-dependent.

5. Significance and Implications

Theoretical Impact: The findings extend cognitive load theory and embodied learning principles to VR sign language contexts, confirming that sustained visual attention and post-presentation processing are critical for encoding complex motor and linguistic information.
Practical Application: These metrics can be computed in real-time to enable attention-aware learning analytics. Systems could use VA and PPVT data to trigger adaptive interventions, such as:
- Prompting users to refocus if attention drops.
- Suggesting targeted content review if PPVT is insufficient.
- Avoiding unnecessary interruptions during high-engagement phases.
Future Directions: The authors note that while the study is correlational, future work should involve experimental designs to establish causality. Additionally, extending these findings to diverse populations (e.g., children, older adults) and more complex sign language tasks is recommended to validate the generalizability of these engagement patterns.

In conclusion, this paper demonstrates that sustained visual attention and strategic post-playback processing are the primary drivers of success in VR sign language learning, offering a robust, data-driven framework for developing adaptive and inclusive immersive educational tools.