Skill-Adaptive Ghost Instructors: Enhancing Retention and Reducing Over-Reliance in VR Piano Learning

Imagine you are learning to play the piano in a virtual reality (VR) headset. To help you, a "ghost hand" appears in front of you, showing you exactly which keys to press and how to move your fingers.

For a long time, these ghost hands have been like a permanent, semi-transparent shadow. They are always there, always the same shade of gray, no matter how well you are playing. The problem? If the shadow is always there, your brain gets lazy. You start relying on the shadow instead of remembering the notes yourself. It's like trying to ride a bike with training wheels that never come off; you might look good while they are on, but the moment you take them away, you fall over.

This paper introduces a smarter solution: The "Skill-Adaptive Ghost Instructor."

The Core Idea: A Ghost That Breathes

Instead of a static shadow, imagine a ghost hand that breathes.

When you are struggling: The ghost hand becomes solid and bright. It says, "Hey, look at me! I need to show you exactly what to do."
When you are doing well: The ghost hand becomes faint and transparent. It fades into the background, whispering, "You've got this, just keep going."
When you make a mistake: The ghost hand instantly becomes solid again to catch you before you fall.

The researchers call this "Dynamic Transparency." It's like a coach who knows exactly when to step in and when to step back.

The Experiment: The "Training Wheels" Test

The researchers tested this idea with 30 people learning two short piano songs in VR. They split the learners into two groups:

The Static Group: Learned with a ghost hand that never changed its opacity (always 50% see-through).
The Dynamic Group: Learned with the "breathing" ghost hand that faded in and out based on their performance.

After practicing, they took two tests:

The Immediate Test: Play the song right away without the ghost hand.
The Retention Test: Play the song again 10 minutes later (after doing a Sudoku puzzle to clear their memory) without the ghost hand.

What Happened? (The Results)

The results were like night and day:

The "Laziness" Effect: The Static group got good at playing while the ghost was there, but as soon as the ghost disappeared, their performance tanked. They forgot the finger positions and hit wrong notes. They had become dependent on the visual crutch.
The "Internalization" Effect: The Dynamic group didn't just play better with the ghost; they played significantly better without it. Because the ghost faded when they were doing well, their brains were forced to remember the finger movements. They internalized the skill.
The "Safety Net": When the Dynamic group made a mistake, the ghost reappeared to help them correct it immediately. This prevented them from practicing the wrong way.

The Analogy:
Think of the Static ghost as a crutch that you lean on constantly. You get used to the weight of it, but your leg muscles never get strong.
Think of the Dynamic ghost as a spotter at the gym. If you lift the weight perfectly, the spotter steps back and lets you do it alone. If you start to wobble, they grab the bar to save you. By the end of the workout, your muscles are stronger because you actually did the lifting.

Why Does This Matter?

This isn't just about piano. This is about how we learn any physical skill, from surgery to sports to playing video games.

Old Way: Constant guidance. "Here is the path, follow it." (Good for starting, bad for mastering).
New Way: Adaptive guidance. "Follow the path, but I'll fade away when you know it, and I'll jump back in if you slip."

The Takeaway

The paper proves that less guidance is often more helpful for long-term learning, if that guidance is smart enough to know when to disappear.

By making the ghost hand "fade out" when you are doing well, the system forces your brain to take ownership of the skill. It reduces the "over-reliance" on the screen and helps you remember the music even after you take off the VR headset.

In short: Don't just show the learner the way; teach them to find the way themselves, but be ready to step in the second they get lost.

Here is a detailed technical summary of the paper "Skill-Adaptive Ghost Instructors: Enhancing Retention and Reducing Over-Reliance in VR Piano Learning."

1. Problem Statement

Virtual Reality (VR) systems for motor-skill learning (e.g., piano training) often rely on persistent visual cues (static ghost instructors) to guide learners. While effective for immediate performance, this approach creates a "guidance dependency" where learners over-rely on external cues, neglecting self-correction and proper technique formation (specifically fingering). Consequently, when cues are removed, learners suffer from significant performance drops and poor short-term retention. Existing systems typically use fixed-visibility overlays that do not adapt to the learner's moment-to-moment proficiency, failing to balance instructional support with learner autonomy.

2. Methodology

System Design: Skill-Adaptive Ghost Instructor

The authors developed a VR piano training application running on Meta Quest 3 (Unity 2022.3) featuring a "Ghost Instructor"—a semi-transparent virtual hand overlaying the learner's view.

Input Capture: Utilizes the Meta XR Hands API for 26-joint skeletal tracking of the learner's right hand.
Performance Metrics: The system calculates real-time sub-scores for:
1. Pitch Accuracy ( $s_p$ ): Correct key press within a temporal window ( $\pm 120$ ms).
2. Timing Precision ( $s_t$ ): Deviation from the reference onset time.
3. Fingering Correctness ( $s_f$ ): Semantic mapping of finger labels (Thumb–Pinky) to intended keys, allowing partial credit for adjacent fingers.
Adaptive Logic:
- A weighted performance score ( $S$ ) is calculated: $S = 0.7s_p + 0.2s_t + 0.1s_f$ .
- An error signal ( $E = 1 - S$ ) drives the ghost's opacity.
- Asymmetric Exponential Moving Average (EMA): To prevent visual jitter and encourage self-correction, the system uses an asymmetric smoothing factor. The ghost fades quickly when performance improves ( $\lambda_{\downarrow}$ ) but reappears cautiously when errors occur ( $\lambda_{\uparrow}$ ).
- Opacity Range: Clamped between 0.08 (faint) and 0.8 (visible), ensuring the ghost never fully occludes the keyboard.

Experimental Design

Study Type: Within-subjects design with $N=30$ participants (15 experienced, 15 inexperienced pianists).
Conditions:
1. Static: Ghost hand with fixed opacity (0.5).
2. Dynamic: Ghost hand with performance-adaptive opacity.
Procedure:
- Participants practiced two short melodies (A and B) under both conditions (counterbalanced).
- Training Phase: 10 loops per phrase/melody with guidance active.
- Immediate Test: 2 trials without guidance.
- Retention Test: 2 trials without guidance after a 10-minute distractor interval (Sudoku).
Metrics: Objective accuracy (pitch, finger, timing, error rate), NASA-TLX workload, and subjective preference.

3. Key Contributions

Conceptual Framing: Identifies persistent cues as a barrier to retention in XR motor learning and proposes adaptive transparency as a mechanism to balance support and autonomy.
Design Intervention: Introduces a skill-adaptive ghost instructor that modulates visibility in real-time based on a multi-dimensional performance signal (pitch, timing, fingering). Unlike prior work that fades cues based on time or simple error, this system uses an asymmetric EMA to encourage retrieval practice.
Empirical Insights: Provides evidence that adaptive transparency improves fingering accuracy and short-term retention while reducing cognitive workload and error rates compared to static cues.

4. Key Results

Objective Performance

Pitch & Fingering Accuracy: The Dynamic condition significantly outperformed the Static condition.
- Pitch: Dynamic maintained ~99% accuracy in retention tests, whereas Static dropped from 96.8% to 87.8%.
- Fingering: Dynamic remained stable (~96%), while Static dropped significantly from 88.2% to 76.7%.
Error Rates: Dynamic resulted in lower error rates (11.6% in retention) compared to Static (19.8%).
Timing: No significant difference was found between conditions, suggesting visibility adaptation alone is insufficient for rhythmic scaffolding without explicit temporal cues.
Skill Transfer: Participants who started with the Dynamic condition performed better when switching to a new melody in the second block, indicating better generalization of self-correction skills.

Subjective & Workload

Workload: Dynamic significantly reduced perceived workload (NASA-TLX), particularly in Performance, Effort, and Frustration dimensions.
Preference: 88.7% of participants preferred the Dynamic mode over Static.
Retention: Dynamic guidance led to stable performance from Immediate to Retention tests, whereas Static guidance showed a marked decline (negative retention scores).

Learning Curves

While group-average learning rates were similar, individual analysis showed that Dynamic-first learners achieved steeper error-reduction slopes and reached performance criteria faster than Static-first learners.

5. Significance and Implications

Mitigating Over-Reliance: The study demonstrates that faded feedback (removing cues when the learner is competent) is superior to constant feedback for long-term skill internalization. The adaptive ghost acts as a "scaffold" rather than a "crutch."
Design Guidelines for XR:
1. Fade with Proficiency: Reduce guidance visibility as performance stabilizes.
2. Recover after Slips: Re-introduce guidance cautiously after errors to prevent drift.
3. Multi-Dimensional Adaptation: Integrate pitch, timing, and technique (fingering) into a unified control signal.
Broader Applicability: The approach is generalizable to other fine-motor tasks requiring spatial precision and temporal sequencing, such as sports training, surgical simulation, and upper-limb rehabilitation.
Limitations & Future Work: The current system relies on virtual keys (no tactile feedback), which can lead to tracking jitter on pinky/ring fingers. Future work aims to integrate physical keyboards (AR/MR), personalize weighting schemes based on learner profiles, and add explicit rhythmic cues (metronomes) to address the timing gap.

In conclusion, the paper validates that dynamically transparent ghost instructors effectively reduce dependency on visual cues, leading to better retention and higher accuracy in VR-based motor skill acquisition compared to traditional static guidance.