AI-Assisted Tutorial Generation for Immersive Training in Virtual and Mixed Reality

Imagine you are trying to learn how to build a complex LEGO set, but the instructions are just a wall of tiny, confusing text. You have no idea which piece goes where, or how to twist your wrist to snap it in. Now, imagine if, instead of reading, you could see a ghostly, floating version of a master builder right next to you, moving their hands exactly how they did when they built it, while a voice whispers exactly what to do. This is the world of Immersive Training, a field where computers create virtual or "mixed" worlds to teach people skills. In a Virtual Reality (VR) world, you are totally inside a computer simulation, like being in a video game. In Mixed Reality (MR), you wear a special headset that lets you see your real room and tools, but overlays digital hints and ghosts on top of them.

The big problem scientists have been wrestling with is that making these "ghost builder" tutorials is incredibly hard and expensive. Usually, you need a team of 3D artists and programmers to manually build every single step of the training. It's like trying to film a movie by hand-painting every single frame. But what if a computer could just watch an expert do the job once, listen to them explain it, and then automatically build the tutorial for you? That's the question this paper asks: Can we use Artificial Intelligence (AI) to turn a single expert demonstration into a reusable, interactive training guide that works in both virtual and mixed worlds?

---

The "One-Shot" Magic Trick

The researchers, a team from Hungary and Germany, built a system that acts like a super-observant robot camera. Their main idea is simple: instead of spending weeks building a training course, an expert just needs to perform the task once.

Think of it like this: You have a master chef who knows how to cook a perfect soufflé. In the old days, you'd have to write down every ingredient, measure every second, and draw diagrams of how to whisk the eggs. In this new system, the chef just puts on a special headset and cooks the soufflé while talking to the camera. The system records everything:

• What they say: Their voice is turned into text.
• Where they look: The system tracks their eyes to see what they are focusing on.
• How they move: It captures the exact position of their hands and head.

Once the chef is done, the computer's brain (an AI called a Large Language Model) steps in. It takes the messy, spoken instructions and the raw video data and organizes them into a clean, step-by-step guide. It fixes grammar mistakes, breaks the big task into small, bite-sized steps, and even creates a digital "avatar" (a 3D ghost) that replays the chef's movements perfectly in sync with the instructions.

The Two Worlds: Virtual and Mixed

The team tested this magic trick in two different playgrounds to see if it actually worked.

1. The Virtual Reality (VR) Test: The Engine Assembly
First, they put the system into a fully virtual world using a headset called the Meta Quest 2. They created a scenario where trainees had to assemble a six-cylinder industrial engine.

• The Setup: Some trainees got a standard tutorial with text and arrows. Others got the "Super Tutorial" with the AI-generated steps plus the ghost avatar of the expert moving their hands right in front of them.
• The Result: The ghost made a huge difference. When the expert's avatar was replaying the moves, the trainees finished the engine assembly 75.4 seconds faster on average (dropping from 579.2 seconds to 503.8 seconds).
• The Feeling: The trainees felt much more confident. They told the researchers that seeing the expert's hand movements helped them understand how to hold and twist parts, not just where to put them. However, they also noted that the ghost wasn't always needed; if a step was super simple or they had done it before, the ghost could sometimes get in the way.

2. The Mixed Reality (MR) Test: The Airplane Mechanic
Next, they moved to the real world using a HoloLens 2 headset. Here, the trainees were in a room with real airplane parts (or simulated real parts) and the digital ghost hovered over them.

• The Twist: In this test, the researchers didn't just watch people learn; they watched people teach. Participants tried to use the system to create their own tutorials by speaking instructions and performing a task.
• The Result: The system worked surprisingly well. When people spoke their instructions, the AI turned their speech into text and then organized it into a tutorial. The AI made very few mistakes. Out of a typical 15–20 sentence instruction, the speech-to-text made about 1.36 errors on average, but the AI's "brain" corrected most of them, leaving only 0.18 errors per tutorial.
• Learning: People who used the MR training actually remembered the steps better 24 hours later. They could put the steps of the airplane maintenance in the correct order much more accurately after training.

What the Numbers Say

The researchers didn't just guess; they measured everything.

• Speed: In VR, the expert replay cut the time by about 13%.
• Confidence: In the final survey, 16 out of 19 participants said the expert replay made the correct technique clearer. 12 out of 19 said they would prefer this method for future learning.
• Usability: Both the VR and MR systems scored very high on "usability" tests. The VR system got an average score of 81.84, and the MR system got 83.18 (where anything over 80 is considered "excellent").
• Accuracy: The AI was good at fixing the messy speech of the experts, turning a rough recording into a polished guide with very little human help.

The Verdict

The paper suggests that this "one-shot" method is a game-changer for training. It proves that you don't need a Hollywood production team to make high-quality training videos. If you have an expert and a headset, the AI can do the heavy lifting.

However, the authors are careful to point out that this isn't a magic wand for everything.

• VR vs. MR: VR is great for practicing in a safe, controlled environment where you can't break real things, but it takes a lot of work to build the virtual world first. MR is faster to set up because you use the real world, but it's harder to make the computer "see" and highlight specific real-world objects perfectly right now.
• The Ghost's Role: The expert replay is most helpful for tricky, confusing steps. For simple, repetitive tasks, it might be better to let the learner do it without the ghost hovering over them.

In short, the researchers have built a bridge between human expertise and computer automation. They showed that by recording an expert once, you can generate a reusable, interactive training guide that helps people learn faster and remember better, whether they are inside a virtual engine room or standing in a real airplane hangar. It's not a solved problem yet—there's still work to be done on making the computer understand the real world perfectly—but the path forward looks very bright.