AI-based Verbal and Visual Scaffolding in a Serious Game: Effects on Learning and Cognitive Load

This study investigated the impact of different AI-based scaffolding types—verbal, visual, or both—within a quantum technology serious game, finding that while all conditions led to significant learning gains, the combination of verbal and visual scaffolding most effectively reduced intrinsic cognitive load.

The Gist

The "Quantum Tutor" Experiment: Teaching the Impossible with AI Sidekicks

Imagine you are trying to solve a complex Rubik's Cube, but the colors keep changing, and the rules are written in a language you only half-understand. This is what learning Quantum Technology feels like for most people—it’s abstract, invisible, and mind-bendingly difficult.

Researchers wanted to see if a "Serious Game" (a game designed to teach, not just entertain) could help people master these concepts using different types of AI "sidekicks."

---

The Three Types of Sidekicks

In their game, called Qookies, the researchers gave players three different levels of help:

1. The Lone Wolf (No Scaffolding): You are on your own. You have the game, the puzzles, and your own brain. It’s just you against the quantum world.
2. The Chatty Professor (Verbal Scaffolding): You have an AI character you can talk to. If you’re stuck, you can type, "Hey, what is a qubit?" and the AI will explain it to you in text. It’s like having a textbook that talks back.
3. The Action Hero (Verbal + Visual Scaffolding): This is the "Super Sidekick." Not only can you chat with the AI to get explanations, but you can also tell it, "Show me what to do!" The AI character will then actually move around the game world, demonstrating the physical actions needed to solve the puzzle. It’s like having a tutor who both explains the math and physically shows you how to turn the wrench.

---

The Big Question: Does the "Super Sidekick" make you smarter?

The researchers tested 152 people (from school kids to adults) to see two things:

• Learning: Did the sidekicks actually help you understand quantum physics better?
• Brain Power (Cognitive Load): Did the help make the game easier to process, or did the AI itself become a distraction that cluttered your brain?

---

The Results: What happened?

1. The "Everyone Wins" Result

Surprisingly, everyone learned! Whether they had a chatty AI, a visual AI, or no AI at all, every single group showed significant improvement from the start to the end of the game. The game itself was a powerful teacher, regardless of the sidekick.

2. The "Show, Don't Just Tell" Discovery

This is the most important finding. While everyone learned, the "Action Hero" (Verbal + Visual) group had a much "lighter" mental load.

Think of it this way: If you are building IKEA furniture, you can either read a long, complicated manual (Verbal) or watch a 30-second video of someone actually clicking the pieces together (Visual). The manual is helpful, but your brain has to work hard to translate those words into mental pictures. The video, however, lets your brain "see" the solution instantly.

The study found that visual demonstrations from the AI saved mental energy, making the difficult concepts feel less heavy and overwhelming.

3. The "Human Pride" Factor

Interestingly, many players didn't use the AI at all! They wanted to solve the puzzles themselves. They viewed the game as a "desirable difficulty"—a challenge they wanted to conquer without "cheating" with an AI.

---

The Bottom Line

If we want to use AI to teach incredibly hard subjects like quantum physics, don't just give students a chatbot. A chatbot is a good librarian, but a visual, acting AI is a great coach. To prevent "brain fog," the best AI support doesn't just talk; it shows.

Technical Summary

Technical Summary: AI-based Verbal and Visual Scaffolding in a Serious Game

1. Problem Statement

The integration of serious games into education offers high engagement but faces a central design challenge: providing sufficient instructional support (scaffolding) to help learners navigate complex tasks without undermining the active problem-solving required for deep learning. While Large Language Models (LLMs) offer new opportunities for personalized, adaptive scaffolding, they introduce risks such as hallucinations (incorrect information) and cognitive offloading (where learners accept AI output without critical reflection, leading to low retention). The research seeks to understand how different modalities of AI-driven scaffolding—specifically verbal (text) versus combined verbal and visual (action-based)—affect learning outcomes and cognitive load in the context of complex subjects like quantum technology.

2. Methodology

The study utilized Qookies, a theory-based serious game designed to teach fundamental quantum technology concepts through interactive puzzles.

• Experimental Design: A one-factor between-groups design was employed with 152 participants (school students, university students, and the general public). Participants were randomly assigned to one of three conditions:
  1. Control Group: No AI scaffolding.
  2. Verbal Scaffolding: Chat-based support via an AI-based Non-Player Character (NPC) using Llama 3.1 70B.
  3. Combined (Verbal + Visual) Scaffolding: Chat-based support plus an action-based NPC that can perform in-game demonstrations using a reinforcement learning model.
• AI Architecture: To mitigate hallucinations, the AI was prompted with specific level concepts but was not given the actual solutions. Instead, the NPC was designed as a "co-learner" that learns alongside the player through reinforcement learning, encouraging collaborative knowledge construction.
• Measurement:
  • Learning Gains: Pre-tests and post-tests measuring conceptual understanding of quantum technologies.
  • Cognitive Load: Self-reported measures of Intrinsic (ICL), Extraneous (ECL), and Germane (GCL) cognitive load.
  • Interaction Analysis: Categorization of user-AI chat queries based on the ICAP framework (Interactive, Constructive, Active, Passive).

3. Key Contributions

• Multimodal AI Scaffolding Framework: The study demonstrates a method for combining generative LLMs (for dialogue) with reinforcement learning (for embodied action) to create a dual-modality scaffolding system.
• Co-Learning Paradigm: By designing the AI to lack the solution at the start of a level, the researchers implemented a "co-learner" model that prevents the "unreflected acceptance" of AI output and promotes active engagement.
• Empirical Validation of Visual Scaffolding: The research provides evidence that visual/action-based demonstrations are more effective at managing cognitive load than text-only instructions in immersive environments.

4. Results

• Learning Outcomes: All three groups showed significant learning gains from pre-test to post-test. However, there were no significant differences in learning outcomes between the control group and the two AI-supported groups, suggesting the game itself was effective and the AI did not hinder learning.
• Cognitive Load:
  • Intrinsic Cognitive Load (ICL): The combined Verbal + Visual condition resulted in significantly lower ICL compared to the Verbal-only condition.
  • Extraneous (ECL) and Germane (GCL) Load: No significant differences were found across groups, suggesting the AI interactions were intuitive and did not add unnecessary mental burden.
• User Interaction: Most player queries were "level-related" and focused on seeking "recommendations for action" (Active engagement). Notably, no players engaged in "hypothesis/idea formation" (Constructive engagement), and many players chose not to use the AI at all, perceiving the difficulty of the levels as "desirable."

5. Significance

This research is significant for the field of AI-mediated education and game-based learning. It proves that AI agents in serious games do not have to be "all-knowing" authorities; instead, they can function as collaborative partners that reduce the mental effort required to translate abstract text into concrete actions. The finding that visual demonstrations significantly lower intrinsic cognitive load provides a design principle for future educational AI: to support complex learning, AI should demonstrate actions rather than just explaining them. This approach balances the need for instructional support with the necessity of maintaining "desirable difficulty" to ensure robust knowledge retention.