Dynamic Multimodal Expression Generation for LLM-Driven Pedagogical Agents: From User Experience Perspective

This paper proposes a large language model-driven method for generating dynamic, semantically aligned speech and gestures for pedagogical agents in virtual reality, demonstrating through user experience experiments that such multimodal expressions significantly enhance learning effectiveness, engagement, and social presence while reducing fatigue and boredom.

The Gist

Imagine you are sitting in a virtual classroom, wearing a VR headset. Standing before you is a digital teacher. In the past, this teacher would sound like a robot reading a script from a teleprompter: flat voice, no pauses, and stiff, repetitive hand movements. It felt like talking to a vending machine that dispensed facts instead of a human.

This paper introduces a new way to make that digital teacher feel much more real and engaging. Here is the story of their research, explained simply.

The Problem: The "Robot Teacher"

The researchers noticed that most virtual teachers in VR are boring. They speak in a monotone voice and use the same few hand gestures no matter what they are teaching.

• The Analogy: Imagine listening to a GPS navigation system that says, "Turn left," "Turn left," and "Turn left" with the exact same robotic tone, even when you are driving through a beautiful forest or a scary storm. It's functional, but it doesn't make you feel anything.
• The Result: Students get bored, lose focus, and feel like they aren't really "learning" from a person.

The Solution: The "Smart Director"

The team built a new system using Large Language Models (LLMs)—the same kind of AI that powers chatbots. But instead of just making the AI talk, they taught it to act like a human director on a movie set.

Here is how their system works:

1. Understanding the Script: The AI reads the lesson content. If the topic is difficult, the AI knows to slow down. If it's a key point, it knows to get excited.
2. The "Prompt" (The Director's Notes): The researchers created a special set of instructions (prompts) that tell the AI: "When you explain a hard concept, pause for a second, say 'um' like you are thinking, and point your finger to emphasize the point."
3. The Performance: The AI then generates speech with natural pauses, filler words (like "you know"), and changes in tone. Simultaneously, it triggers hand gestures that match the words (like a "thinking" pose or an "emphasizing" point).

The Analogy: Think of the old virtual teacher as a mannequin that just stands there. The new teacher is like a skilled actor who knows when to pause for dramatic effect, when to lean in to whisper a secret, and when to throw their hands up to show excitement.

The Experiment: Putting It to the Test

To see if this actually works, they put 36 students in a VR classroom and had them talk to the teacher under four different "moods":

1. The Robot: Flat voice, stiff gestures (The Control Group).
2. The Radio: Dynamic voice, but stiff gestures.
3. The Mime: Stiff voice, but dynamic gestures.
4. The Superstar: Dynamic voice and dynamic gestures.

What They Found

The results were clear: The "Superstar" teacher won.

• Better Learning: Students felt they learned more and paid closer attention when the teacher used natural pauses and gestures. It was like the teacher was giving them time to digest the information, rather than just dumping it on them.
• Less Boredom: The dynamic teacher made students feel less tired and frustrated. The "robot" teacher made them feel impatient.
• More Human: The students felt a stronger connection to the "Superstar" teacher. They felt like they were talking to a real person, not a computer program.

The Catch: It's Not Perfect Yet

While the new system was a huge improvement, the students still noticed it wasn't quite human.

• The "Uncanny Valley": Sometimes the hand movements felt a little stiff or didn't perfectly sync with the voice.
• The Analogy: It's like watching a very good puppet show. You are impressed by the skill, but you still know it's a puppet. To make it truly feel like a human, the "puppeteer" needs to make the movements even smoother and more responsive.

Why This Matters

This research is a big step forward for the future of education. It shows that for AI teachers to be truly effective, they can't just be smart; they have to be expressive.

Just as a human teacher uses their voice and body to keep a class engaged, a digital teacher needs to do the same. By teaching AI to "act" naturally, we can create virtual classrooms that are less lonely, less boring, and much more effective at helping people learn.

Technical Summary

Here is a detailed technical summary of the paper "Dynamic Multimodal Expression Generation for LLM-Driven Pedagogical Agents: From User Experience Perspective."

1. Problem Statement

In Virtual Reality (VR) educational environments, Pedagogical Agents (PAs) are designed to enhance immersion and learning. However, existing PAs suffer from significant limitations:

• Static and Rigid Interactions: Most current systems rely on fixed speech outputs and simple, pre-defined gestures that are disconnected from the semantic context of the instructional content.
• Lack of Adaptability: Unlike human teachers who naturally adjust their tone, pacing, pauses, and gestures based on the difficulty of the concept or the need to emphasize key points, current PAs appear mechanical and unnatural.
• Negative User Experience: This lack of semantic awareness leads to reduced engagement, lower perceived learning effectiveness, and increased feelings of fatigue and boredom among learners.

2. Methodology

The authors propose and evaluate an LLM-driven multimodal expression generation framework designed to align an agent's expressive behaviors (speech and gestures) with the semantic context of instructional content.

A. System Architecture

The proposed system consists of five core modules:

1. Speech-to-Text (STT): Uses OpenAI Whisper API for real-time transcription of user input.
2. Large Language Model (LLM): Based on GPT-4o, responsible for semantic understanding, dialogue generation, and the critical task of generating semantically sensitive prompts.
3. Prompt Construction (PC) Module: The core innovation. It constructs prompts containing:
   • Background Information: Role definition and scenario context.
   • Expression Information: A "Knowledge" base defining rules for speech tags (e.g., <prosody>, <break>, <filler>) and gesture tags (e.g., <bookmark> for emphasis). It includes libraries for filler words (e.g., "umm," "you know") and gesture categories (thinking, emphasizing, summarizing).
   • User Question: The transcribed input.
   • Output: The LLM generates a response that includes not only text but also specific instruction tags for speech prosody and gestures grounded in the semantic meaning of the content.
4. Text Parsing (TP) Module: Parses the LLM's output, mapping tags to specific items in the preset libraries (e.g., mapping <filler type="thinking"> to "umm...") and converting the text into SSML (Speech Synthesis Markup Language) format compatible with the TTS engine.
5. Text-to-Speech (TTS): Uses Microsoft Azure API to synthesize speech with fine-grained control over intonation, rate, volume, and pauses based on the SSML tags. It also triggers synchronized animations in the Unity engine via gesture tags.

B. Experimental Design

• Setup: A VR classroom environment (Unity) with a digital teacher (Ready Player Me avatar) on an Oculus Quest 2.
• Participants: 36 university students (20–26 years old) with no prior knowledge of the specific course topic ("Multimedia Communication").
• Design: A 2×2 within-subjects factorial design with four conditions:
  • Condition A (Control): Static Speech + Static Gestures (Default).
  • Condition B: Dynamic Speech + Static Gestures.
  • Condition C: Static Speech + Dynamic Gestures.
  • Condition D: Dynamic Speech + Dynamic Gestures.
• Procedure: Participants engaged in Q&A sessions under all four conditions. Data was collected via:
  • Quantitative: Questionnaires measuring Perceived Usefulness (PU), Learning Engagement (EN), Intention to Use (IU), Human-likeness (HL), Social Presence (SP), and Discomfort (DC).
  • Qualitative: Semi-structured interviews analyzed via thematic analysis.

3. Key Contributions

1. LLM-Driven Multimodal Generation: Proposed a novel method using semantically adaptive prompt construction to generate coordinated speech and gesture instructions. This bridges the gap between "what to say" and "how to say it," enabling dynamic variations in speech rate, tone, and gesture based on instructional semantics.
2. Unified Framework: Created a system that integrates verbal (prosody, fillers) and non-verbal (gestures) patterns observed in human teaching, moving beyond fixed templates.
3. Comprehensive Evaluation: Conducted a rigorous 2×2 experiment and qualitative analysis to isolate the effects of dynamic speech and gestures on user experience, providing empirical evidence for design guidelines.

4. Results Analysis

Quantitative Findings

• Perceptual Effectiveness (PU, EN, IU): Both dynamic speech and dynamic gestures significantly improved Perceived Usefulness, Learning Engagement, and Intention to Use compared to the static baseline. Condition D (Dynamic + Dynamic) yielded the highest scores across all metrics.
• Social Realism (HL, SP): Dynamic expressions significantly enhanced Human-likeness and Social Presence. However, absolute scores for these metrics remained lower than for effectiveness metrics, suggesting that while dynamic expression helps, it does not yet fully bridge the "uncanny valley" or create a fully human-like presence on its own.
• Emotional Comfort (DC): Dynamic speech and gestures significantly reduced Discomfort (frustration, fatigue). Interestingly, the interaction between speech and gestures was not significant, indicating that each modality independently contributes to reducing negative emotions.
• Interaction Effects: The statistical interaction between speech and gestures was generally non-significant, suggesting that the benefits of combining them are additive rather than multiplicative in the current implementation.

Qualitative Findings

• Monotony vs. Naturalness: Participants found static speech "mechanical" and prone to causing distraction. Dynamic speech (pauses, fillers) provided necessary cognitive processing time.
• Attention Guidance: Dynamic gestures (e.g., emphasizing motions) were crucial for refocusing attention and signaling key information.
• Limitations Identified: Participants noted that the gesture library was too small (leading to repetition) and that transitions between gestures were sometimes stiff. They also suggested a need for better speech-gesture coordination and "interruptible" conversation mechanisms.

5. Significance and Implications

• Design Guidelines: The study provides concrete evidence that dynamic, context-aware multimodal expressions are essential for creating effective VR pedagogical agents. Static agents are insufficient for high-engagement learning.
• Cognitive Load Management: By mimicking human teaching behaviors (pausing for difficult concepts, emphasizing key points), the system helps manage cognitive load and improves information retention.
• Future Directions: The authors highlight that while low-level expression changes (prosody/gestures) are effective, future work must address semantic consistency between modalities, gesture variety, and bidirectional interaction (e.g., interruptibility) to achieve higher levels of social presence and human-likeness.
• Scalability: The proposed framework is applicable beyond education to medical training, corporate learning, and psychological counseling.

In conclusion, this paper demonstrates that integrating LLMs with semantically sensitive prompt engineering to drive multimodal expressions significantly enhances the pedagogical value and user experience of VR agents, transforming them from static information dispensers into more natural, engaging, and effective teaching assistants.