LLM-based Multimodal Feedback Produces Equivalent Learning and Better Student Perceptions than Educator Feedback

Technical Summary: LLM-based Multimodal Feedback Produces Equivalent Learning and Better Student Perceptions than Educator Feedback

Problem Statement
Providing timely, targeted, and multimodal feedback is critical for effective learning, yet scaling this practice remains a significant challenge due to high student-to-teacher ratios and the time-intensive nature of manually composing coherent, context-relevant messages that integrate text, visuals, and audio. While Large Language Models (LLMs) offer potential for automated, context-aware feedback, existing research largely focuses on single-modality (text-only) outputs. Furthermore, the educational effectiveness of cutting-edge, real-time multimodal LLMs (e.g., streaming audio, low-latency variants like GPT-5) in authentic learning tasks remains under-documented. There is a risk that poorly coordinated multimodal feedback could induce information overload and extraneous cognitive load, hindering rather than helping learning.

Methodology
The authors conducted a randomized controlled online crowdsourcing experiment to compare a novel AI-facilitated multimodal feedback system against a Business-as-Usual (BAU) baseline of fixed, educator-authored feedback.

• System Design: The AI system integrates structured textual explanations with dynamic multimedia resources. It utilizes Retrieval-Augmented Generation (RAG) to dynamically retrieve the most relevant slide pages from a course deck. The feedback interface (embedded via <iframe>) presents:
  1. Visually Enhanced Corrective Feedback: Structured text (score, statement, explanation, advice) generated by GPT-5 (configured with reasoning_effort=low and verbosity=low for latency). Key concepts are highlighted with hover tooltips to reduce cognitive load.
  2. Resource Area: Displays the retrieved relevant slide page and offers an optional streaming AI audio narration (via GPT Realtime API) that connects the question, student answer, and feedback without verbatim duplication.
• Experimental Setup:
  • Participants: 197 university students (US-based) recruited via Prolific.
  • Task: A "learning-by-doing" phase involving 13 activities (8 Multiple-Choice Questions [MCQs] and 5 Open-Ended Questions [OEQs]) based on the Multimedia Principle in e-learning.
  • Conditions:
    • BAU Group: Received pre-written text feedback from educators and a generic link to the full slide deck.
    • AI Group: Received the real-time, multimodal AI feedback described above.
  • Metrics: Learning gains (pre-test vs. post-test), engagement (process logs: submission counts, revision behaviors, time-on-task), and learner perceptions (survey on clarity, specificity, motivation, trust, cognitive load).

Key Contributions

1. System Implementation: A real-time, AI-facilitated multimodal feedback system that integrates structured text, visual retrieval of course materials, and streaming audio narration, designed for easy integration into existing Intelligent Tutoring Systems (ITS) via <iframe>.
2. Empirical Evaluation: A rigorous comparison demonstrating that AI multimodal feedback achieves learning outcomes equivalent to human educator feedback while significantly outperforming it in user perception and engagement patterns.

Results

• Learning Effectiveness (RQ1): Both groups showed significant learning gains from pre-test to post-test. An ANCOVA controlling for pre-test scores revealed no significant difference in learning outcomes between the AI and BAU groups ($p=0.34$). The AI system achieved equivalent pedagogical effectiveness to human feedback.
• Time Efficiency: The AI system demonstrated high time efficiency. While human feedback required significant authoring time, the AI system generated feedback in median times of 0.299 seconds for MCQs and 6.23 seconds for OEQs. The authors estimate the system is approximately 2.77x more time-efficient than authoring equivalent human text feedback, excluding the additional time educators would need to curate multimedia resources.
• Learner Engagement (RQ2):
  • Total time-on-task showed no significant difference between groups.
  • Question-Type Divergence: For MCQs, the BAU group made more submissions (suggesting trial-and-error), whereas the AI group made fewer. For OEQs, the AI group showed a trend toward more submissions, suggesting that targeted AI scaffolding lowered the barrier for iterative revision.
• Learner Perceptions (RQ3): The AI group reported significantly higher satisfaction, lower mental effort, and higher ratings for clarity, specificity, and conciseness. Crucially, there were no significant differences in perceived correctness, trust, or acceptance between the AI and human feedback groups, indicating the AI system reached parity with human feedback on these critical trust metrics.

Significance and Claims
The paper claims that well-designed, real-time AI multimodal feedback can serve as a scalable complement to human instruction. The study demonstrates that AI systems can match the learning gains of traditional educator feedback while offering superior user experiences in terms of clarity, motivation, and reduced cognitive load.

The authors emphasize that their findings support a design strategy of on-demand multimodal augmentation that is:

• Scalable: Capable of providing immediate, targeted support to large cohorts without increasing instructor workload.
• Integrable: Can be embedded into existing digital learning platforms without fundamental changes to course design.
• Effective: Reduces extraneous cognitive load through structured, progressively disclosed information, thereby enhancing the learning experience without compromising trust or accuracy.

The study concludes that while long-term retention and transfer effects require further investigation in authentic classroom settings, the current evidence supports the deployment of such systems to reduce instructor burden and enhance student engagement in real-time.