💻 computer science

EduGage: Methods and Dataset for Sensor-Based Momentary Assessment of Engagement in Self-Guided Video Learning

This paper introduces EduGage, a multimodal dataset and system that leverages wearable and camera-based sensors to estimate momentary learner engagement in self-guided video learning, demonstrating that lightweight combinations of behavioral and physiological signals outperform various baselines while releasing a comprehensive resource for reproducible research.

Original authors: Zikang Leng, Edan Eyal, Yingtian Shi, Jiaman He, Yaqi Liu, Thomas Plötz

Published 2026-05-05

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zikang Leng, Edan Eyal, Yingtian Shi, Jiaman He, Yaqi Liu, Thomas Plötz

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Reading the "Vibe" of a Student

Imagine you are watching a friend study a video about rocket science. Sometimes they look totally focused, nodding along. Other times, they look bored, or maybe they are staring at the screen but their mind is wandering to what they had for lunch.

In a classroom, a teacher can see this. But in self-guided learning (watching a video alone on your laptop), no one is there to see if you are actually "getting it" or just zoning out.

This paper asks: Can we use wearable gadgets to "read" a student's engagement level in real-time, second-by-second, without a teacher looking over their shoulder?

The researchers built a system called EduGage to answer this. They didn't just guess; they built a "super-sense" using a bunch of sensors to see if they could predict when a student was struggling to pay attention.

The Experiment: The "Wearable Zoo"

To test this, the researchers gathered 16 college students and put them in a lab. They made the students watch educational videos (like a mini-lecture on X-rays or rocket equations).

But here's the twist: The students were covered in sensors like a sci-fi character. They wore:

A smart ring on their finger (measuring pulse and temperature).
A chest strap (like a heart-rate monitor for athletes).
A wristband (measuring sweat and heart rate).
An earbud (measuring movement).
A headband (measuring brain waves and eye movement).
A webcam (tracking where their eyes were looking).

The "Check-In" Game:
Every minute or so, the video would pause, and the students would get a vibration. They had to quickly answer one simple question: "How hard was it to pay attention during the last minute?" (1 = Easy, 5 = Very Hard).

This created a "ground truth" map: The researchers knew exactly when the students felt engaged and when they felt distracted.

The Challenge: Connecting the Dots

The researchers had a mountain of data: heartbeats, brain waves, eye movements, and sweat levels. They wanted to build a computer program (an AI) that could look at the sensors and guess the answer the student would give.

Think of it like trying to guess if a car engine is running smoothly just by listening to the sound of the tires, feeling the vibration of the steering wheel, and looking at the temperature gauge, without ever opening the hood.

The Results: The "Smart Fusion" Wins

The researchers tried many different ways to build this AI:

The "Gut Feeling" Models: Simple math that looked at averages (like "was the heart rate high?").
The "Deep Learning" Models: Complex AI that tried to learn patterns from raw data.
The "Big Brain" Models: Using massive pre-trained AI models (Foundation Models) that had learned from millions of other medical or time-series datasets.
The "Chatbot" Models: Asking a Large Language Model (LLM) to read a summary of the data and guess the answer.

The Winner:
The best system was a custom-built model that acted like a smart conductor.

Instead of treating all sensors equally, it learned that some sensors are more important at some times than others.
For example, if the brain-wave sensor was noisy, the model learned to trust the heart-rate sensor more. If the eye-tracking was blurry, it leaned on the movement sensors.
It combined these signals dynamically, creating a "weighted average" that was much smarter than just adding them all up.

The Score:
The model was pretty good at guessing the students' feelings. It was accurate enough to tell the difference between "I'm cruising" and "I'm struggling." It beat all the other methods, including the fancy pre-trained AI models and the chatbots.

The Reality Check: It's Noisy, Not Perfect

The authors are very honest: This isn't a magic mind-reading device.

The "Noisy" Reality: The sensors aren't perfect. Sometimes a student is thinking hard (high cognitive load), but their body is calm. Sometimes they are distracted, but their heart rate is steady. The model can't get 100% accuracy because human attention is messy.
The "Too Many Gadgets" Problem: The study used a "zoo" of devices. The researchers realized that in the real world, nobody wants to wear a chest strap, an earbud, a headband, and a ring just to watch a YouTube video.
The Sweet Spot: They found that you don't need everything. A combination of a smart ring (for pulse/temperature) and a webcam (for eye tracking) might be enough to get a decent estimate. You don't need the heavy-duty medical gear for everyday use.

The Gift to the World: The EduGage Dataset

Because this kind of data is hard to collect (it's expensive and annoying to wear all those sensors), the researchers are releasing their entire dataset to the public.

They are giving away the synchronized sensor data, the video clips, the quiz scores, and the student answers.
This allows other scientists to test their own ideas without having to strap 16 people into a "wearable zoo" themselves.

Summary

EduGage is a study that proves we can estimate how engaged a student is while watching a video by using wearable sensors.

The Method: Use a mix of heart rate, brain waves, eye movement, and sweat data.
The Secret Sauce: Use a smart AI that knows which sensor to trust at any given moment.
The Catch: It's not perfect, and you don't need a full medical lab setup to do it; a few lightweight sensors might be enough.
The Legacy: They are sharing their data so everyone else can try to build better "engagement detectors" for the future of online learning.

Technical Summary: EduGage: Methods and Dataset for Sensor-Based Momentary Assessment of Engagement in Self-Guided Video Learning

Problem Statement

Learning effectiveness relies heavily on learner engagement, a multidimensional construct involving behavioral, emotional, and cognitive dimensions. However, engagement is difficult to measure because these dimensions often do not align and are not directly observable; for instance, a learner may appear behaviorally attentive while mind-wandering. While prior work has explored automatic engagement detection using visual and physiological signals, most studies focus on classroom environments, broader session-level ratings, or coarse observer-based annotations. There is a significant gap in understanding whether wearable sensing can support fine-grained, segment-level engagement tracking in self-guided, asynchronous video learning contexts. In such settings, learners must self-regulate attention without continuous instructor feedback, making it crucial to identify specific moments of disengagement rather than relying on aggregate session scores. Furthermore, it remains unclear which sensing configurations offer the best balance between predictive value and practical deployment feasibility.

Methodology

Study Design and Dataset (EduGage)

The authors conducted a user study with 16 college participants in a laboratory setting designed to approximate self-guided video learning. Participants watched four instructional videos (approx. 10 minutes each) from the MIT Open Learning Library across four domains (X-Rays, Aerospace Engineering, Environmental Science, Business).

Sensing Setup: Participants wore multiple concurrent devices to capture a wide range of physiological and behavioral signals:
- EEG: Muse S Athena (256 Hz).
- ECG: Polar H10 chest strap (130 Hz).
- PPG & Temperature: $\tau$ -Ring (25 Hz) and Muse S Athena (64 Hz).
- EDA & Heart Rate: Microsoft Band 2 (5 Hz EDA, 2 Hz HR).
- IMU: Muse S Athena (52 Hz) and eSense earable (50 Hz).
- Eye Tracking: Webcam-based Beam Eye Tracker (30 Hz).
Ground Truth: Engagement was measured via Ecological Momentary Assessment (EMA). Approximately every minute, the video paused, and participants provided a single-item self-report on a 5-point Likert scale regarding the "difficulty of paying attention" during the preceding segment. Lower scores indicated effortless attention; higher scores indicated high effort/difficulty.
Dataset: The resulting EduGage dataset contains 64 video sessions, 715 probe-aligned windows, and approximately 12 hours of synchronized multimodal recordings. It includes sensor streams, probe-aligned labels, video metadata, and quiz results.

Modeling Framework

The authors developed a multimodal engagement prediction framework operating on probe-aligned windows (44 seconds preceding each probe).

Architecture: The model employs modality-specific temporal encoders to generate latent embeddings for each sensor stream.
Gating Mechanism: A context-informed gated fusion module adaptively weights the contribution of each modality based on its embedding and the video progress context. This allows the model to handle heterogeneous signal quality and missing data by learning soft contribution weights rather than forcing early fusion.
Prediction: The fused representation is mapped to a scalar engagement score via a regression head.

Experimental Evaluation

The framework was evaluated using participant-based 4-fold cross-validation. It was compared against:

Sensor-free baselines: Mean, mode, and random distribution sampling.
Statistical regressors: Random Forest, Linear Regression, LightGBM, and SVM on hand-crafted features.
Deep temporal baselines: DeepConvLSTM, DeepConvLSTM with Self-Attention, and TinyHAR on raw sequences.
Foundation-model transfer: Using frozen encoders (PulsePPG, ECGFounder, NeuroLM, NormWear, MOMENT) with linear or gated fusion heads.
LLM-based few-shot reasoning: A multi-agent framework using GPT-5-mini to reason over structured sensor summaries.

Key Results

Performance: The proposed modality-aware framework achieved the best performance across all metrics:
- MAE: 0.81 (vs. 1.03 for best statistical regressor and 1.12 for best deep baseline).
- Within-1 Accuracy: 83.75%.
- Binary Accuracy: 73.93%.
- Binary Macro-F1: 68.45%.
Baseline Comparisons:
- The model significantly outperformed sensor-free baselines and statistical regressors, indicating that raw temporal dynamics and modality-specific integration capture meaningful engagement signals.
- Foundation models (frozen encoders) underperformed the task-specific supervised model (MAE 1.25–1.54), suggesting that general biosignal pretraining does not automatically transfer to fine-grained engagement estimation without task-specific adaptation.
- LLM-based few-shot reasoning failed to capture fine-grained distinctions (MAE 1.42), indicating that in-context reasoning on structured summaries is insufficient for this task compared to supervised learning.
Modality Analysis:
- Full multimodal setup yielded the best overall balance.
- Ring PPG was identified as the most critical single modality; its removal caused the largest performance drop.
- Muse EEG and IMU signals also contributed significantly.
- EDA-only configurations achieved high within-1 accuracy but lower binary Macro-F1, suggesting EDA captures coarse arousal trends but struggles to distinguish specific attention-difficulty states.
- Redundancy: Removing Muse PPG had a minimal effect, suggesting redundancy with other cardiovascular streams.

Significance and Claims

The paper claims that fine-grained engagement estimation is feasible but inherently noisy. The results demonstrate that wearable and contextual signals can approximate moment-to-moment engagement dynamics, though perfect agreement with subjective self-reports is not expected due to the latent nature of engagement and the indirectness of physiological signals.

Key contributions include:

The EduGage Dataset: A release of synchronized multimodal sensor data with dense, segment-level self-reports, addressing the lack of fine-grained, self-guided learning datasets in the field.
Methodological Insight: Evidence that modality-aware integration (preserving stream-specific structure before fusion) outperforms simple early fusion or transfer learning from foundation models for this specific task.
Deployment Guidance: The study suggests that practical systems do not require full multimodal instrumentation. Instead, lightweight combinations of behavioral (e.g., eye tracking) and physiological (e.g., PPG, EDA) signals may offer a sufficient balance between predictive performance and deployment burden.
Practical Application: The authors posit that such systems can support adaptive learning by identifying moments of sustained attention difficulty, enabling targeted interventions (e.g., prompting reflection or review) without replacing the role of teachers.

The authors maintain a modest stance, acknowledging that their study was conducted in a controlled laboratory setting and that future work must extend these findings to larger, more naturalistic environments. They emphasize that engagement estimates should be treated as uncertain indicators rather than definitive judgments.