Technical Summary: DIPSER Dataset for In-Person Student Engagement Recognition

Problem Statement

The application of deep learning algorithms to assess student attention in naturalistic ("in-the-wild") settings is hindered by a scarcity of high-quality, multimodal datasets. Existing resources often suffer from significant limitations:

• Contextual Artificiality: Many datasets, such as BAUM-1, are recorded in laboratory settings with actors, potentially exaggerating emotions and lacking the unpredictability of real classrooms.
• Remote Limitations: Prominent datasets like DAISEE and EmotiW rely on online session recordings. These lack the ability to capture physical student interactions, posture, and environmental context.
• Sensor Deficiencies: Current datasets frequently omit Inertial Measurement Unit (IMU) data (e.g., heart rate, accelerometers, gyroscopes), preventing the investigation of correlations between physiological metrics and attention.
• Demographic and Temporal Constraints: Existing collections often feature limited ethnic diversity (e.g., predominantly Indian or Asian subjects) and short video durations (often under 5 minutes), which restricts generalizability and the analysis of sustained engagement.

Methodology

The paper introduces DIPSER (Dataset for In-Person Student Engagement Recognition), a comprehensive multimodal dataset captured in authentic classroom environments at the University of Alicante.

Experimental Design

• Setting: Data was collected in a standard classroom within the Faculty of Education, involving three pre-existing student groups (two undergraduate early childhood education groups and one master's degree group).
• Sessions: Nine distinct experimental activities were conducted, ranging from traditional lectures and news reading to brainstorming, robotics experimentation, and collaborative design tasks.
• Duration: Each session was recorded for five minutes, providing a more substantial temporal window than many existing datasets.

Data Acquisition Setup

The capture infrastructure utilized a centralized system based on Raspberry Pi Model 4-B units to manage data flow from multiple sources:

• Visual Data:
  • Context Cameras: Up to six high-resolution (1280 × 720 pixels) RGB cameras captured the general classroom environment and student posture at ~9 fps.
  • Individual Cameras: Each student was assigned a dedicated camera (640 × 480 pixels) focused on their face to capture micro-expressions at ~10 fps.
• Sensor Data: Each participant wore a Samsung Galaxy Watch 5, transmitting IMU data (Linear Acceleration, Gyroscope, Light Sensor, Rotation Vector) and Cardiac Frequency (Heart Rate) via a local Wi-Fi network to the Raspberry Pis.
• Synchronization: A custom synchronization protocol ensured that camera timestamps and sensor data were aligned with a discrepancy of less than 0.5 seconds, verified via forced brightness changes.

Labeling and Annotation

• Ground Truth: Attention and emotion labels were generated through a rigorous process involving four expert evaluators and one self-evaluation per student.
  • Attention: Measured on a 1–5 scale (1 = low, 5 = maximum).
  • Emotion: Categorized into nine classes (Enjoyment, Hope, Pride, Relief, Anger, Anxiety, Shame, Despair, Boredom) based on Pekrun's and Goldberg's scales.
• Preprocessing: The dataset includes processed files with bounding boxes (face/body), demographic estimates (age/gender/ethnicity), face/hand meshes, gaze estimation, and head pose, generated using tools like YOLO, MiVOLO, MediaPipe, and DeepFace.

Key Contributions

1. Multimodal Integration: DIPSER is the first dataset to combine high-resolution RGB video (both contextual and individual), IMU sensor data, and detailed attention/emotion labels within a single, in-person educational setting.
2. Scale and Diversity: The dataset comprises 1,311,761 images totaling 3,080.73 minutes of uncompressed recording. It includes underrepresented ethnicities (specifically Caucasian subjects in Western educational contexts) and diverse educational levels.
3. Comprehensive Annotation: Beyond raw data, the release includes extensive metadata, including pre-computed bounding boxes, mesh landmarks, and multi-rater labels, facilitating immediate application in machine learning research.
4. Real-World Validity: Unlike lab-based or online datasets, DIPSER captures the complexity of "in-the-wild" interactions, including student movement, collaboration, and varying classroom dynamics.

Results and Technical Validation

The paper presents a qualitative and quantitative validation of the dataset's reliability:

• Synchronization: Verified that the time discrepancy between camera and sensor timestamps is negligible (<0.5s).
• Completeness: Confirmed the dataset captures all intended attributes, including biometric indicators and emotional states.
• Consistency: Data collection methods were uniform across sessions, with no significant data absence attributed to system failures, aside from expected random variability.
• Data Integrity: The dataset structure was validated to ensure efficient retrieval, with consistent data types and formats across all sessions.

Significance

The authors position DIPSER as a critical resource for advancing the field of affective computing and educational data mining. By addressing the deficiencies of prior datasets—specifically the lack of physiological sensors, the artificiality of lab settings, and the limitations of online-only recordings—DIPSER enables:

• The training of deep learning models that can generalize to real-world classroom scenarios.
• The investigation of the correlation between physiological metrics (heart rate, movement) and cognitive states (attention, emotion).
• The analysis of student engagement across diverse educational contexts and demographic groups.

The dataset is made publicly available under a CC-BY license, with strict ethical guidelines ensuring data is used solely for academic and research purposes, adhering to the Declaration of Helsinki and University of Alicante ethics committee approvals.