Technical Summary: Exploring Factors Affecting Student Learning Satisfaction during COVID-19 in South Korea

Problem Statement

The COVID-19 pandemic necessitated a rapid shift to online learning in South Korean higher education, creating a complex environment for student engagement and satisfaction. While existing research has identified significant factors influencing learning satisfaction—such as self-efficacy, performance, and engagement—traditional statistical models often fail to explain the intricate, non-linear relationships among these variables. Furthermore, there is a need to understand how specific cohorts (e.g., STEM vs. Humanities, Online vs. Offline) differ in their satisfaction levels and how individual learner characteristics, instructional design, and environmental factors interact to shape these outcomes. This study addresses the gap in deep, interpretable analysis of these multifaceted relationships during the pandemic.

Methodology

The study employed a mixed-methods approach combining statistical hypothesis testing with interpretable machine learning.

Data Collection:

• Participants: 302 students from Sungkyunkwan University in South Korea.
• Timeline: Data was collected between November 2021 and November 2022, covering four semesters of pandemic-era learning.
• Demographics: The sample included a balanced mix of genders and majors, with 88% full-time domestic students and 12% exchange students.
• Instrument: An online survey utilizing 5-point Likert scales, binary (Yes/No), and categorical questions.
• Features: Data was categorized into three domains:
  1. Individual Learner Characteristics: Self-efficacy, perceived performance, motivation, emotional state (e.g., boredom, missing out).
  2. Instructional Design Elements: Suitability of methods, instructor support, feedback mechanisms.
  3. Social and Environmental Factors: Peer interaction, study space availability, stress coping mechanisms, and technological tools.

Analytical Framework:

1. Statistical Comparison: A two-sided Wilcoxon Rank Sum test was applied to identify significant differences in learning attributes between two distinct paired groups:
   • Online vs. Offline course takers.
   • STEM (Science, Technology, Engineering, Medicine) vs. HASS (Humanities and Social Sciences) majors.
2. Machine Learning Modeling: An Explainable Boosting Machine (EBM), a type of Generalized Additive Model (GAM) with interactions, was utilized to predict learning satisfaction.
   • Target Variable: A binary classification of satisfaction derived from an OR combination of perceived material value (m_valuable) and task performance satisfaction (m_taskSatisfaction).
   • Data Preprocessing: The dataset (302 samples) was split 80/20 (241 training, 61 testing). Due to class imbalance, Synthetic Minority Over-sampling Technique (SMOTE) was applied to the training set.
   • Interpretability: The model's decision-making process was analyzed via feature importance scores (FIS), single feature plots, and pairwise interaction heatmaps to uncover non-linear relationships and feature interactions.

Key Results

1. Group Comparisons (Wilcoxon Rank Sum Test):

• Online vs. Offline: Students attending offline classes demonstrated significantly higher learning satisfaction than those in online classes. Online students reported significantly higher levels of boredom, frustration regarding lack of instant feedback, and a feeling of "missing out" on proper learning.
• STEM vs. HASS: STEM majors reported significantly higher satisfaction levels across seven attributes compared to HASS majors. The study attributes this to STEM students' higher familiarity with technology and course structures that may translate more easily to online formats (facts/theories) compared to HASS courses (critical thinking/discussion).

2. Machine Learning Findings (EBM Model):

• Model Performance: The EBM model achieved 95.08% accuracy and an F1 score of 94.33, indicating high reliability in identifying factors influencing satisfaction.
• Top Five Predictive Factors:
  1. Perceived performance (m_selfEval).
  2. Perception of participating in class activities (m_activity).
  3. Study major (specifically being a STEM major).
  4. Ability to conduct discussions with classmates (m_discuss).
  5. Study space availability at home.
• Feature Interactions:
  • Performance vs. Activity: Students with high self-perceived performance remained satisfied even if they found class activities unhelpful. Conversely, students with low perceived performance who also found activities unhelpful were the most dissatisfied.
  • Major vs. Activity: STEM students maintained satisfaction regardless of their view on class activities. HASS students, however, showed a strong dependency on finding class activities beneficial to maintain satisfaction.
  • Peer Discussion as a Buffer: Students with low self-perceived performance but high ability to discuss with peers remained content, suggesting peer interaction can compensate for perceived academic struggles.

Key Contributions

• Interpretability in AI for Education: The study demonstrates the utility of Explainable AI (specifically EBM) in educational research, moving beyond "black box" predictions to reveal how specific factors and their interactions drive student satisfaction.
• Cohort-Specific Insights: It provides empirical evidence that the impact of learning modes and majors is not uniform; STEM and HASS students react differently to online environments, and online vs. offline modes yield distinct satisfaction profiles.
• Nuanced Factor Analysis: The research highlights that while perceived performance is the strongest predictor, the interaction between performance, major, and peer engagement is critical. For instance, the ability to discuss with peers can mitigate the negative effects of low self-perceived performance.

Significance and Claims

The paper claims that its findings offer actionable insights for institutions and educators to tailor teaching strategies. Specifically:

• Differentiated Strategies: Educators should recognize that STEM and HASS students have different needs and satisfaction drivers, particularly regarding class activities and online formats.
• Intervention Points: The study identifies that fostering peer discussion and supporting students' self-perception of performance are critical levers for improving satisfaction, especially for HASS students and those struggling academically.
• Limitations and Future Work: The authors modestly note that the study is limited to one university and specific pandemic conditions. They propose future work to examine other aspects like motivation in greater depth and to utilize explainable deep learning models for further comparative analysis. The study does not claim to solve all online learning challenges but provides a data-driven framework for understanding the complex interplay of factors affecting student satisfaction during crisis periods.