Assessing the Social Validity of Canvas LMS-Based Cybersecurity Online Modules in High School STEM Classes

Technical Summary: Assessing the Social Validity of Canvas LMS-Based Cybersecurity Online Modules in High School STEM Classes

Problem Statement
Cybersecurity has become an integral component of daily life, necessitating urgent public awareness and education. Despite the integration of cybersecurity into high school STEM curricula, there is a lack of research measuring the social validity—the degree of social acceptance and appropriateness of intervention goals, procedures, and outcomes—of online cybersecurity modules delivered via Learning Management Systems (LMS). Furthermore, the extent to which students' technology self-efficacy influences their perception of these modules remains under-investigated. This study addresses the need to evaluate the social validity of Canvas LMS-based cybersecurity modules for high school students and to determine if technology self-efficacy acts as a significant predictor of this validity.

Methodology
The study employed a mixed-methods approach involving 30 high school students (grades 9–12) from a public STEM school in the southern United States. The intervention consisted of ten online modules developed on the Canvas LMS, covering ten units ranging from cyber ethics and privacy to metaverse security and career frameworks. The modules utilized instructional design components including sequenced videos (5–10 minutes), hands-on activities, simulations, online discussions, and quizzes.

• Data Collection: Participants completed a survey containing five 5-point Likert scale questions measuring social validity (e.g., improved understanding, interest, and confidence) and three questions measuring technology self-efficacy (navigating tools, following learning sequences, and self-regulation). Additionally, five open-ended questions gathered qualitative data on preferred/least preferred topics and overall experiences.
• Quantitative Analysis: Due to the small sample size and the ordinal nature of the Likert data (collapsed to a 4-point scale as no "strongly disagree" responses were recorded), the authors utilized a cross-classified Bayesian cumulative link mixed effects model (CLMM). This model accounted for the hierarchical structure of the data (items nested within students) and employed weakly informative priors to prevent overfitting. The model estimated logit coefficients and odds ratios for the relationship between technology self-efficacy predictors and social validity ratings.
• Qualitative Analysis: Open-ended responses were analyzed using word network analysis. Text preprocessing involved tokenization, lowercasing, and stop-word removal. Co-occurring words were mapped into undirected graphs, and centrality metrics (degree, betweenness, closeness, and eigenvector) were calculated using the igraph package. Community structures were identified using the Leiden algorithm to visualize clusters of sentiment and topic association.

Key Results

• Social Validity Ratings: The Bayesian CLMM revealed that students rated social validity more frequently in the lower to mid-range (ratings 3 or lower) compared to the highest rating (5). Specifically, the cumulative log odds of a rating being 4 or lower were significantly higher than a rating of 5 (logit coefficient = 6.09; Odds Ratio [OR] = 441.18). No students selected "strongly disagree," and responses were generally positive, though sparse at the extremes.
• Impact of Technology Self-Efficacy: The study found a statistically significant positive relationship between technology self-efficacy and social validity ratings.
  • Self-Regulation: The strongest predictor was self-regulation and monitoring behavior during instruction. For every one-scale increase in self-regulation self-efficacy, the odds of rating higher social validity increased by 3.70 times (logit coefficient = 1.31; OR = 3.70; 95% CI = 1.06 to 16.09).
  • Learning Sequence: Following a learning sequence effectively also increased the odds of a higher rating by 2.40 times (OR = 2.40).
  • Navigation: Confidence in navigating tools showed a minimal positive effect (OR = 1.17).
• Word Network Findings:
  • Preferred Topics: Words such as "cybersecurity," "metaverse," and "online" showed high degree centrality and were connected to positive sentiment words like "interesting," "enjoy," and "good." Topics involving "types of hackers" and "cyber ethic" were also favored.
  • Least Preferred Topics: Words like "information," "long," "video game," and "metaverse" were connected to negative sentiment words ("least," "dislike"). Students specifically cited the length of video content and discussion sections as deterrents.
  • Overall Experience: The word "enjoy" showed high centrality, connected to "learn" and "program," while "information," "keep," and "safe" formed distinct clusters reflecting the utility of the course.

Key Contributions

1. Methodological Application: The study demonstrates the utility of Bayesian cumulative link mixed effects models for analyzing social validity data in small-sample educational settings, offering a robust alternative to frequentist approaches when dealing with ordinal data and complex random effect structures.
2. Integration of Word Networks: By applying word network analysis to open-ended survey responses, the study provides a visual and quantitative validation of student sentiment patterns, linking specific curriculum topics (e.g., metaverse, ethical hacking) to positive or negative affective responses.
3. Empirical Evidence on Self-Efficacy: The research empirically establishes that technology self-efficacy, particularly in the domain of self-regulation and monitoring, is a significant predictor of social validity in online cybersecurity education. This suggests that students who feel capable of managing their own learning process in a digital environment are more likely to perceive the intervention as valid and appropriate.

Significance and Claims
The authors claim that the study validates the social validity of using Canvas LMS-based modules as a supplementary instructional method for high school STEM cybersecurity education. The findings suggest that while the modules are generally accepted, their effectiveness is closely tied to students' ability to self-regulate within the digital learning environment.

The paper modestly concludes that while the results indicate overall positive sentiments and the potential effectiveness of this approach, the study has limitations. The small sample size (N=30) necessitated the use of Bayesian methods, and the lack of pre/post-test data means the study measures perception (social validity) rather than knowledge acquisition. The authors note that some students found the content length and asynchronous nature insufficient for deep understanding, suggesting a need for explicit teacher guidance and hands-on demonstrations in future iterations. Ultimately, the study supports the democratization of cybersecurity education through online platforms but highlights the critical role of student self-efficacy in the success of such interventions.