Understanding Self-Regulated Learning Behavior Among High and Low Dropout Risk Students During CS1: Combining Trace Logs, Dropout Prediction and Self-Reports

1. Problem Statement

The introductory programming course (CS1) is a critical bottleneck in Computer Science education, characterized by high dropout rates and significant challenges for novice learners. While predictive learning analytics can identify at-risk students with high accuracy, these models often lack explainability. Educators struggle to understand why a student is predicted to drop out or how their specific learning behaviors contribute to that risk.

Current research often relies on post-course comparisons of high vs. low performers or uses self-reports that lack temporal granularity. There is a need for a method that:

• Links real-time behavioral data (trace logs) with predictive dropout models.
• Distinguishes between temporary struggles and irreversible paths to dropout.
• Integrates self-regulated learning (SRL) theory to interpret these behaviors.

2. Methodology

The study employed a mixed-methods approach combining quantitative learning analytics with qualitative self-reports, grounded in Self-Regulated Learning (SRL) theory (specifically the Winne & Hadwin tactic-strategy framework).

2.1 Data Collection

• Context: A 6-credit CS1 course at the University of Jyväskylä (Fall 2023) with 321 enrolled students.
• Participants: 47 consenting students provided trace logs; 29 of these completed a post-course self-report questionnaire.
• Data Sources:
  • Trace Logs: 86,738 raw events from a Virtual Learning Environment (VLE), including coursebook usage, lectures, task submissions (introductory, core, basic, bonus, guru), TA sessions, and model answer reviews.
  • Performance Data: Weekly task scores, exam scores, and final grades.
  • Self-Reports: Open-ended questions regarding study habits, challenges, and resource usage.

2.2 Data Analysis Pipeline

The authors developed a five-step pipeline to transform raw logs into interpretable learning profiles:

1. Log Preprocessing: Raw events were grouped into student sessions using a 25-minute cut-off threshold (based on estimated task completion time). Sessions with single events were removed to filter noise.
2. Learning Tactic Discovery:
   • Sessions were converted into event frequency vectors (82 distinct event codes).
   • $k$-medoids clustering (using $\ell_1$ distance) was applied to identify 12 distinct Learning Tactics (e.g., "Focusing on course materials," "Attempting core tasks," "Reviewing previous weeks").
3. Learning Strategy Formulation:
   • Sessions were relabeled with their corresponding tactic codes.
   • Logs were split into weekly sub-logs to capture the evolution of strategies over the 11-week course.
   • A weekly sequence of tactics constitutes a Learning Strategy.
4. Learning Strategy Characterization (Risk Grouping):
   • A pre-trained Logistic Regression Dropout Prediction Model (based on task performance) was applied to each student-week to generate a dropout probability ($p_{drop}$).
   • Strategies were labeled High Risk ($p_{drop} > 0.5$) or Low Risk ($p_{drop} \leq 0.5$).
   • Clustering: Strategies within each risk group were clustered using Expectation Maximization (EM) on First-Order Markov Models (FOMM) of tactic transitions.
   • Interpretation: Clusters were visualized using Heuristic Nets (process mining) and SHAP waterfall plots (to explain prediction features) to define distinct strategy types.
5. Student Characterization:
   • Students were clustered based on the frequency of strategy types they used throughout the course (Agglomerative Clustering).
   • Brunner-Munzel tests were used to statistically compare clusters regarding grades, dropout predictions, and self-reported opinions.
   • Qualitative content analysis of self-reports was used to validate and contextualize the quantitative clusters.

3. Key Contributions

• Methodological Integration: A novel pipeline that combines process mining, predictive modeling, and self-reports to explain why a student is at risk, moving beyond simple classification.
• Granular Strategy Taxonomy: Identification of 12 distinct weekly learning strategy types, categorized by risk level:
  • 3 Low-Risk Strategies: Task-oriented (performance-focused), Seeking understanding (resource-heavy), and Resource-focused (materials > tasks).
  • 9 High-Risk Strategies: Ranging from recoverable struggles (e.g., "Slow start," "Attempting with examples") to critical failure modes (e.g., "Falling behind, realizing struggle too late," "Risky focus on mandatory tasks only").
• Longitudinal Profiling: Definition of 5 Student Learning Profile Types based on how students navigated these strategies over the entire semester.
• Triangulation: Demonstrated how self-reports can resolve ambiguities in trace log data (e.g., explaining why a student using "risky" strategies still passed).

4. Key Results

4.1 Low Dropout Risk Behaviors (RQ1)

Students with low dropout risk generally exhibited three main weekly strategies:

1. Task-Oriented: Focused on completing mandatory and basic tasks efficiently.
2. Seeking Understanding: Prioritized deep comprehension via lectures and supplementary resources before attempting tasks.
3. Resource-Focused: Spent significant time on course materials and model answers rather than immediate task completion.

• Profiles: These behaviors correlated with profiles like "Adaptive Understanding Seekers" (who adjusted strategies based on context) and "Active Task-Focused Learners."

4.2 High Dropout Risk Behaviors (RQ2)

High-risk students exhibited nine distinct strategies. Crucially, the study distinguished between temporary and terminal risks:

• Recoverable Risks: Strategies like "Slow start" or "Struggling to understand" were often used by students who eventually recovered and passed.
• Terminal Risks: Strategies like "Falling behind, realizing struggle too late" (viewing model answers without doing tasks) and "Risky focus on mandatory tasks only" were strongly associated with students who dropped out.
• Insight: Students who dropped out typically reverted to high-risk strategies 1–2 weeks before leaving, often after a specific difficult topic was introduced.

4.3 Student Profiles and Self-Reports (RQ3)

The study identified five student profiles:

1. Adaptive Understanding Seekers (N=20): Adjusted strategies based on weekly topics; high engagement with diverse resources.
2. Diligent Course Content Followers (N=14): Strictly followed the suggested schedule; high reliance on TAs and peer groups.
3. Active Task-Focused Learners (N=5): Consistently used low-risk strategies; high task completion rates.
4. Persevering until the End (N=1): An outlier who used high-risk strategies (minimal task work) but passed with top marks due to exceptional exam performance. Self-reports revealed this student had poor time management but strong information retrieval skills.
5. Course Dropouts (N=7): All dropped by the midpoint. They exclusively used high-risk strategies in the weeks leading to departure.

Validation: Self-reports confirmed the quantitative findings. For instance, the "Adaptive" group reported adjusting routines due to personal circumstances, while the "Diligent" group reported heavy reliance on external help. The single "Persevering" outlier's self-report explained their unique path (exam-focused) which trace logs alone might have misclassified as a failure.

5. Significance and Practical Implications

• Targeted Interventions: The study provides a framework for TAs and instructors to identify not just who is at risk, but what specific behavior is causing the risk. For example, a student using "Late reliance on model answers" needs a different intervention (e.g., forcing task completion) than one using "Slow start" (e.g., time management coaching).
• Early Warning Systems: By linking weekly strategy types to dropout probabilities, educators can intervene during the course rather than post-mortem.
• Holistic View: The combination of trace logs and self-reports prevents misinterpretation. A student appearing "risky" in logs might be a "Persevering" learner who simply has a different (but effective) strategy, preventing unnecessary or misdirected interventions.
• Scalability: The pipeline uses existing VLE data and standard machine learning models, making it applicable to large CS1 cohorts without requiring intrusive new data collection methods.

In conclusion, this paper demonstrates that learning strategy types are more predictive and explanatory than raw activity counts. By categorizing behaviors into specific risk profiles, the study offers a actionable roadmap for reducing CS1 attrition through timely, behavior-specific support.