💻 computer science

From Partial Correctness to Completion: Predicting Learners’ Early Success in Programming Practice

This study proposes and validates a partial-correctness and progress-aware formulation for early success prediction in programming practice, demonstrating that it significantly outperforms traditional binary and ordinal approaches by more effectively capturing learner progression and enabling targeted instructional support.

Original authors: Md. Shahajada Mia, Yutaka Watanobe, Md. Mostafizer Rahman, Md Faizul Ibne Amin, Daniel M. Muepu

Published 2026-07-21

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Md. Shahajada Mia, Yutaka Watanobe, Md. Mostafizer Rahman, Md Faizul Ibne Amin, Daniel M. Muepu

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a friend try to solve a tricky puzzle. In the old days, a teacher might only care about the final moment: did they finish the puzzle or did they give up? That's a simple "yes" or "no" answer. But in the world of computer programming, learning is more like a long, messy journey of trial and error. Students write code, the computer says "wrong," they fix a tiny mistake, the computer says "wrong again but closer," and they keep trying until they finally get it right. This process happens on special websites called "Online Judges," which act like giant, automated grading machines that record every single attempt a student makes.

The big question researchers are asking is: Can we look at a student's messy middle attempts and guess if they are about to succeed, or if they are stuck in a loop of failure? This is a bit like a coach watching a basketball player miss a few shots. If the player is just missing by a tiny bit and their form is improving, the coach knows a basket is coming soon. But if the player is missing wildly and getting frustrated, they might need help. This paper dives into that exact idea, trying to figure out the best way to predict a student's future success based on their current struggle, using math and computer models to spot the difference between "almost there" and "stuck."

The Paper: From "Wrong" to "Right" in the Blink of an Eye

This study is like a detective story set in the digital playground of the Aizu Online Judge (AOJ), a massive website where students practice coding by solving hundreds of problems. The researchers, a team from the University of Aizu and the University of Notre Dame, wanted to solve a specific mystery: Can we predict if a student will solve a coding problem in their next few tries, just by looking at how they are failing right now?

To do this, they didn't just look at the final result (solved vs. not solved). Instead, they tested three different ways of describing a student's progress, kind of like describing a runner's race in three different ways:

The "Binary" View (The Old Way): This is the simplest view. It only sees two things: Did the student get it right? Yes or No. It ignores everything in between. It's like a coach who only cares if you crossed the finish line, ignoring whether you were running or crawling.
The "Ordinal" View (The Ranking Way): This is a bit better. It ranks the student's attempts from "completely wrong" to "almost right." It's like saying, "You missed the basket, but you were closer than last time." It gives a little more detail but still treats the steps as just levels on a ladder.
The "Partial-Correctness & Progress" View (The New, Super-Detailed Way): This is the star of the show. It looks at how much of the problem the student solved in each attempt (like getting 40% of the test cases right) and, crucially, whether they are getting better with every try. It tracks the "gain"—did they improve from the last attempt? It's like a coach who notices not just that you are closer to the basket, but that your jump height is increasing and your aim is steadying out.

The Big Experiment

The researchers took a huge dataset of over 295,000 coding attempts from a course called "Algorithm and Data Structures I" (ALDS1) and a smaller set from a "Computational Geometry" course (CGL). They fed this data into six different computer brain models (machine learning algorithms), including a popular one called Random Forest, to see which of the three "views" of student progress worked best at predicting the future.

What They Found

The results were clear and consistent. The new, detailed "Partial-Correctness & Progress" view (M3) was the winner, beating the other two methods every time.

The Score: When using the best model (Random Forest), the new method achieved a PR-AUC of 75.39%. This is a fancy score that measures how good the model is at finding the "success" cases.
The Comparison: This was a significant jump. The old "Binary" method only scored 70.99%, and the "Ordinal" method scored 71.21%.
The Improvement: The new method improved accuracy by about 6.63% over the binary method and 6.78% over the ordinal method.

In simpler terms, by paying attention to how students were failing and how they were improving, the computer could guess their success much more accurately than by just waiting to see if they eventually got it right.

Why It Matters: The "Risk Groups"

The researchers didn't just stop at scores; they used their winning model to sort students into three "risk groups" to see if it made sense in the real world:

Low-Risk: These students were almost certainly going to solve the problem soon.
Moderate-Risk: They were in the middle, struggling but making progress.
High-Risk: They were likely to keep failing.

The cool part? The new method created the clearest separation between these groups. The difference in success rates between the "Low-Risk" group (who solved it 71.20% of the time) and the "High-Risk" group (who solved it only 24.06% of the time) was huge. The old methods couldn't separate the groups as clearly.

This suggests that the new method can act like a smart early-warning system. If a teacher sees a student in the "High-Risk" zone, they know to step in with a hint or a simpler problem before the student gets too frustrated. If a student is "Low-Risk," the teacher knows to let them keep going or maybe give them a harder challenge.

The Bottom Line

The paper suggests that in programming education, how you fail matters just as much as whether you succeed. By tracking the tiny improvements and the specific types of errors students make, we can build better tools to help them learn. The study confirms that looking at the "partial correctness" and the "progress" of a student's attempts gives us a much clearer crystal ball for their future success than just looking at the final "Pass" or "Fail" button. It turns the messy process of debugging into a roadmap for learning.

Technical Summary: From Partial Correctness to Completion: Predicting Learners' Early Success in Programming Practice

Problem Statement
In online judge (OJ) systems, learners improve programming skills through iterative submissions, receiving automatic feedback and gradually progressing toward a correct solution. While existing educational data mining (EDM) and learning analytics (LA) research has extensively utilized machine learning (ML) to predict broad outcomes like course grades or dropout risks, there is a gap in modeling problem-level early success. Most current approaches rely on binary outcomes (solved/unsolved) or final states, which fail to capture the intermediate progress and evolving capability of learners during the problem-solving process. Specifically, predicting whether a learner will solve a specific problem within the next $K$ attempts, based on their partial correctness and progression patterns, remains underexplored.

Methodology
The study proposes an early-success prediction framework evaluated on submission logs from the Aizu Online Judge (AOJ) system, utilizing two datasets: Algorithm and Data Structures I (ALDS1) and Computational Geometry (CGL). The core of the methodology involves comparing three distinct prediction formulations to determine the most effective representation of learner state:

Binary Outcome (M1): A baseline formulation using only base interaction features (attempt index, time gaps, and submission status indicators like WA, RE, TLE) without explicit correctness scores.
Ordinal Correctness (M2): Incorporates base features plus a discretized correctness level (0–5) derived from the proportion of passed test cases.
Partial-Correctness and Progress-Aware (M3): The proposed formulation. It utilizes base features combined with continuous partial correctness scores and progress-aware metrics, specifically "correctness gain" (the difference in correctness between consecutive attempts) and "peak correctness" (the highest score achieved so far).

Experimental Setup

Data: The study analyzed over 295,000 prediction instances from ALDS1 and over 10,000 from CGL. Data was preprocessed by cleaning logs, transforming skewed features (e.g., logarithmic transformation of time gaps), and standardizing numerical values.
Task: Predict if a learner will solve a problem within the next $K=5$ attempts based on history up to the current attempt.
Models: Six ML classifiers were evaluated under a consistent protocol: Logistic Regression (LR), K-Nearest Neighbors (KNN), Multilayer Perceptron (MLP), Decision Tree (DT), Extreme Gradient Boosting (XGB), and Random Forest (RF).
Validation: A user-disjoint stratified split (80% training, 20% testing) was used to prevent data leakage. Five-fold GroupKFold cross-validation was employed for hyperparameter tuning, with PR-AUC (Precision-Recall Area Under the Curve) as the primary selection metric.

Key Results

Formulation Performance: The proposed M3 formulation consistently outperformed both M1 and M2 across all six classifiers and evaluation metrics (Accuracy, Precision, Recall, F1, ROC-AUC, and PR-AUC).
Model Performance: Random Forest (RF) achieved the best overall performance. On the ALDS1 test set, the RF-M3 combination achieved a PR-AUC of 75.39% and Accuracy of 70.42%. This represented a relative improvement of 6.20% in PR-AUC and 6.63% in Accuracy over the binary-outcome formulation (M1), and 5.87% and 6.78% over the ordinal-correctness formulation (M2).
Robustness: The superiority of M3 was confirmed on the smaller CGL dataset and across different prediction windows ( $K=3, 5, 7$ ), indicating stability across varying data conditions and time horizons.
Statistical Validation: A Logistic Generalized Linear Model (GLM) confirmed that progress-aware features, particularly "Peak Correctness" (Odds Ratio = 1.694) and "Log Time Gap," were statistically significant predictors of early success.
Risk Grouping: Converting predicted probabilities into risk groups (Low, Moderate, High) revealed that M3 provided clearer separation between groups. The low-risk group in M3 showed a 71.20% observed success rate, while the high-risk group showed 24.06%, creating a risk gap of 47.14 percentage points—significantly wider than the gaps produced by M1 or M2.

Significance and Claims
The paper claims that the proposed framework offers a more effective approach for early success prediction by moving beyond binary or ordinal representations to capture the continuum of learner progress. The significance lies in three main areas:

Improved Predictive Accuracy: By incorporating partial correctness and progress signals, the model better reflects the reality that learners often solve problems through gradual refinement rather than immediate success.
Interpretability and Actionability: The risk-group analysis provides an interpretable basis for educational intervention. The distinct separation of risk groups allows educators to identify learners who are struggling (high-risk) versus those progressing well (low-risk) at the specific learner-problem level, rather than relying on fixed learner labels.
Instructional Support: The findings suggest that high-risk instances often exhibit repeated attempts with limited near-term success, indicating a need for timely hints or instructor feedback. Conversely, the framework can help instructors refine problem sets by identifying problems that consistently generate high-risk profiles.

The authors conclude that while the study is limited to AOJ data, the partial-correctness and progress-aware concepts are adaptable to other OJ platforms, MOOCs, and automated assessment systems, offering a pathway for data-driven, personalized support in programming education.

The Paper: From "Wrong" to "Right" in the Blink of an Eye

The Big Experiment

What They Found

Why It Matters: The "Risk Groups"

The Bottom Line

More like this