📊 statistics

Incremental Predictive Value of Item-Adjusted Process Indicators for Mathematics Achievement among Korean Students in PISA 2022: A School-Level Out-of-Sample Validation Study

This study demonstrates that item-adjusted process indicators derived from computer-based assessments significantly improve the prediction of mathematics achievement among Korean students in PISA 2022 beyond traditional questionnaire data, even under a rigorous school-level out-of-sample validation design.

Original authors: Hejing Li¹

Published 2026-09-18

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Hejing Li¹

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Every year, millions of fifteen-year-olds around the world take a massive test called PISA, designed to see how well they can use what they know in real-life situations. For the math portion of this test, students sit at a computer and solve problems. While the final score tells us how many questions they got right, the computer also records a hidden layer of data: exactly how long they spent on each problem, how many times they clicked, and how often they went back to change an answer. For years, researchers have wondered if these digital footprints of behavior could tell us something about a student's ability that a simple questionnaire cannot. We know that asking students about their confidence, their anxiety, and their school environment gives us a good picture of their potential, but we also know that what a student says they do and what they actually do while solving a problem can be very different. The big question is whether the raw, unfiltered record of a student's interaction with the test adds any new, useful information once we have already asked them about their background and feelings.

A researcher in South Korea set out to answer this question with a very careful approach, using data from over 6,000 students who took the 2022 math test. They started by building a strong baseline model using only the information students provided in a survey. This survey covered a wide range of topics, including their family's economic status, how much they believed in their own math skills, how anxious they felt, how much they liked school, and what kind of math problems they had seen in class. This survey data alone proved to be a powerful predictor of the students' final scores. When the researcher looked at the computer records of how students actually worked through the test—how long they took and how many actions they took—they found that this behavioral data was also useful on its own, but it was not as accurate as the survey data. A student's self-reported feelings and background conditions were simply better at predicting their final score than a raw count of their clicks and seconds spent.

However, the story changed when the researcher looked at how these two types of information worked together. The challenge was that the computer records were messy; a long time spent on a problem might mean a student was struggling, or it might just mean that specific problem was harder than the others. To fix this, the researcher adjusted the data so that every student's time and actions were compared only to other students who faced the exact same problem. This removed the influence of the test design itself and left only the student's relative behavior. When they added these adjusted behavioral indicators to the survey model, the prediction of the students' math scores improved significantly. The combined model made errors that were about 11 points lower on the test scale than the survey model alone. This improvement was consistent across different computer algorithms and held true even when the researcher tested the model on schools that were completely new to the system, meaning the model could predict the performance of students it had never seen before.

The researcher was careful to explain what this improvement meant and what it did not mean. They emphasized that the behavioral data did not prove that spending more time or clicking more buttons causes a student to get a better score. Instead, the study showed that the digital record of a student's effort and strategy contains extra clues about their ability that are not captured by asking them questions. The study also ruled out the idea that this improvement was just a result of the test format. By adjusting the data to account for the specific difficulty of each problem, they showed that the value came from the student's actual behavior, not from the fact that the computer test happened to route certain students to harder or easier questions. The findings suggest that in large-scale testing, the way a student interacts with a computer is a valuable supplement to what they tell us about themselves, offering a clearer, more complete picture of their mathematical abilities.

Technical Summary: Incremental Predictive Value of Item-Adjusted Process Indicators for Mathematics Achievement

Problem Statement
Computer-based assessments like PISA 2022 generate process data (e.g., response times, action counts) that offer observed behavioral evidence distinct from final scores and self-reports. However, raw process metrics are confounded by item difficulty, test design, and adaptive routing; a specific duration or action count may signify different behaviors depending on the item context. Furthermore, previous studies often utilized student-level random splits for validation, which risks data leakage in hierarchical data structures where students from the same school share unobserved characteristics. There is a need to determine if process indicators retain predictive value after rigorous item-level adjustment and when evaluated against a strong, pre-specified questionnaire baseline under a stringent school-level out-of-sample validation design.

Methodology
The study analyzed 6,038 Korean students from 186 schools in PISA 2022. The outcomes were the ten mathematics plausible values (PV1MATH–PV10MATH), representing the proficiency distribution.

Predictor Blocks:
- Q-extended: A pre-specified block of 11 questionnaire variables covering antecedent conditions (gender, ESCS), propensity/affect (self-efficacy, anxiety, persistence), experience (familiarity with concepts), and context/opportunity (school belonging, teacher support, disciplinary climate, exposure to tasks).
- P-adjusted: Five student-level indicators derived from public item-level process variables (total response time, time to first action, number of actions, number of visits, number of short visits). These were transformed using a log transformation and item-specific robust standardization (centering and scaling by item median and IQR/1.349) calculated only within the training fold. This adjustment aimed to remove item-level location and dispersion differences before aggregation.
Validation Design: A school-level nested cross-validation was employed. The 186 schools were divided into five outer folds, repeated three times with different assignments. Models were trained on four folds and evaluated on the held-out schools. Crucially, all preprocessing (imputation, item-adjustment) and hyperparameter tuning (via inner cross-validation) were performed strictly within the outer training data to prevent information leakage.
Models: Weighted linear regression served as a benchmark. Three ensemble tree algorithms (Random Forest, XGBoost, LightGBM) were compared. All models utilized the final student weights and Fay-BRR replicate weights to account for the complex survey design and plausible value uncertainty.
Metrics: Performance was measured via weighted Root Mean Squared Error (RMSE), Mean Absolute Error (MAE), and $R^2$ . Incremental value was assessed by comparing the combined model (Q-extended + P-adjusted) against the questionnaire-only model (Q-extended).

Key Results

Baseline Performance: The questionnaire-only model (Q-extended) consistently outperformed the process-only model (P-adjusted) across all algorithms. For XGBoost, Q-extended achieved an $R^2$ of .426, while P-adjusted achieved .356.
Incremental Predictive Value: Adding the item-adjusted process block to the questionnaire model yielded substantial improvements.
- RMSE Reduction: Decreased by approximately 11.25 to 11.72 PISA score points across the ensemble algorithms.
- $R^2$ Increase: Increased by .151 to .159.
- Consistency: The improvement direction was identical in all 90 comparisons (3 algorithms × 3 repetitions × 10 plausible values).
- Statistical Significance: Design- and plausible-value-adjusted 95% confidence intervals for the improvements excluded zero (e.g., XGBoost RMSE improvement: 11.32 [9.76, 12.88]).
Robustness: Similar improvements were observed in the weighted linear regression benchmark (RMSE reduction of 11.25), indicating the gain was not an artifact of non-linear tree ensembles. Sensitivity analyses confirmed the results held for the linear test path only, complete cases, expanded questionnaire baselines, and fixed hyperparameters.
Item Adjustment Impact: Unadjusted process variables yielded higher raw predictive performance ( $R^2$ = .440 for P-raw) than adjusted ones ( $R^2$ = .356 for P-adjusted). However, the author notes that raw values likely captured test-path and item-exposure signals, whereas the adjusted values provided a more conservative estimate of comparable student behavior.

Significance and Claims
The paper claims that item-adjusted process indicators provide incremental out-of-sample predictive information for mathematics achievement beyond a broad, pre-specified questionnaire model. The study demonstrates that this value persists even under a stringent validation design that accounts for school-level clustering, survey weights, and plausible value uncertainty.

The author explicitly states that these findings:

Do not establish causality: The study is predictive, not causal; improved prediction does not imply that manipulating response times or actions would change achievement.
Do not validate psychological constructs: The indicators are not proven to directly measure effort, strategy, or engagement; they are supplementary behavioral evidence.
Are conditional: The results are specific to the PISA 2022 Korean sample with observed process data and the specific questionnaire baseline used.
Support supplementary use: Process data should be viewed as a supplement to well-specified questionnaire information rather than a replacement, particularly when evaluated with rigorous leakage-controlled validation.

The study positions itself as a methodological follow-up to prior Korean research (Woo & Choi, 2025), refining the evidence by introducing item-level adjustments and a more conservative validation framework to ensure the predictive value is not merely an artifact of test design or data leakage.

Technical Summary: Incremental Predictive Value of Item-Adjusted Process Indicators for Mathematics Achievement

More like this