📊 statistics

Measurement-bounded predictive inference in international large-scale assessment: how the plausible-value ceiling governs attainable accuracy and the stability of predictor rankings in PISA 2022

This study demonstrates that the measurement precision ceiling inherent in PISA 2022 plausible values fundamentally limits attainable predictive accuracy and destabilizes predictor rankings across domains and education systems, necessitating that performance comparisons and model interpretations be grounded in these computable measurement bounds.

Original authors: Jonas A. Mandalunes

Published 2026-08-28

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jonas A. Mandalunes

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Every year, millions of teenagers around the world take a massive test called PISA, designed to see how well they understand math, reading, and science. The results are used by governments and educators to judge which schools and countries are doing the best job. In recent years, researchers have started using these test scores to build computer models that try to predict a student's success based on their background, such as their family's income, the books in their home, or their school's climate. These studies often produce a ranked list of the most important factors, suggesting that if we want to improve scores, we should focus on the top items on that list. However, there is a hidden problem with this approach. The test scores used in these studies are not direct observations of a student's ability. Instead, they are statistical estimates, created by drawing ten different possible numbers for each student from a cloud of possibilities. Because these numbers are estimates, they contain a certain amount of built-in noise, or uncertainty, that no amount of data can ever explain away. This uncertainty sets a hard limit on how accurately any model can predict a student's performance, a limit that changes depending on the subject being tested and the country where the test was taken.

A new study by Jonas Mandalunes investigates exactly how much this hidden noise limits our ability to learn from these massive tests. The researcher used the international database from the 2022 PISA cycle, which includes data from over 600,000 students across 80 different education systems. The study focused on four areas: mathematics, reading, science, and a newer addition called creative thinking. The core question was simple: how much of a student's score can a computer model actually explain, and does this limit change the list of factors that seem to matter most? To answer this, the researcher calculated a "ceiling" for each subject in each country. This ceiling represents the maximum possible accuracy any prediction model could ever achieve, simply because the test scores themselves are not perfect. The study found that this ceiling varies wildly. For creative thinking, the limit ranged from about 65 percent to nearly 90 percent depending on the country. For mathematics, the range was tighter but still varied, from about 81 percent to 93 percent. This means that in some places, no model can ever explain more than two-thirds of the differences in student scores, while in others, the limit is much higher.

The study went further to see if this limit affects the rankings of the factors that influence scores. The researcher ran the same computer models over and over again, each time using a different one of the ten possible score estimates for the students. In the most precisely measured subjects, the list of top factors stayed mostly the same. But in the less precise subjects, like creative thinking, the list changed dramatically. When the researcher switched from one score estimate to another, more than 40 percent of the top five factors on the list would swap places. This means that a study reporting a single list of "most important" factors is essentially showing just one random snapshot of a much wider, shifting reality. If a researcher had picked a different score estimate, they might have concluded that a completely different set of factors was driving student success.

Another critical finding concerned how the data was split up to test the models. Many studies randomly mix students from different schools into training and testing groups. The new research showed that this method creates a false sense of accuracy. Because students in the same school tend to be similar, a model that sees students from a specific school during training can learn patterns specific to that school, rather than learning about the students themselves. When the researcher forced the model to keep entire schools separate during training and testing, the predicted accuracy dropped significantly. In some cases, the drop was massive, revealing that previous studies had inflated their results by as much as 0.36 points simply because they did not account for the school structure. The size of this inflation depended on how many schools were in the study, not on the complexity of the computer model used.

The study also looked at why the precision of the scores differed so much between subjects. It found that the more a test relied on human graders to score open-ended answers, the less precise the scores tended to be, especially in very large countries. Creative thinking, which was scored entirely by humans reading written responses, showed the widest variation in precision. In contrast, mathematics, which is mostly scored by machines, was more consistent. This suggests that the way a test is administered and scored is just as important as the questions themselves when it comes to how much we can learn from the results.

The implications of these findings are straightforward for anyone trying to understand education data. You cannot compare the performance of a model in one country to another, or in one subject to another, without first knowing the precision ceiling for that specific context. A model that explains 45 percent of the variance in a country with a low ceiling is actually performing quite well, capturing most of what is possible to know. The same model in a country with a high ceiling might be performing poorly, leaving much of the signal unexplained. Furthermore, any list of important factors derived from these tests should be treated with caution. Because the rankings shift so much depending on which score estimate is used, a single list is not a stable fact but a temporary realization. The study concludes that researchers should always report the precision ceiling alongside their results and should test their models across all possible score estimates to see if their conclusions hold up. Without these checks, the rankings and comparisons that guide education policy may be built on a foundation of statistical noise rather than solid evidence.

Technical Summary: Measurement-Bounded Predictive Inference in PISA 2022

Problem Statement
Research utilizing International Large-Scale Assessments (ILSAs) like PISA increasingly employs flexible predictive models to identify student, family, and school characteristics associated with achievement. However, three structural properties of these assessments complicate the interpretation of such studies, often leaving them unaddressed:

Unobserved Achievement: Student achievement is not directly observed but represented by "plausible values" (PVs) drawn from a posterior distribution. Consequently, a portion of the outcome variance is measurement noise that no predictor can explain.
Clustered Sampling: Students are sampled in school clusters. Random student-level data splits allow models to learn school-level structures that reappear in evaluation samples, artificially inflating performance estimates.
Non-Comparable Signal: Reported predictive accuracy and predictor rankings are often compared across education systems as if the quantity of learnable signal were constant, ignoring that measurement precision varies by domain and system.

While the principle that measurement unreliability attenuates associations is established (Spearman, 1904), and noise ceilings have been derived for specific contexts (e.g., brain-behavior prediction, automated essay scoring), it remains unestablished how these factors govern the attainable accuracy and the stability of predictor rankings across the diverse domains and education systems within ILSA databases.

Methodology
The study utilizes the PISA 2022 international database, covering 80 education systems for mathematics, reading, and science, and 63 systems for creative thinking.

Measurement-Precision Ceiling ( $\rho$ ): The author estimates a ceiling for attainable predictive performance for every domain in every system. This ceiling is defined as the intraclass correlation (ICC) of the ten released plausible values treated as repeated measures within a student. The estimator applies a finite-draw correction to account for the fact that the variance of student means is inflated by residual measurement noise ( $1/M$ rather than zero).
$\rho = \frac{\text{Var}(\mu)}{\text{Var}(\mu) + E(\sigma^2)}$
where $\mu$ is the posterior mean and $\sigma^2$ is the posterior variance.
Validation Architecture: To prevent leakage of school-level structure, all primary estimates use school-disjoint five-fold cross-validation. Entire schools are assigned to folds, ensuring no school contributes to both training and evaluation. This is contrasted with conventional random student-level splits to quantify inflation.
Models: Two models were fitted in parallel on identical folds: a weighted ridge regression (linear baseline) and a weighted histogram-based gradient boosting regressor. Hyperparameters were held fixed across all systems and domains to ensure paired comparisons were interpretable.
Stability Analysis: The pipeline was refitted once for each of the ten plausible values. Predictor importance was assessed via weighted permutation importance. Stability was measured using the mean pairwise Kendall rank correlation and Jaccard overlap of the top five predictors across the ten refits.
Attribution of Instability: A null design compared three conditions to distinguish instability caused by measurement noise from instability caused by the permutation importance estimator itself: (1) varying PVs with fixed estimator seed, (2) fixed PV with varying estimator seed, and (3) both fixed.
Paired Within-System Design: To isolate the effect of measurement precision from other system differences, the study paired domains within the same system, holding students, schools, predictors, and folds constant. Only the measured construct (and thus $\rho$ ) varied.

Key Contributions

Estimation of Individual-Level Precision: The study provides a measurement-precision ceiling ( $\rho$ ) for every domain in every participating system, demonstrating that this individual-level statistic is distinct from published group-statistic reliability measures (which correlate weakly, $r=0.195$ ).
Demonstration of Non-Comparability: The author shows that $\rho$ varies significantly between systems and domains (ranging from 0.654 to 0.933) and is weakly correlated across domains within the same students. Therefore, raw predictive performance is not comparable across domains or systems without referencing this bound.
Graded Relationship: Using a paired design, the study establishes that both attainable accuracy and the stability of predictor rankings scale with the measurement ceiling in a graded manner. As the difference in ceilings between two domains approaches zero, the difference in downstream performance and ranking stability also approaches zero.
Quantification of Split Inflation: The study quantifies the cost of random student-level splits in clustered data, identifying the number of sampled schools as the governing parameter for performance inflation.

Results

Ceiling Variability: The measurement-precision ceiling ranged from 0.654 to 0.894 for creative thinking, 0.766 to 0.915 for reading, 0.790 to 0.916 for science, and 0.811 to 0.933 for mathematics. The median ceilings ordered as Mathematics (0.900) > Science (0.871) > Reading (0.857) > Creative Thinking (0.802).
Orthogonality: Within the 63 systems assessed in both creative thinking and mathematics, the ceilings correlated weakly ( $r=0.148$ ), indicating that measurement quality in one domain is a poor guide for another.
Attainable Accuracy: Out-of-fold $R^2$ was positively associated with $\rho$ across all domains. In the paired analysis, a larger gap in ceilings ( $\Delta\rho$ ) corresponded to a larger gap in $R^2$ , ranking stability (Kendall's $\tau$ ), and predictor list overlap (Jaccard). For the largest gap (Mathematics vs. Creative Thinking, $\Delta\rho \approx 0.088$ ), the difference in $R^2$ was highly significant ( $p < 10^{-10}$ ). At the smallest gap (Science vs. Reading, $\Delta\rho \approx 0.008$ ), differences were near zero and often non-significant.
Ranking Instability: In the least precisely measured domain (creative thinking), over 40% of a five-predictor list changed between two plausible values of the same data. This instability was confirmed to be attributable to measurement noise rather than the permutation estimator alone.
Split Inflation: Random student-level splits inflated performance in 604 of 606 analyses. The magnitude of inflation was governed by the number of sampled schools (Spearman $r \approx -0.4$ ), with extreme cases showing an inflation from $R^2=0.002$ (school-disjoint) to $0.361$ (random split).
Exploratory Finding: Measurement precision (noise-to-signal ratio) appeared to decline with the number of students assessed, particularly in domains requiring human coding of open-ended responses (Creative Thinking), though this relationship was driven by the contrast between the largest systems and typical ones.

Significance and Claims
The paper argues that predictive performance from ILSA databases is not comparable across domains or systems without reference to the measurement bound. A model achieving $R^2=0.45$ in a system with a ceiling of 0.65 has captured significantly more of the available signal than a model with the same $R^2$ in a system with a ceiling of 0.93.

The author claims that:

Predictor rankings from a single plausible value are merely one realization of a wide distribution. In low-precision domains, a single ranking is unstable, with a substantial portion of the top predictors changing between plausible values.
The "noise ceiling" is a governing property of the assessment design and operational execution, not a constant. It dictates both the maximum attainable accuracy and the stability of the explanatory variables reported.
Two opposing biases exist: Measurement noise deflates attainable performance, while conditioning the plausible values on background variables inflates apparent associations. Neither is currently acknowledged in applied literature.

Practical Recommendations
The author proposes two modest, low-cost adjustments for researchers:

Report the Ceiling: Alongside any predictive performance metric, report the measurement-precision ceiling ( $\rho$ ) for that specific domain and system. Performance should be expressed as a share of this bound ( $R^2/\rho$ ) to enable valid cross-system comparisons.
Report Stability Statistics: When reporting predictor rankings, researchers should fit models across all plausible values and report a stability statistic (e.g., rank correlation or Jaccard overlap) alongside the ranking. A single ranking from a single draw is insufficient to characterize the distribution of important predictors.

The paper concludes that the analytic capacity of large-scale assessment databases is not uniform; where measurement is less precise, less can be learned, and conclusions are less stable. Since precision is partly a design and operational property, it is a variable that assessment programs can potentially improve.

Technical Summary: Measurement-Bounded Predictive Inference in PISA 2022

More like this