📊 statistics

A General Framework for Stability Assessment of Non-Deterministic Prediction Models

This paper proposes a general framework for quantifying the stability of non-deterministic prediction models across metric, categorical, and ordinal outcomes by introducing data-based and model-based stability measures that overcome the limitations of existing methods, such as their inability to handle individual test objects and their sensitivity to data composition.

Original authors: Thomas Martin Lange, Armin Otto Schmitt, Felix Heinrich

Published 2026-07-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Thomas Martin Lange, Armin Otto Schmitt, Felix Heinrich

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to predict the weather. You ask a single, very smart meteorologist for a forecast, and they say, "It will rain tomorrow." But then, you ask them to run the exact same calculation again, using the exact same data, and this time they say, "It will be sunny." If you ask a third time, they might say, "Maybe a little drizzle." This isn't because the meteorologist is confused; it's because their brain works a bit like a slot machine. Every time they pull the lever (run the model), a tiny, random spin happens that changes the result just a little bit. In the world of computer science, this is called a "non-deterministic" model. It's a fancy way of saying the computer uses a little bit of randomness to make its decisions, which is actually a good thing for making smart guesses, but it creates a headache: if the answer changes every time you ask, how can you trust it?

This is the puzzle that researchers Thomas Martin Lange, Armin Otto Schmitt, and Felix Heinrich are tackling. They are working in the field of predictive modeling, where computers learn from past data to guess future outcomes—like how much corn a farm will grow, how sick a patient might get, or what the stock market will do. The big problem they are solving is "stability." If you build a prediction machine that gives you a different answer every time you turn it on, is it reliable? The old ways of checking this reliability were like trying to measure the stability of a single coin flip by looking at a whole bag of coins. They worked okay for some situations, but they broke down when you only had one thing to predict (like a single future event) or when the answers were tricky, like ranking things from "very early" to "very late."

The authors of this paper have built a new, more flexible ruler to measure how steady these wobbly prediction machines really are. They didn't just find a better ruler; they invented two new ways to hold it. The first way, which they call "data-based" stability, asks: "How much does this model's answer wiggle compared to the wild variety of answers we saw in the training data?" It's like checking if a dart player is throwing wildly compared to the whole range of the dartboard. The second way, "model-based" stability, asks a deeper question: "How much does the model's own internal logic wiggle?" This is like checking if the dart player is throwing wildly compared to their own personal average, regardless of how big the dartboard is.

The researchers tested these new rulers using a digital playground where they simulated thousands of scenarios. They created fake data for things like crop yields and disease scores, then ran their prediction models over and over again, changing the number of "decision trees" (the brain cells of the model) from a few hundred to ten thousand. They found that their new rulers worked beautifully. Unlike the old rulers, which got confused and gave weird results when the test data was unbalanced (like having way more "sunny" days than "rainy" days) or when there was only a single object to predict, the new rulers stayed steady. They showed a clear, smooth curve: as you added more "brain cells" to the model, the predictions got more stable, just like adding more people to a choir makes the singing sound more consistent.

However, the paper also warns us that not all rulers are created equal. In their simulations, they discovered that the old methods sometimes lied. For instance, if a model predicted the same thing for everyone (a common problem when data is unbalanced), the old ruler would say the model was perfectly stable, even if it was just being lazy. The new rulers caught this, showing that the model was actually stable because it was consistent, but they did it in a way that made sense mathematically. They also found that when you have a tiny test group (like only two or three objects), the old ruler went crazy, jumping up and down wildly. The new rulers, especially the "model-based" one, stayed calm and reliable, even with very small groups.

The authors didn't just stop at theory; they built a tool called the "APS" package for the R programming language, which anyone can use to check their own models. They suggest that while we usually only care if a model is accurate (did it get the right answer?), we should also care if it is stable (does it give the same right answer every time?). Their work suggests that by using these new measures, we can find the "sweet spot" for how complex a model needs to be—adding enough complexity to be smart, but not so much that it becomes a jittery mess. It's a reminder that in the world of AI, consistency is just as important as correctness, and now, we finally have a better way to measure it.

Technical Summary: A General Framework for Stability Assessment of Non-Deterministic Prediction Models

Problem Statement
Modern prediction models, including Random Forests, Artificial Neural Networks, and Bayesian models, are inherently non-deterministic. Stochastic elements in their fitting procedures (e.g., bootstrap sampling, random weight initialization, or Markov Chain Monte Carlo sampling) mean that repeated training on identical data can yield different models and predictions. This variability can lead to inconsistent decisions, yet existing methods for quantifying this "prediction stability" are insufficient.

Current approaches rely on the Intraclass Correlation Coefficient (ICC) for metric predictions and Fleiss' $\kappa$ for categorical predictions. These methods suffer from three critical limitations:

Inapplicability to Single Objects: Both ICC and Fleiss' $\kappa$ require multiple objects in a test set to estimate variance components or expected chance agreement. They cannot quantify stability for single-object forecasting tasks (e.g., predicting a single future GDP or epidemic rate).
Restricted Prediction Types: They do not naturally extend to ordinal predictions, which possess a natural ordering but are neither purely nominal nor metric.
Sensitivity to Data Structure: These measures are heavily influenced by the composition of the test data. Fleiss' $\kappa$ is susceptible to the "prevalence problem" (the $\kappa$ paradox), where imbalanced class distributions artificially lower stability scores even when predictions are consistent. Similarly, ICC estimates are unreliable with small or homogeneous test sets.

Methodology
The authors propose a general framework for quantifying prediction stability based on Krippendorff's $\alpha$ , adapted to distinguish between observed instability and expected instability. The core definition of prediction stability ( $S$ ) is:
$S = 1 - \frac{D_o}{D_e}$
where $D_o$ is the observed disagreement (instability) across repeated model fits, and $D_e$ is the expected disagreement under a reference model of instability.

The framework introduces two distinct definitions of expected instability, leading to two complementary stability measures:

Data-Based Prediction Stability ( $\phi$ ):
- Concept: Assumes the maximal instability is bounded by the distribution of the response variable in the training data.
- Calculation: $D_e$ is derived from the empirical distribution of training responses. For categorical data, it uses category proportions; for ordinal data, it uses an ordinal distance function weighted by category frequencies; for metric data, it uses the variance of training responses.
- Applicability: Suitable when the model's predictions are constrained by the observed training range (e.g., standard Random Forest regression).
Model-Based Prediction Stability ( $\psi$ ):
- Concept: Assumes the maximal instability is determined by the distribution of predictions the model itself can generate.
- Calculation: $D_e$ is derived from an empirical approximation of the model's predictive distribution. This is achieved using out-of-sample predictions (e.g., Out-of-Bag predictions for Random Forests or K-fold cross-validation) to avoid optimistic bias.
- Applicability: Necessary when the model's predictive behavior restricts the range of predictions compared to the raw training data (e.g., Random Forests averaging terminal nodes).

Key Contributions

Generalization to Single Objects: Unlike ICC and Fleiss' $\kappa$ , the proposed measures ( $\phi$ and $\psi$ ) are defined at the object level, allowing stability assessment for a single test object.
Extension to Ordinal Data: The framework incorporates ordinal distance functions, enabling stability assessment for ordered categorical responses.
Robustness to Data Structure: By decoupling the expected instability from the specific test data composition (using training data or model behavior as the reference), the measures are less sensitive to class imbalance and small sample sizes.
Implementation: The framework is implemented in the R package APS (version 1.0.1), available on CRAN.

Results
Simulation studies using Random Forests (with varying numbers of trees) and three response types (metric, integer-valued/ordinal, and categorical) yielded the following:

Non-Linear Relationship: Both $\phi$ and $\psi$ exhibit a non-linear relationship with the number of learners (trees), which can be accurately modeled using a two-parameter logistic (2PL) model. This allows for the optimization of the number of trees to achieve a desired stability threshold.
Prevalence Problem: In scenarios with highly imbalanced class distributions:
- Fleiss' $\kappa$ failed to reflect stability, often remaining near zero or becoming undefined when all predictions converged to a single class.
- $\phi$ and $\psi$ correctly identified high stability (approaching 1) when predictions were consistent, even under extreme prevalence.
Small Sample Sizes: When tested on small subsamples of test data (2–30 objects):
- The ICC showed high variability and downward bias with fewer than 15 objects.
- $\psi$ (model-based) and $\phi$ (data-based) maintained stable mean values and significantly reduced variance (e.g., $\psi$ showed a seven-fold reduction in standard deviation compared to ICC with 2 objects).
Optimization Differences: While all measures followed a 2PL trend, they suggested different optimal numbers of trees. For instance, in the prevalence scenario, $\phi$ suggested ~3,000 trees, while Fleiss' $\kappa$ suggested ~68,000 (or was undefined), highlighting that the choice of measure influences optimization outcomes.

Significance and Claims
The paper claims to provide a "principled approach" to quantifying the reproducibility of non-deterministic prediction models, an aspect often overlooked in favor of prediction accuracy. The authors emphasize that:

Stability assessment is critical for decision-making reliability, particularly in single-object forecasting scenarios where traditional metrics fail.
The proposed measures mitigate the "prevalence problem" and the sensitivity to test set size that plague existing agreement coefficients.
The distinction between data-based and model-based stability allows researchers to choose a reference distribution that aligns with their specific modeling context (inherent data variability vs. model-induced variability).
The framework is model-agnostic and applicable to any non-deterministic model, though the simulations focused on Random Forests.

The authors conclude that evaluating non-deterministic models should not rely solely on accuracy but must include a formal assessment of prediction stability to ensure decision consistency.

Technical Summary: A General Framework for Stability Assessment of Non-Deterministic Prediction Models

More like this