📊 statistics

Clustered Flexible Calibration Plots For Binary Outcomes Using Random Effects Modeling

This paper proposes and evaluates three random-effects modeling approaches for generating flexible calibration plots across multiple clusters, recommending a two-stage meta-analysis with splines for overall calibration and a mixed model for cluster-specific curves to effectively assess heterogeneity in clinical prediction models.

Original authors: Lasai Barreñada, Bavo D. C. Campo, Laure Wynants, Ben Van Calster

Published 2026-04-22

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Lasai Barreñada, Bavo D. C. Campo, Laure Wynants, Ben Van Calster

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

This paper introduces a new method for evaluating how accurately 'predictive models' used by physicians to diagnose patients perform across multiple hospitals or regions (clusters).

Let me explain this simply using an analogy.

🏥 Scenario: "A Nationwide Culinary Evaluation"

Imagine a famous chef claiming, "Using this ingredient results in a delicious dish 80% of the time."
We must test how well this chef's recipe actually works across 14 regions nationwide (hospitals), such as Seoul, Busan, and Jeju.

The Problem: Ingredient quality, kitchen environments, and chef skills vary by region. (This is the 'cluster' effect in data.)
The Flawed Traditional Approach: If we mix all national data together and evaluate only that "80% accuracy overall," we might mistakenly think "it's fine" by looking at the overall average, even if some regions achieve 90% success while others fail at 50%. This could lead to incorrect diagnoses for patients.

💡 Three New Methods Proposed by This Paper

The authors developed three new evaluation tools to solve this problem.

1. CG-C (Grouped Evaluation)

Analogy: Divide each region into 10 small groups, classify them as "Very Delicious," "Average," or "Not Delicious," and then calculate the average score for each group.
Advantage: Computation is relatively simple.
Disadvantage: Results can vary depending on how groups are divided, making it somewhat arbitrary.

2. 2MA-C (Two-Stage Meta-Analysis)

Analogy:
1. Stage 1: Evaluate cooking skills separately for each region (hospital). (e.g., Seoul gets Grade A, Busan gets Grade B)
2. Stage 2: Aggregate these individual evaluation results to set a range predicting the "overall average skill" and "whether similar performance can be expected in other new regions."
Feature: Most accurate when viewing the overall trend (average curve), and it provides a Prediction Interval indicating, "At this level, similar results should occur in other regions."

3. MIX-C (Mixed Model)

Analogy: Analyze all regional data at once, but allow the model to learn the "unique characteristics (random effects) of each region" itself. It is as if one super-chef experiences the tastes of regions nationwide and provides customized regional evaluations like, "Seoul is like this, Busan is like that."
Feature: Most accurate when evaluating the performance of specific regions (e.g., small hospitals). Even in regions with little data, it borrows information from other regions (Shrinkage) to provide more accurate evaluations.

🔬 Research Results: Which is Best?

The authors tested these methods using real ovarian cancer data and computer simulations.

When viewing the overall trend (average curve): 2MA-C (using splines) was best. It provides accurate overall predictions and clearly shows the confidence interval indicating, "At this level, similar results should occur in other regions."
When viewing specific regions (regional curves): MIX-C was best. Especially for small hospitals with fewer patients, MIX-C utilized information from other regions to deliver more accurate regional diagnoses.
Problem with existing methods: Ignoring clustering (regional differences) and analyzing mixed data can lead to underestimating or overestimating risks, potentially harming patients.

📝 Conclusion and Recommendations

This paper offers the following advice to physicians and researchers:

"When validating models across multiple hospitals nationwide, use 2MA-C to examine the overall average, and use MIX-C to accurately assess the performance of specific hospitals."

These methods are provided as code in the R programming language, making them ready for easy use by anyone. This will help medical staff deliver more accurate and reliable diagnoses to patients.

One-sentence summary:
"When evaluating a single recipe nationwide, regional differences must not be ignored; we have developed new evaluation tools that effectively capture both the overall average and regional characteristics simultaneously."

Technical Summary: Clustered Flexible Calibration Plots for Binary Outcomes Using Random Effects Modeling

Problem Statement
The evaluation of clinical prediction models (CPMs) increasingly relies on data from multiple clusters (e.g., medical centers or studies). Traditional calibration assessment methods often assume independence between observations, an assumption violated by clustered data where patients within the same center are correlated. Ignoring this clustering can lead to unreliable calibration assessments and potentially misguided clinical decisions. While summary statistics (e.g., Observed-to-Expected ratios) can be meta-analyzed, they are less informative than calibration plots. Furthermore, flexible calibration curves that explicitly account for clustering to quantify heterogeneity and provide prediction intervals for new clusters have been understudied.

Methodology
The authors propose three novel methodologies to construct flexible calibration plots for binary outcomes using random effects modeling. These methods aim to estimate an overall calibration curve (representing a cluster with an average effect) and cluster-specific curves, while providing confidence intervals (CI) and prediction intervals (PI).

Clustered Group Calibration (CG-C): An extension of traditional grouped calibration. Data within each cluster is divided into $Q$ quantiles (or equal intervals). A bivariate random effects meta-analysis is performed for each quantile, modeling the logit-transformed observed proportion and average estimated risk. This approach captures between-cluster heterogeneity via random intercepts.
Two-Stage Meta-Analysis Calibration (2MA-C): A two-step approach.
- Stage 1: A flexible calibration curve (using restricted cubic splines or LOESS) is fitted independently for each cluster.
- Stage 2: The predicted proportions from these cluster-specific curves are pooled across a fixed grid of risk values using a univariate random effects meta-analysis. This yields an overall curve with CIs and PIs.
Mixed Model Calibration (MIX-C): A one-stage approach using a logistic generalized linear mixed model (GLMM). The outcome is modeled as a function of the logit-transformed predicted probabilities using restricted cubic splines for both fixed and random effects. This allows for the simultaneous estimation of the overall curve and cluster-specific curves (via empirical Bayes/shrinkage) in a single step.

Key Contributions

Methodological Framework: Introduction of three distinct approaches (CG-C, 2MA-C, MIX-C) to handle clustering in calibration assessment, moving beyond simple pooling or summary statistics.
Prediction Intervals: All methods generate prediction intervals, representing the expected range of calibration curves in hypothetical new clusters, thereby quantifying heterogeneity.
Software Implementation: The authors provide ready-to-use R functions (to be integrated into the CalibrationCurves package) for implementing these methods.
Comprehensive Evaluation: The methods are evaluated through a real-world case study (ADNEX model for ovarian cancer), a simulation study with known ground truth, and a synthetic data study generated from real patient data to test performance under non-linear data generating mechanisms.

Results

Case Example: Applied to the ADNEX model (14 centers, N=2489), all methods indicated overestimation of malignancy risk, consistent with previous non-clustered analyses. However, the methods differed in their estimation of uncertainty and heterogeneity.
Simulation Study:
- Overall Curve Accuracy: MIX-C and 2MA-C (with splines) produced estimated curves closest to the true overall curve across various scenarios (varying intra-cluster correlation, AUC, events per cluster, and number of clusters). Standard flexible logistic calibration (ignoring clustering) performed considerably worse, especially with high clustering.
- Cluster-Specific Accuracy: MIX-C produced cluster-specific curves closest to the truth, particularly when sample sizes per cluster were limited, due to the benefits of shrinkage.
- Prediction Interval Coverage: Coverage was generally suboptimal across all methods and scenarios. However, 2MA-C with splines demonstrated the best coverage performance relative to others, though it still tended to underestimate heterogeneity (yielding intervals that were too narrow), particularly at the tails of the risk distribution.
Synthetic Data Study:
- When the "truth" was based on a logistic regression, MIX-C was the best performer.
- When the "truth" was based on a Random Forest (non-linear), MIX-C performed best with small validation samples (EPC < 500), while LOESS performed better with larger samples.
- Borrowing information across clusters (as done in MIX-C) improved accuracy for clusters with modest sample sizes.

Significance and Claims
The paper positions itself as a "phase 2 methodological study" aimed at presenting and assessing these methods in a limited number of settings. The authors claim that their methodologies address a critical gap in the external validation of CPMs by explicitly accounting for the clustered nature of data.

Key claims regarding significance include:

Refined Assessment: The proposed methods enable more refined and reliable calibration assessments by estimating overall curves and providing prediction intervals for hypothetical new clusters, directly supporting informed decision-making.
Differentiation of Calibration: The authors emphasize the importance of distinguishing between the calibration of the "average cluster" and specific clusters. They argue that stating a model is "well calibrated" based solely on the average curve is misleading if significant heterogeneity exists; the prediction intervals are crucial for understanding this variability.
Recommendations: Based on their findings, the authors recommend using 2MA-C with splines to estimate the overall calibration curve and its 95% prediction interval, and MIX-C for estimating cluster-specific curves, particularly when sample sizes per cluster are limited.
Limitations: The authors modestly acknowledge that prediction interval coverage was not perfect in all scenarios and that the methods rely on specific smoothing parameters or grouping strategies. They suggest future work could explore Bayesian approaches for deriving prediction intervals and testing the methods on more complex data generating mechanisms (e.g., random slopes, machine learning models).

The study concludes that neglecting clustering undermines calibration analysis and that the proposed random effects modeling approaches offer a necessary advancement for multicenter validations and meta-analyses of prediction models.