📈 economics

Q-squared as a mapping model performance metric: comparison of models mapping to EQ-5D in gastrointestinal disease

This study evaluates the Q-squared statistic as a novel performance metric for mapping quality-of-life measures to EQ-5D in gastrointestinal diseases, finding that it effectively classifies predictive accuracy and reveals that internal validation models generally outperform external ones.

Original authors: Kristian Mallon, Rositsa Koleva-Kolarova, Richeal Burns, Yaling Yang, Helen Dakin

Published 2026-09-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kristian Mallon, Rositsa Koleva-Kolarova, Richeal Burns, Yaling Yang, Helen Dakin

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of medical research, doctors and economists constantly ask a simple but vital question: how good is a patient's life right now, and how will a new treatment change it? To answer this, researchers use surveys that ask people to rate their own health, covering things like mobility, pain, and mood. These surveys produce a single number, a score that represents a person's quality of life. This score is so useful that health agencies rely on it to decide which treatments deserve funding and support. However, a problem arises when a clinical study uses a specific survey designed for a particular disease, like stomach cancer or irritable bowel syndrome, but the funding agencies need the generic health score to make their decisions. Since the two surveys ask different questions, they cannot be directly compared. To bridge this gap, scientists use a method called "mapping." They build mathematical models that act as translators, taking the specific disease survey results and predicting what the generic health score would have been.

The challenge is knowing which translator is the best. For years, researchers have used standard tools to check if these models work well, but these tools often fail when comparing models built on different groups of people or different surveys. A new study by Kristian Mallon and his colleagues at Atlantic Technological University and the University of Oxford suggests a better way to judge these translators. They tested a metric called Q-squared, a measure of how much better a model is at predicting a patient's health score compared to simply guessing the average score for everyone. By applying this metric to a large collection of studies focused on gastrointestinal diseases, the team found that Q-squared provides a clear, fair way to see which models actually work and which ones are just guessing.

The researchers set out to examine fourteen studies that had already created these mapping models for people with digestive diseases, ranging from common issues like constipation to serious conditions like liver and stomach cancers. These studies involved data from nearly 15,000 individuals and included 118 different mathematical formulas designed to translate specific disease surveys into the standard health score. The team took the results reported in these studies and calculated the Q-squared value for each one. This calculation essentially measures the accuracy of the prediction by comparing the errors made by the model against the natural variation in the health scores of the patients. If a model performs no better than just guessing the average, the score is zero or negative; if it predicts perfectly, the score approaches one.

The results revealed a wide range of performance among the models. The Q-squared values for the 118 formulas ranged from a negative number, indicating a model that was worse than a simple guess, up to a high value of 0.91, which represents a very strong prediction. On average, about half of the models showed moderate predictive power, meaning they were reasonably accurate but not perfect. Only a small fraction, about 8 percent, achieved substantial accuracy. The study also highlighted that the type of disease and the specific surveys used mattered greatly. Models that translated a general cancer quality-of-life survey into the standard health score tended to perform better than those trying to translate surveys designed specifically for irritable bowel syndrome or constipation. In fact, some models for constipation performed so poorly that they were worse than simply assuming every patient had the average health score.

Another key finding was the difference between how models performed on the data they were built with versus how they performed on new, unseen data. The researchers found that models often looked much more accurate when tested on the same group of patients used to create them, compared to when they were tested on a completely different group of patients from another study. This is a common issue in statistics, where a model might memorize the details of one group rather than learning the general rules that apply to everyone. The study showed that while many models appeared strong in their initial tests, their accuracy dropped significantly when applied to new populations. This suggests that researchers need to be very careful about trusting a model just because it worked well in the original study.

The authors conclude that Q-squared is a valuable tool for sorting through the many available models. Unlike other measures that can be misleading when comparing different studies, this metric allows for a direct and fair comparison of accuracy across different groups and different types of surveys. It helps researchers and policymakers identify which models are reliable enough to use for making real-world decisions about healthcare funding. The study does not claim to have solved the problem of mapping once and for all, but it offers a clearer lens through which to view the existing tools. By using this metric, the medical community can avoid relying on models that perform poorly and instead select the ones that truly reflect the health and quality of life of patients with digestive diseases.

Technical Summary: Q-squared as a Mapping Model Performance Metric in Gastrointestinal Disease

Problem Statement
In health economics, mapping algorithms are frequently employed to predict EQ-5D utility values from disease-specific health-related quality of life (HRQoL) instruments when EQ-5D data is unavailable. The performance of these mapping models is traditionally assessed using metrics such as Root Mean Square Error (RMSE), Mean Squared Error (MSE), and Mean Absolute Error (MAE). However, these metrics are sensitive to the standard deviation (SD) of the sample, making direct comparisons of predictive accuracy across different studies or populations difficult. While $R^2$ is sometimes used, it measures correlation strength rather than prediction accuracy and fails to penalize biased predictions. Although the Q-squared ( $Q^2$ ) statistic—defined as the proportional reduction in error—has been proposed as a superior metric for assessing prediction accuracy across different samples, it has not previously been applied to evaluate mapping models in quality of life studies. This paper addresses the gap by investigating the utility of $Q^2$ as a performance metric specifically within the context of gastrointestinal (GI) disease mapping studies.

Methodology
The authors conducted a secondary analysis using the Oxford Health Economics Research Centre (HERC) Database of Mapping Studies (version 10.0).

Data Selection: The study identified 14 published studies (12 primary mapping studies and 2 external validation studies) focusing on GI disease populations (including colorectal, oesophago-gastric, and pancreatic cancers, as well as IBS and constipation). The studies were required to map various source instruments to EQ-5D-3L or EQ-5D-5L and report sufficient statistics (MSE or RMSE, and the SD of the target instrument) to allow for the calculation of $Q^2$ .
Calculation: $Q^2$ was calculated for 118 mapping functions using the formula: $Q^2 = 1 - (MSE / SD^2)$ . Where only RMSE was reported, it was squared to derive MSE.
Interpretation Framework: To categorize performance, the authors adopted threshold values commonly used for $R^2$ $R^{2}$ in methodological literature:
- Substantial: $\ge 0.75$
- Moderate: $0.50 - 0.74$
- Weak: $0.25 - 0.49$
- Poor: $0 - 0.24$
- Negative: $< 0$ (indicating the model performs worse than predicting the mean).
Analysis: The study explored variations in $Q^2$ across source-to-target instrument combinations, model types (e.g., OLS, Tobit, CLAD), sample sizes, and validation types (estimation, internal, and external).

Key Results

Overall Performance: Across 118 mapping functions involving 14,609 individuals, $Q^2$ $Q^{2}$ values ranged from -0.69 to 0.91 (median: 0.52).
- Substantial performance ( $\ge 0.75$ ) was observed in 8% (10/118) of algorithms.
- Moderate performance (0.50–0.74) was observed in 50% (59/118).
- Weak and Poor performance accounted for 24% and 16% of the algorithms, respectively.
- Negative performance ( $<0$ ) was found in two instances (an OLS model and a CLAD model), indicating severe bias.
Instrument Specificity:
- Models mapping the EORTC QLQ-C30 to EQ-5D generally demonstrated the strongest performance. For EORTC QLQ-C30 to EQ-5D-5L, 97% of functions showed moderate or substantial performance.
- Models mapping IBS-QOL and PAC-QOL showed significantly weaker performance. For IBS-QOL to EQ-5D-5L, 53% of functions were classified as poor. The lowest value (-0.69) occurred when mapping PAC-QOL to EQ-5D-3L in an opioid-induced constipation population.
Validation Differences: $Q^2$ values in internal validation samples were consistently higher than in external validation samples. 81% of internal validation functions showed moderate or substantial performance, compared to 62% of external validation functions showing weak performance.
Model Types: Ordinary Least Squares (OLS) was the most common model type. No clear pattern emerged regarding $Q^2$ superiority across different model specifications (e.g., OLS vs. CLAD vs. Mixture models).
Sample Size: Predictive performance was most variable in small samples ( $n \le 200$ ). The most consistent moderate-to-substantial performance was observed in the 501–800 sample size group.

Significance and Claims
The paper posits that $Q^2$ is an informative and practical metric for mapping studies between HRQoL measures. Its primary significance lies in its ability to:

Enable Cross-Study Comparison: Unlike RMSE or MSE, $Q^2$ accounts for sample variability (SD), allowing for the direct comparison of predictive accuracy across different samples and studies.
Facilitate Model Selection: It provides a standardized way to benchmark algorithms and select the most appropriate mapping model for a specific population or economic evaluation.
Support Model Averaging: The authors suggest $Q^2$ could be used to weight models in model averaging, accounting for variability and predictive power in validation samples.
Identify Poor Models: The metric effectively flags models with negative values, which perform worse than a simple mean prediction and should not be used.

The authors conclude that while the proposed thresholds for categorizing performance are pragmatic and based on existing $R^2$ literature rather than empirical validation for $Q^2$ , they offer a valuable framework for interpreting predictive power. They recommend that future mapping studies routinely report $Q^2$ alongside traditional error metrics to enhance transparency and comparability in the field. The study acknowledges limitations, noting that findings are specific to GI diseases and may not generalize to other conditions, and that the performance cut-offs require further validation.

Technical Summary: Q-squared as a Mapping Model Performance Metric in Gastrointestinal Disease

More like this