⚡ electrical engineering

Why Variance Metrics Cannot Validate ComBat Harmonization: Evidence from Three-Vendor Mammography Radiomics

This study demonstrates that conventional variance-based metrics provide a tautological and misleading validation of ComBat harmonization, as they fail to detect persistent non-linear vendor-specific patterns and biological signal loss, thereby necessitating the use of independent non-linear separability tests to truly assess cross-vendor comparability.

Original authors: XU ZHANG, YUNJIANG LI, RUI FENG, QINQIN AI, YUHAI XIE, WENBIN NI

Published 2026-09-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: XU ZHANG, YUNJIANG LI, RUI FENG, QINQIN AI, YUHAI XIE, WENBIN NI

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where doctors could diagnose disease by translating the subtle textures of medical scans into thousands of precise numbers. This field, known as radiomics, turns images of the human body into data that computers can analyze to spot tumors or predict how a patient will respond to treatment. However, a major problem threatens this promise: the numbers change depending on the machine that took the picture. A scan from one manufacturer looks different from a scan of the same patient taken on a machine from a different company. These differences, caused by the equipment rather than the patient's health, create a barrier that prevents doctors from combining data from different hospitals. To fix this, researchers use a statistical tool called ComBat, which acts like a translator, adjusting the numbers so they look as if they all came from the same machine. For years, scientists have checked if this tool worked by measuring how much the numbers varied between groups. If the variation disappeared, they assumed the translation was successful. But a new study suggests that this way of checking is flawed, potentially giving a false sense of security while hiding the very differences that matter most.

In a study focused on breast cancer screening using full-field digital mammography, researchers set out to test whether this standard method of checking actually works. They gathered images from three different hospitals, each using a mammography machine from a different manufacturer. From these images, they carefully outlined the tumors and extracted 59 different texture features, which are the specific numbers used to describe the shape and pattern of the tissue. They then applied the ComBat tool to harmonize the data, hoping to remove the unique "fingerprints" left by each machine. Following the usual rules of the field, they first checked the results with standard statistical tests that measure variance, or how spread out the numbers are. As expected, these tests showed that the differences between the machines had vanished. The variation dropped to nearly zero, and the statistical tests declared the data successfully harmonized.

However, the researchers did not stop there. They suspected that while the simple spread of the numbers might have been fixed, the deeper, more complex relationships between the numbers might still carry the machine's signature. To test this, they used a different kind of check: they asked a computer to act as a detective and guess which machine took each scan based on the harmonized data. They used two types of detectives: one that looks for simple, straight-line patterns and another that can spot complex, curved patterns in the data. The simple detective failed, guessing no better than chance, which matched the results of the standard variance tests. But the complex detective, which could see the hidden structure in the data, performed perfectly. It identified the correct machine for every single scan with 100 percent accuracy. This revealed that while the tool had smoothed out the surface differences, it had left the deeper, non-linear fingerprints of the manufacturer completely intact.

The study also uncovered a more dangerous side effect of using the tool without extra care. In their data, the size of the tumors happened to be linked to which machine was used, a coincidence of how the patients were collected. When the researchers applied the standard version of the tool, it removed the machine differences but also accidentally erased the real biological differences in tumor size. The data no longer showed that tumors were different sizes across the groups, effectively deleting a meaningful medical fact. A more careful version of the tool, designed to protect biological information, kept the tumor sizes distinct, but the researchers noted that this approach was difficult to use in their specific setup because the data from one machine only included large tumors, making the groups impossible to balance perfectly.

The findings suggest that the current standard for proving that medical data is ready to be combined is insufficient. Relying only on tests that measure simple variation is like checking if a room is quiet by only listening for a shout; it misses the whispering that is still going on. The researchers argue that to truly know if data from different machines can be trusted together, scientists must use more rigorous tests that look for these hidden, complex patterns. They found that even after the standard tool was applied, the machine that took the picture could still be identified with perfect certainty, and real biological signals could be lost. This does not mean the tool is useless, but it means that declaring success based on simple statistics is premature. For radiomics to truly help patients across different hospitals, the methods used to combine the data must be judged by their ability to preserve the complex truth of the patient's biology, not just by their ability to make the numbers look the same on the surface.

Technical Summary: Why Variance Metrics Cannot Validate ComBat Harmonization

Problem Statement
In multi-center radiomics, particularly in Full-Field Digital Mammography (FFDM), features extracted from images acquired by different vendors often exhibit significant variability ("vendor effects") unrelated to patient biology. The empirical Bayes method, ComBat, is the standard approach for harmonizing these features by correcting location (mean) and scale (variance) differences. However, the validation of ComBat's success in the literature relies almost exclusively on first- and second-order variance statistics: the Intraclass Correlation Coefficient (ICC), eta-squared ( $\eta^2$ ), and the Kruskal-Wallis test.

The authors argue that this validation approach is circular (tautological). Since ComBat is mathematically constructed to align means and variances, reporting that these specific metrics improve after ComBat application merely restates the method's function rather than providing independent evidence that vendor information has been truly removed. Furthermore, there is a risk that "standard" ComBat application may inadvertently remove biologically meaningful variation that correlates with the batch variable (e.g., tumor size).

Methodology
The study utilized a retrospective cohort of 120 invasive breast cancer patients from three institutions, each equipped with a distinct mammography vendor: United Imaging (UIH), Hologic, and GE. After excluding cases with near-whole-breast masks, a clean subset of 91 cases (40 UIH, 40 Hologic, 11 GE) was analyzed.

Feature Extraction: 102 IBSI-compliant radiomic features (shape, first-order, and texture) were extracted from expert-delineated tumor ROIs. After filtering near-zero variance features, 59 features remained for analysis.
Harmonization: ComBat (both standard and covariate-protected versions) and per-vendor Z-score normalization were applied to correct for vendor effects.
Validation Framework: The authors contrasted conventional variance metrics with five independent, non-circular evaluation lines (E1–E5) that do not target the same moments ComBat aligns:
- E1 (Non-linear Separability): Training linear (Logistic Regression) and non-linear (Random Forest) classifiers to predict the vendor based on features. A successful harmonization should render the vendor unidentifiable (AUC $\approx$ 0.5) by any classifier.
- E2 (PCA Geometry): Assessing vendor clustering via silhouette coefficients in principal component space.
- E3 (Biological Preservation): Testing if a batch-correlated biological covariate (tumor pixel surface area) is preserved or erased.
- E4 (Downstream Transfer): Evaluating the ability of texture features to predict tumor size across vendors using Ridge regression.
- E5 (Higher-Order Structure): Measuring the persistence of inter-vendor feature-covariance structures (correlation matrices).

Key Results

Circularity of Conventional Metrics: As expected by construction, ComBat reduced median $\eta^2$ to 0.000 and median ICC to 0.000, and significantly reduced the number of features with significant inter-vendor differences (Kruskal-Wallis). The authors note these results confirm ComBat aligns moments but do not prove vendor information is gone.
Persistence of Vendor Information (E1 & E5): Despite the reduction in linear variance, a non-linear Random Forest classifier could identify the vendor with perfect accuracy (AUC 1.00, Accuracy 0.99) after standard ComBat harmonization. In contrast, the linear classifier's performance dropped to chance levels (AUC 0.56). This indicates that while ComBat removes linear location/scale effects, it leaves higher-order covariance structures intact, which non-linear models exploit.
Loss of Biological Signal (E3): Standard ComBat erased a significant inter-vendor difference in tumor size (a biological covariate correlated with the batch), reducing the Kruskal-Wallis p-value from $4.0 \times 10^{-8}$ to 0.10. Covariate-protected ComBat preserved this difference, but the authors note that in this specific cohort, structural zeros (GE only having large tumors) made covariate-protected ComBat statistically unstable.
Downstream Transfer (E4): While harmonization improved the cross-vendor transfer of texture features to predict tumor size, the authors attribute the gains of Z-score normalization partly to the removal of the biological variation (tumor size) that the model was intended to predict, rather than true harmonization.

Significance and Claims
The paper does not claim that ComBat is useless, but rather that its standard validation metrics are insufficient and misleading. The authors assert that:

Validation Must Be Independent: Harmonization adequacy cannot be certified by variance statistics ( $\eta^2$ , ICC) that ComBat is designed to alter. Instead, success must be judged by independent non-linear separability tests (e.g., leakage-free Random Forest classifiers) and covariance structure diagnostics.
Risk of Biological Erasure: Standard application of ComBat can remove batch-correlated biological signals, potentially compromising prognostic models that rely on such variations.
Design Implications: Covariate-protected ComBat is theoretically superior but requires careful study design to ensure overlap between batch variables and biological covariates; otherwise, the method becomes unstable.

The study concludes that in FFDM radiomics, vendor information persists in higher-order feature distributions even after ComBat harmonization. Therefore, the field must adopt non-linear, independent validation metrics to avoid false assurance of cross-vendor comparability.

Technical Summary: Why Variance Metrics Cannot Validate ComBat Harmonization

More like this