🧠 neuroscience

Imputation-Based Harmonization Mitigates Site Effects Without Data Leakage in Machine Learning Studies

The paper introduces MIRTH, a novel imputation-based harmonization method that effectively mitigates site effects in neuroimaging machine learning studies without causing data leakage or attenuating true biological signals.

Original authors: Hillman, N., Chen, A., Hu, F., Vandekar, S., Melhem, R., Beason-Held, L., Satterthwaite, T., Davatzikos, C., Shou, H., Shinohara, R.

Published 2026-10-05

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Hillman, N., Chen, A., Hu, F., Vandekar, S., Melhem, R., Beason-Held, L., Satterthwaite, T., Davatzikos, C., Shou, H., Shinohara, R.

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

Imagine trying to understand the human brain by looking at thousands of MRI scans taken in different hospitals across the country. Each hospital uses its own machines, often from different manufacturers, and follows slightly different scanning routines. Even when scientists try to standardize these procedures, the images still carry subtle fingerprints of where they were taken. A scan from one hospital might look slightly brighter or have different textures than a scan from another, not because the patient's brain is different, but because of the equipment. When researchers combine these images to study diseases like Alzheimer's, these technical differences can trick computer programs into thinking they have found a pattern that is actually just a reflection of the hospital's location. This problem, known as a "site effect," threatens the reliability of medical research, potentially leading to false conclusions about how the brain works or how a disease progresses.

To solve this, scientists have developed mathematical tools to smooth out these differences, effectively making all the scans look as if they came from the same machine. However, a new study by Noah Hillman and his colleagues reveals a hidden trap in how these tools are currently used. When researchers try to build computer models to predict a patient's condition, they often face a difficult choice: if they include the patient's diagnosis in the smoothing process, they risk introducing bias by letting the answer influence the preparation of the data; if they leave the diagnosis out, they might accidentally erase the very biological signals they are trying to find. The researchers propose a new method called MIRTH that navigates this dilemma. By using a technique that fills in missing information with multiple plausible guesses, they can clean the data without peeking at the answers, ensuring that the final predictions are based on real biology rather than technical artifacts.

The researchers tested their idea using data from two major studies: the Alzheimer's Disease Neuroimaging Initiative and the Baltimore Longitudinal Study of Aging. They gathered brain scans from over 2,000 participants, including both healthy individuals and those with Alzheimer's disease. These scans were taken on machines from three different manufacturers—GE, Philips, and Siemens—at two different power levels. The team focused on measuring the volume of 145 specific regions within the brain, creating a detailed map of brain structure. They then set up a series of challenges to see if their new method could accurately predict a person's age, sex, or diagnosis without being misled by the hospital where the scan was taken.

In their experiments, the team compared their new approach against several existing methods. One common approach involved smoothing the data without telling the computer what the patient's diagnosis was. The results showed that this method often weakened the connection between the brain scans and the disease, making the computer less accurate at predicting who had Alzheimer's. Another approach tried to use the diagnosis during the smoothing process, but this created a form of data leakage where the computer essentially memorized the answer before it was even supposed to be tested, leading to overly optimistic and unreliable results. A third method tried to guess the diagnosis by creating fake scenarios, but this struggled when the data was complex or when information was missing.

The new MIRTH method worked differently. It started by training the computer on a set of data where the diagnosis was known, teaching it how to remove the technical differences between hospitals. Then, when it encountered new patients whose diagnoses were unknown, the method used a statistical process to generate several possible versions of the missing information. For each of these versions, it applied the smoothing rules learned from the training data, ensuring that the technical noise was removed without ever revealing the true answer to the computer. Finally, it combined the results from all these versions to make a single, robust prediction.

The results were striking. In simulations where the computer was supposed to find no link between brain structure and disease, the old methods sometimes found a fake link simply because the disease was more common in certain hospitals. The new method, however, correctly found no link, showing it could distinguish between real biology and technical noise. When the researchers tested the method on real data to predict Alzheimer's, age, and sex, it performed comparably to the best possible scenario where the computer was allowed to use the answer in advance, with only slight differences in specific metrics like error rates. Crucially, it did this without using the answer. The predictions remained accurate even when the data was messy or when some information was missing, a situation that often stumbles other methods.

The study also looked closely at whether the method truly removed the influence of the hospital. When they asked the computer to guess which hospital a scan came from after the smoothing process, it performed significantly worse than before, though the results were still slightly above random guessing. This confirmed that the technical differences had been largely erased, though not perfectly. In contrast, when they used the older methods that did not include the diagnosis, the computer could still easily tell which hospital a scan came from, meaning the technical noise was still there, potentially skewing the medical results. The researchers found that this was particularly important when the disease was not evenly distributed across hospitals; in those cases, ignoring the diagnosis during the cleaning process led to significantly worse predictions.

While the new method proved highly effective, the researchers noted that it is not a perfect fix for every possible problem. For instance, if a new hospital appears that was never seen before, the method might need extra steps to adjust. Additionally, the study relied on simulations and existing data, so further work is needed to see how it performs in live clinical settings. Nevertheless, the findings offer a clear path forward for researchers who want to combine data from many sources. By using a process that fills in the blanks with multiple possibilities, scientists can now clean their data to remove technical errors without accidentally hiding the biological truths they are trying to discover. This ensures that when a computer model predicts a disease, it is doing so because it has learned from the brain itself, not because it has learned from the machine that took the picture.

Technical Summary: Imputation-Based Harmonization Mitigates Site Effects Without Data Leakage in Machine Learning Studies

Problem Statement
Neuroimaging studies increasingly rely on multi-site cohorts to increase sample sizes and reproducibility. However, pooling data across sites introduces "site effects" (or scanner effects)—systematic variability in imaging measures arising from technical heterogeneity (e.g., scanner hardware, acquisition parameters) rather than biological signal. While methods like ComBat have been developed to remove these effects, integrating them into machine-learning (ML) pipelines presents three critical challenges:

Data Leakage: Harmonizing the entire dataset prior to train-test splitting allows the harmonization model to "see" the test data, inflating performance estimates.
Outcome Imbalance: In predictive modeling, the outcome variable is often unavailable for the test set in real-world applications. If the outcome is imbalanced across sites (e.g., different prevalence of a disease at different sites), excluding the outcome from the harmonization model can attenuate true imaging-outcome associations. Conversely, including the true outcome in the harmonization model for the test set causes data leakage.
Limitations of Existing Solutions: Previous attempts to resolve this, such as using surrogate outcome labels (e.g., PrettYharmonize), often involve discretization that limits flexibility or fail to accommodate missing data in covariates.

Methodology: MIRTH
The authors propose MIRTH (Multiple Imputation for Removing Technical Heterogeneity), a framework designed to integrate ComBat into ML pipelines while preventing data leakage and preserving biological associations. The procedure operates as follows:

Training Phase:
- The ComBat harmonization model is fitted exclusively on the training data ( $D_{train}$ ), including the outcome variable ( $y$ ) and covariates ( $z$ ) to preserve their associations with imaging features ( $x$ ).
- A prediction model ( $\hat{f}$ ) is trained on the harmonized training data ( $D^*_{train}$ ). Hyperparameters are tuned via cross-validation.
Test Phase (Handling Missing Outcomes):
- Since the outcome is unavailable for the test set ( $D_{test}$ ) in real-world scenarios, MIRTH treats these values as missing.
- Multiple Imputation: Using Multiple Imputation by Chained Equations (MICE), the authors generate $M$ complete versions of the test dataset. The imputation model is fitted on the unharmonized training data to generate plausible values for the missing outcomes (and potentially missing covariates) in the test set.
- Harmonization: For each of the $M$ imputed test datasets, the ComBat harmonization operator (using parameters estimated from the training set in Step 1) is applied. This removes site effects without leaking information from the true test outcomes.
- Prediction & Pooling: The trained prediction model $\hat{f}$ is applied to each of the $M$ harmonized test datasets to generate $M$ predictions per subject. These predictions are pooled (averaged for continuous outcomes or majority voting/probability averaging for categorical outcomes) to produce the final prediction.

Key Contributions

Novel Integration: MIRTH provides a formal procedure to integrate ComBat into ML workflows that strictly adheres to the principle of separating training and test data processing.
Handling Imbalanced Outcomes: By using imputed outcomes to harmonize the test set, MIRTH preserves imaging-outcome associations even when the outcome is strongly imbalanced across sites, a scenario where excluding the outcome leads to biased attenuation.
Robustness to Missing Data: Unlike previous methods that only address missing outcomes, MIRTH extends to settings where both outcomes and covariates contain missing values in the test set.
Benchmarking: The method is rigorously evaluated against four baselines: No Harmonization, Oracle (using true test outcomes, representing an upper bound with leakage), No Outcome (excluding outcome from harmonization), and PrettYharmonize.

Results
The authors evaluated MIRTH using both simulated data and real-world volumetric MRI data from the Alzheimer's Disease Neuroimaging Initiative (ADNI) and the Baltimore Longitudinal Study of Aging (BLSA).

Simulation Studies:
- Null Settings (No True Association): In scenarios where imaging features were not predictive of the outcome but the outcome was correlated with site, "No Harmonization" and PrettYharmonize (for continuous outcomes) produced inflated performance (AUC > 0.5, RMSE < 1), indicating they failed to remove site effects. MIRTH and Oracle performed at chance levels, confirming successful removal of site-driven bias.
- Alternative Settings (True Association): When true imaging-outcome associations existed, "No Outcome" consistently yielded the worst predictive performance due to the attenuation of biological signals. MIRTH performed comparably to the Oracle (which uses true outcomes), achieving high predictive accuracy (AUC/RMSE) without generating predictions correlated with site.
- Missing Data: MIRTH demonstrated negligible performance differences between settings with missing outcomes only versus missing outcomes and covariates, validating its utility for complex missingness patterns.
Real-World Data (ADNI + BLSA):
- AD Diagnosis (Binary): MIRTH achieved performance comparable to Oracle (AUC ~0.95–0.96) and significantly outperformed the "No Outcome" approach (AUC ~0.91). Crucially, while "No Harmonization" achieved high AUC, it retained strong site effects (AUC for predicting site ~0.83). MIRTH reduced site predictability to near-chance levels (AUC ~0.62) while maintaining high diagnostic accuracy.
- Age (Continuous) and Sex (Binary): MIRTH matched Oracle performance for age prediction (RMSE ~7.06 vs. Oracle ~7.12) and outperformed "No Outcome" (RMSE ~7.37). For sex prediction, where outcomes were balanced across sites, all harmonization methods performed similarly, consistent with the finding that outcome imbalance is the primary driver of bias.

Significance and Claims
The paper claims that MIRTH offers a robust solution for multivariate prediction in neuroimaging by resolving the trade-off between data leakage and the preservation of biological associations. Specifically:

It prevents the inflation of predictive performance caused by site effects when outcomes are imbalanced across sites.
It avoids the attenuation of true biological signals that occurs when outcomes are excluded from the harmonization model.
It generalizes to complex missing data scenarios involving both outcomes and covariates.
The authors note that while MIRTH significantly reduces site effects, residual site predictability (AUC ~0.62) remains in real data, likely due to violations of ComBat assumptions regarding disease effect heterogeneity across sites. They suggest this is a limitation of the underlying harmonization model rather than the MIRTH framework itself.

The study concludes that MIRTH should be strongly considered for predictive modeling in multi-site neuroimaging, particularly when outcome distributions vary across clinical sites.

More like this