💻 computer science

When To Adapt? Adapting the Model or Data in Federated Medical Imaging

This paper systematically compares model personalization and data harmonization strategies in federated medical imaging, revealing that the optimal adaptation approach depends on the nature of domain heterogeneity, with harmonization excelling for appearance-based variations and personalization for structural differences.

Original authors: Chamani Shiranthika, Parvaneh Saeedi

Published 2026-05-05

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Chamani Shiranthika, Parvaneh Saeedi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a group of hospitals trying to build a single, super-smart AI doctor to help diagnose diseases. Because of privacy laws, they can't share their actual patient photos with each other. Instead, they use a technique called Federated Learning, where the AI travels to each hospital, learns from the local data, and brings back only its "lessons" (math updates) to improve the main model.

However, there's a big problem: every hospital is different. Some use different cameras, some have different lighting, and some see different types of patients. This is like trying to teach a student using textbooks written in different languages, with different fonts, and different pictures. The AI gets confused and performs poorly.

This paper asks a simple but crucial question: When the data is messy, should we fix the pictures (the data) or should we fix the student (the AI model)?

The authors tested two main strategies across six different medical tasks (like spotting polyps in the colon, skin lesions, or tuberculosis in chest X-rays) to see which approach works best.

The Two Strategies

Data Harmonization (Fixing the Pictures):
- The Analogy: Imagine you have a pile of photos taken in bright sunlight, dim rooms, and with old cameras. Before showing them to the student, you use a filter to make all the photos look like they were taken in the same studio with the same lighting. You are "harmonizing" the input so the student sees a consistent world.
- The Goal: Make the data look the same across all hospitals so one single AI model can learn from everyone.
Model Personalization (Fixing the Student):
- The Analogy: Instead of changing the photos, you give the student a special pair of glasses or a custom notebook for each hospital. The student learns to understand that "Hospital A's photos look a bit grainy" and "Hospital B's photos are very bright," and adjusts their thinking accordingly.
- The Goal: Let the AI adapt its internal brain to fit the specific quirks of each hospital's data.

The Big Discovery: It Depends on What is Different

The paper found that there is no "one size fits all" answer. The best strategy depends entirely on what kind of difference exists between the hospitals.

1. When the difference is "Style" (Appearance)

The Scenario: Imagine the hospitals are taking X-rays of the same type of lung disease. The lungs look the same, but one hospital's machine makes the images look very dark, while another makes them very bright. The structure of the disease hasn't changed, only the "style" or "look" of the image.
The Winner: Data Harmonization.
Why? It's easier to just fix the lighting on the photos than to teach the AI a whole new way of thinking. The paper found that for tasks like Tuberculosis classification, fixing the image style worked best.

2. When the difference is "Structure" (Content)

The Scenario: Imagine hospitals looking at colon polyps. In one hospital, the polyps might be small and round; in another, they might be large and flat. The shape and geometry of what they are looking for are fundamentally different.
The Winner: Model Personalization.
Why? You can't just "fix the lighting" to make a flat polyp look like a round one. The AI needs to learn that "in this hospital, polyps look like this, but in that hospital, they look like that." The paper found that for Colon Polyp segmentation, letting the AI adapt its brain was far superior.

3. When the difference is Small

The Scenario: If the hospitals are very similar (like Brain Tumor or Breast Tumor tasks in the study), the data doesn't vary much.
The Winner: It doesn't matter.
Why? Both strategies performed about the same. If the data is already consistent, you don't need to do much extra work.

The Takeaway

The authors conclude that you shouldn't just pick a strategy because it's popular. You have to look at your data first:

If your data looks different but means the same thing (different colors, lighting, noise), fix the data (Harmonization).
If your data looks different because the actual objects or shapes are different (different anatomy, different disease presentations), fix the model (Personalization).

The paper provides a "rule of thumb" for doctors and engineers building these systems: Don't guess. Check the type of variation in your data, and then choose the tool that fits that specific problem.

Technical Summary: "When to Adapt? Adapting the Model or Data in Federated Medical Imaging"

Problem Statement
Federated learning (FL) enables privacy-preserving collaborative model training across medical institutions but faces significant performance limitations due to inter-client heterogeneity. In medical imaging, data distributions are non-independent and identically distributed (non-IID) due to variations in scanner hardware, acquisition protocols, patient demographics, and annotation practices. This heterogeneity causes severe model bias and poor cross-site generalization.

Current research addresses this challenge through two distinct paradigms:

Model-side Personalization: Adapting model parameters to specific clients via local fine-tuning, modular layers, or meta-learning (e.g., FedPer, FedBN, pFedMe).
Data-side Harmonization: Reducing inter-client variation at the input level by aligning appearance statistics (e.g., scanner style, intensity) while preserving anatomical content, often using generative models (e.g., CycleGAN, CUT) or frequency-domain adaptation.

Despite the widespread use of both strategies, there is a lack of systematic comparison regarding when each approach is preferable. It remains unclear whether the choice of strategy should depend on the specific type of domain shift (structural vs. appearance-based) or the magnitude of heterogeneity.

Methodology
The authors propose a comprehensive empirical study to evaluate the trade-offs between harmonization and personalization across six diverse medical imaging tasks:

Segmentation Tasks: Colon polyp, skin lesion, and breast tumor segmentation.
Classification Tasks: Tuberculosis CXR, brain tumor, and breast tumor classification.

The study utilizes a unified experimental framework involving four clients per task, drawn from distinct public datasets (e.g., Kvasir-SEG, HAM10K, BUSI, Shenzhen Hospital CXR). The authors categorize domain shifts into two qualitative types:

$\Delta_{style}$ (Input-level/Style-level shift): Variations in scanner hardware, protocols, intensity, and noise that alter global appearance but preserve anatomical layout.
$\Delta_{content}$ (Content-level/Structural shift): Variations in anatomy, pathology morphology, viewpoint, and scene context that alter spatial structure and semantic content.

The authors evaluate 18 state-of-the-art (SoTA) methods under identical federated settings and communication budgets. These include:

Baselines: Locally centralized, centralized (globally/locally tested), and Vanilla FedAvg.
Harmonization Methods: Preprocessing/augmentation, histogram matching, frequency-domain adaptation (FDA), MixStyle (input and feature level), and generative translation (CycleGAN, CUT, AdaIN, CoMoGAN).
Personalization Methods: Layer-wise personalization (FedPer, FedRep, FedBN), optimization-based approaches (FedProx, SCAFFOLD, FedAdam), and bi-model/meta-learning (pFedMe, Ditto).

Key Contributions

Comprehensive Comparison: The paper provides the first systematic comparison of harmonization and personalization strategies across six medical imaging tasks with varying heterogeneity types.
Empirical Analysis of Domain Shift: It empirically demonstrates that the effectiveness of an adaptation strategy is conditional on the type and magnitude of the domain shift, rather than the task alone.
Modular Codebase: The authors release a modular codebase supporting both paradigms to facilitate future research and hybrid approaches.

Experimental Results
The results reveal a conditional trade-off driven by the nature of heterogeneity:

Structural Heterogeneity (Colon Polyp Segmentation): In tasks with large structural variations (e.g., different endoscopy datasets with varying polyp shapes and geometries), personalization methods consistently outperform harmonization. Adapting the model to client-specific structures is crucial here.
Appearance-Driven Heterogeneity (Tuberculosis CXR Classification): In tasks where anatomy is preserved but scanner/intensity variations are dominant, harmonization methods consistently outperform personalization. Reducing input variability allows a shared global model to perform effectively.
Moderate/Limited Variation (Skin Lesion, Breast Tumor, Brain Tumor): When inter-client differences are small or mixed (moderate structural and appearance shifts), both strategies perform similarly, with neither offering a clear advantage. In these cases, performance often approaches the "ceiling" of the centralized baseline.

Specifically, for colon polyp segmentation, personalization (e.g., Ditto, Local Finetuning) achieved superior Dice scores compared to harmonization. Conversely, for TB CXR classification, harmonization (e.g., Histogram matching, MixStyle) significantly outperformed personalization, which struggled with the high variance in input styles.

Significance and Claims
The paper concludes that there is no universally superior strategy for federated medical imaging. Instead, the choice between adapting the model or the data depends on the specific characteristics of the domain shift:

Adapt the Model (Personalization) when heterogeneity is primarily structural (e.g., varying anatomy or pathology morphology).
Adapt the Data (Harmonization) when heterogeneity is primarily appearance-based (e.g., scanner style, intensity, noise).
Either Strategy suffices when variation is limited or moderate.

The authors emphasize that practical federated systems should assess cross-site variation a priori to select the appropriate strategy, rather than assuming a fixed preference. They acknowledge a limitation: in real-world scenarios, the type and magnitude of heterogeneity are often unknown and dynamic. Consequently, they highlight future directions for developing metrics to characterize heterogeneity directly from data and creating adaptive frameworks that combine both harmonization and personalization at multiple levels.