IHF-Harmony: Multi-Modality Magnetic Resonance Images Harmonization using Invertible Hierarchy Flow Model

IHF-Harmony is a novel unified framework that leverages an invertible hierarchy flow and artifact-aware normalization to achieve scalable, high-fidelity, and anatomy-preserving harmonization of multi-modality MRI data using unpaired datasets.

The Gist

Imagine you are a detective trying to solve a mystery using clues (MRI brain scans) collected from five different police stations. Each station uses a slightly different camera, lighting, and film type. Even though they are all photographing the same crime scene, the photos look different: some are too bright, some are too grainy, and the colors are off. If you try to compare them directly, you might think the differences in the photos mean the crime scene itself changed, when really, it's just the cameras.

This is the problem scientists face with MRI scans. Hospitals use different machines (GE, Siemens, Philips) with different settings. This creates "noise" or "artifacts" that hide the real biological differences in patients' brains.

The paper introduces a new tool called IHF-Harmony to fix this. Here is how it works, explained through simple analogies:

1. The Goal: The "Universal Translator"

Most old methods tried to fix these photos by either:

• The "One-Size-Fits-All" approach: Just turning up the brightness or contrast for everyone (Statistics-based). This is like putting a filter on a photo; it fixes the color but might blur the details.
• The "Traveling Student" approach: Asking a person to get scanned at every single hospital to see exactly how the machines differ. This is expensive and impossible for large studies.

IHF-Harmony is different. It's like a smart, universal translator that can look at a photo from any machine and translate it to look like it came from any other machine, without needing a "traveling student" to teach it. It works even if the photos aren't paired up (i.e., it doesn't need the same person scanned twice).

2. The Secret Sauce: The "Reversible Magic Box"

The core of this system is something called an Invertible Hierarchy Flow. Let's break that down:

• The Problem: When you edit a photo to change the lighting, you often accidentally erase the details of the object (like the shape of a nose or a tumor).
• The Solution: Imagine a magic box that takes a photo and separates it into two distinct piles:
  1. The "Anatomy" Pile: The actual shape of the brain, the wrinkles, the tumors. This is the truth.
  2. The "Artifacts" Pile: The graininess, the brightness, the scanner-specific noise. This is the noise.

The magic box is reversible. It can take the "Anatomy" pile and the "Artifacts" pile, mix them together, and perfectly reconstruct the original image. Because it never throws anything away, it guarantees that when we change the "Artifacts" to match a new machine, the "Anatomy" stays exactly the same. No details are lost.

3. The Process: How it Fixes the Image

The system works in three main steps, like a high-end photo editing studio:

• Step A: The Squeeze (Organizing the Clutter)
  First, the system takes the 3D brain scan and "squeezes" it. Think of this like folding a large map so it fits into a small envelope. It rearranges the data so the computer can process the details more efficiently.

• Step B: The Subtractive Coupling (Peeling the Onion)
  The system uses a process called Invertible Hierarchy Flow to peel away the "noise" layer by layer.

  • Imagine an onion. The outer layers are the scanner noise. The inner core is the brain structure.
  • The system mathematically subtracts the outer layers (the noise) to isolate the core (the brain).
  • Because it's "invertible," it remembers exactly how to put the layers back together later.

• Step C: The "Art-Aware" Makeover (The Stylist)
  Now, the system has the pure brain structure. It needs to dress it up in the style of the target machine (e.g., a Siemens scanner).

  • It looks at a sample image from the target machine to see what the "style" looks like (the lighting, the contrast).
  • It uses a special Art-Aware Normalization module. Think of this as a tailor. The tailor looks at the brain structure (the body) and the target style (the fabric).
  • Crucially, the tailor knows not to change the body shape. They only change the fabric texture and color. They apply the new "Siemens style" to the "Siemens-style fabric" while keeping the "Anatomy" body perfectly intact.

4. The Result: A Perfect Match

The system then reverses the process (un-folds the map, puts the layers back together). The result is a brain scan that:

1. Looks like it was taken on the target machine (so it matches the other data).
2. Has the exact same anatomical details as the original scan (so no medical information is lost).

Why is this a big deal?

• No Traveling Required: You don't need to fly patients to different hospitals to calibrate the machines. You can just use the software.
• Works on Many Types: It handles T1, T2, and Diffusion MRI (different ways of looking at the brain) all at once.
• Safe for Doctors: Because it guarantees "lossless reconstruction," doctors can trust that the shapes of tumors or brain folds haven't been accidentally smoothed over or distorted by the software.

In summary: IHF-Harmony is a smart, reversible editing tool that strips away the "camera noise" from brain scans and re-applies a consistent "studio lighting" style, ensuring that all the data from different hospitals looks like it came from the same place, without ever losing the precious medical details inside.

Technical Summary

1. Problem Statement

Multi-site Magnetic Resonance Imaging (MRI) studies are essential for achieving large sample sizes and population diversity. However, aggregating data from different scanners and protocols introduces site-specific, non-biological variability (artifacts) that confounds downstream analysis and obscures subtle biological effects.

Existing solutions face significant limitations:

• Prospective harmonization (standardizing acquisition protocols) is costly, requires pilot studies, and cannot be applied to historical data.
• Retrospective harmonization methods often rely on:
  • Statistics-based approaches: Limited in modeling spatially varying, anatomy-dependent artifacts.
  • Supervised learning: Requires paired "traveling subject" datasets (scans of the same person on different scanners), which are rarely available in large-scale studies.
  • Unsupervised deep learning: Often struggles with scalability across multiple modalities and fails to preserve fine-grained anatomical structures, leading to anatomical distortion.

2. Methodology: IHF-Harmony

The authors propose IHF-Harmony, a unified, retrospective harmonization framework designed for multi-modality MRI using unpaired data. The core philosophy is to decompose the translation process into reversible feature transformations to guarantee bijective mapping and lossless reconstruction.

The architecture consists of three main components:

A. Overall Architecture

The model employs a fixed VGG encoder for feature extraction and loss computation. It defines a forward pass ($E$) to map a source image $x$ to latent features $z$, and a backward pass ($E^{-1}$) to reconstruct the harmonized image $\hat{x}$. The process is conditioned on target artifact statistics extracted from a target image $y$.

B. Squeeze Operation

To facilitate efficient feature interaction between blocks, the model uses a "squeeze" operation. This reduces spatial resolution by splitting input features into non-overlapping patches and concatenating them along the channel dimension, compacting spatial information into channel representations.

C. Invertible Hierarchy Flow (IHF)

This is the core mechanism for preserving anatomy while removing artifacts. It operates via hierarchical channel-wise coupling:

• Forward Pass (Subtractive Coupling): The input image is processed through an affine network to generate affine components. The model iteratively subtracts these components from the feature maps ($h_i = h_{i-1} - a_i$). This progressively disentangles and removes artifact-related features while retaining anatomical information.
• Reverse Pass (Additive Coupling): To reconstruct the image, the model performs an additive coupling. It fuses the transformed latent features with the affine components in a recursive manner, using a learnable fusion weight ($\alpha$) to balance contributions. This ensures lossless reconstruction of the original anatomy.

D. Artefact-Aware Normalization (AAN)

To transfer target characteristics without distorting source anatomy, the model uses AAN:

• It extracts multi-scale statistics (mean $\mu$ and variance $\sigma$) from the target image using a pre-trained VGG encoder.
• It generates anatomy-guided affine parameters ($AGA_\mu, AGA_\sigma$) by jointly inferring from the source anatomy ($z$) and target artifacts ($z_s$).
• The source features are normalized and modulated using these parameters to align with the target domain while maintaining structural identity.

E. Loss Functions

The training is guided by two consistency losses to ensure fidelity:

1. Anatomical Consistency Loss ($L_{ac}$): Based on feature self-similarity (spatial correlation) using a pre-trained VGG network. It explicitly constrains the model to maintain structural consistency between the source and reconstructed image.
2. Artefact Consistency Loss ($L_{art}$): Addresses semantic misalignment in unpaired data by selectively matching feature distributions. It calculates channel-wise discrepancies between the harmonized image and the target, focusing on channels with the smallest discrepancies to align artifact statistics while ignoring anatomical variations.

3. Key Contributions

1. Unified Multi-Modality Framework: A single deep learning model capable of harmonizing multiple MRI modalities (T1, T2, Diffusion) using unpaired data, eliminating the need for traveling subject datasets.
2. Invertible Hierarchy Flow (IHF): Introduces a bijective formulation with subtractive coupling that guarantees accurate reconstruction and preserves fine-grained anatomical structures often lost in unsupervised methods.
3. Artefact-Aware Normalization (AAN): A novel module that applies anatomy-preserving transformations using learnable, anatomy-guided affine parameters, allowing images to adopt target artifact characteristics without losing structural identity.
4. Robust Consistency Objectives: A comprehensive loss formulation that explicitly enforces structural fidelity and realistic artifact translation across heterogeneous protocols.

4. Experimental Results

The method was evaluated on the ABCD study (multi-vendor T1, T2, DWI), SRPBS-TS (structural MRI), and HDD (diffusion MRI) datasets.

• Multi-Modality Performance: IHF-Harmony successfully harmonized T1, T2, and Diffusion-weighted images across GE, Philips, and Siemens scanners. Qualitative results showed effective contrast adaptation while preserving anatomical details.
• Structural MRI (SRPBS-TS): Compared against baselines (Histogram Matching, SSIMH, cGAN, SiMix), IHF-Harmony achieved superior performance in RMSE, MS-SSIM, LPIPS, and PSNR. It effectively aligned intensity distributions across nine source sites to a target domain while maintaining structural integrity.
• Diffusion MRI (HDD): The method demonstrated high generalizability in downstream tasks, preserving complex microstructural parameter maps (FA, NDI, ODI) and reducing scanner/channel-specific bias.
• Ablation Studies: Removing either the anatomical or artifact loss resulted in significant performance degradation, confirming that both objectives are necessary for high-fidelity harmonization.

5. Significance

IHF-Harmony represents a significant advancement in medical image analysis by solving the critical bottleneck of scalability and anatomical preservation in multi-site studies.

• Clinical Impact: It enables the integration of heterogeneous datasets from large-scale clinical studies without requiring expensive prospective standardization or rare traveling subject data.
• Technical Innovation: By leveraging invertible flows, it provides a mathematically guaranteed solution for lossless reconstruction, addressing the "black box" nature of generative models that often hallucinate or distort anatomy.
• Future Utility: The framework facilitates robust downstream analyses (e.g., biomarker discovery, disease classification) by ensuring that observed variations are biological rather than scanner-induced.

The code is scheduled for release upon acceptance, promoting reproducibility and further research in multi-modal medical imaging harmonization.