💻 computer science

Harmonized Feature Conditioning and Frequency-Prompt Personalization for Multi-Rater Medical Segmentation

This paper proposes a harmonized probabilistic framework that combines adaptive feature conditioning for artifact removal with frequency-domain prompts for rater-specific personalization, effectively disentangling acquisition noise from genuine clinical ambiguity to achieve state-of-the-art, well-calibrated multi-rater medical image segmentation.

Original authors: Sanaz Karimijafarbigloo, Armin Khosravi, Alireza Kheyrkhah, Reza Azad, Mauricio Reyes, Dorit Merhof

Published 2026-05-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sanaz Karimijafarbigloo, Armin Khosravi, Alireza Kheyrkhah, Reza Azad, Mauricio Reyes, Dorit Merhof

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to draw a map of a mysterious island based on the descriptions of four different explorers. Each explorer has seen the island through a slightly different pair of glasses, and they are using different drawing styles.

Explorer A draws the coastline very sharply and precisely.
Explorer B draws it a bit more loosely, focusing on the general shape.
Explorer C sees the island through a foggy window (maybe their camera lens was dirty).
Explorer D has a shaky hand, making their lines wobbly.

If you just ask them to "agree on one map" and average their drawings, you lose the unique insights of each explorer, and the foggy or shaky parts might ruin the whole picture. This is the problem doctors face when multiple experts try to outline tumors or organs in medical scans. They often disagree, and the machines themselves (scanners) can add "noise" or blur.

This paper introduces a new AI system called the Harmonizer Network that solves this by acting like a super-smart editor who knows how to separate "bad signal" from "good style."

Here is how it works, broken down into three simple steps:

1. The "Noise-Canceling Headphones" (The Harmonizer)

First, the AI puts on a pair of "noise-canceling headphones."

The Problem: Medical scanners (like CT or MRI machines) from different manufacturers or with different settings can make images look slightly different. Some look grainy, some look blurry. This is "scanner noise," not real anatomy.
The Solution: The AI has a special module called the Harmonizer. Think of it as a translator that listens to the "static" of the scanner and cancels it out. It smooths out the grainy or blurry parts so that the AI sees the true shape of the organ, regardless of which machine took the picture.
The Result: Now, the AI has a clean, stable "mental map" of the anatomy that isn't confused by bad equipment.

2. The "Style-Adjusting Filter" (Frequency-Prompt Personalization)

Once the image is clean, the AI needs to remember how each specific doctor likes to draw.

The Problem: Even with a clean image, Doctor A might draw a tumor's edge very sharply, while Doctor B might draw it softly.
The Solution: The paper uses a clever trick involving sound waves (frequencies). Imagine the image is a song. The "low notes" are the big shapes (the organ itself), and the "high notes" are the fine details (the edges and textures).
The AI uses a Frequency-Prompt Module. Think of this as a set of equalizer sliders for the image.
- If the doctor likes sharp edges, the AI turns up the "high notes" (high frequencies) to make the lines crisp.
- If the doctor prefers soft edges, it turns them down.
The Result: The AI can now generate a version of the map that looks exactly like it was drawn by Doctor A, or Doctor B, or Doctor C, without needing to retrain the whole system for each person. It captures their unique "handwriting."

3. The "Group Consensus Check" (GED Regularization)

Finally, the AI needs to make sure it's not just guessing randomly.

The Problem: If the doctors disagree wildly on a specific spot, the AI shouldn't be 100% sure it knows the answer. If they all agree, the AI should be confident.
The Solution: The system uses a mathematical rule called Generalized Energy Distance (GED). Think of this as a "group vibe check."
- If the doctors' drawings are spread out (disagreeing), the AI generates many different possible maps to show that uncertainty.
- If the doctors all drew the same thing, the AI generates one very confident map.
The Result: The AI knows when to say, "I'm not sure, look at these different possibilities," and when to say, "I'm confident, here is the answer."

Why This Matters (According to the Paper)

The authors tested this on two real-world datasets:

LIDC-IDRI: A collection of lung scans where radiologists marked lung nodules.
NPC-170: A collection of head and neck cancer scans.

The findings were:

Better Accuracy: The AI got better scores (Dice scores) than previous methods, meaning its maps matched the doctors' drawings more closely.
Handles Noise: When the researchers intentionally added "fog" (noise) or "blur" to the images, this AI didn't panic. It kept working well because of its "noise-canceling" step.
Smart Uncertainty: The AI correctly identified the "fuzzy" areas where doctors disagreed and showed multiple options, rather than forcing a single, potentially wrong answer.

In a Nutshell

This paper presents a tool that first cleans up the mess caused by different medical machines, then learns the unique drawing style of each doctor, and finally knows when to be confident and when to be unsure. It creates a system that respects the diversity of human experts while filtering out the technical errors of the machines.

Technical Summary: Harmonized Feature Conditioning and Frequency-Prompt Personalization for Multi-Rater Medical Segmentation

Problem Statement
Multi-rater medical image segmentation aims to capture the inherent ambiguity of clinical interpretation, where diagnostic boundaries vary across experts and imaging devices. Existing approaches often reduce this diversity to consensus labels (e.g., majority voting) or treat rater differences as noise, resulting in models that are overconfident and poorly calibrated. Furthermore, current deep learning frameworks operating primarily in the spatial domain are vulnerable to scanner-specific noise and acquisition artifacts. These perturbations often conflate clinically meaningful uncertainty with irrelevant variability, hindering cross-scanner generalization. Additionally, while some recent methods attempt personalized predictions, they often fail to explicitly disentangle noise effects from genuine diagnostic variability or rely on static fusion strategies that do not adapt to individual expert styles.

Methodology
The authors propose a unified probabilistic framework, the Harmonizer Network, designed to disentangle acquisition artifacts from genuine annotator variability. The architecture builds upon the Probabilistic U-Net backbone and introduces two cooperating modules within a single training pipeline:

Noise Harmonizer (Data-Level Standardization):
- Goal: To mitigate scanner- and acquisition-induced artifacts that distort latent features.
- Mechanism: This module implicitly models scanner-specific artifacts using a learnable bank of artifact tokens. It predicts spatially broadcasted modulation parameters ( $\gamma_l, \beta_l$ ) conditioned on decoder features and these tokens.
- Operation: It performs an affine transformation ( $\tilde{f}_l = \gamma_l \odot f_l + \beta_l$ ) on features across network layers. This acts as a data-driven normalization mechanism that dynamically suppresses intensity drifts, motion artifacts, and domain biases without requiring explicit knowledge of the noise distribution. Weight sharing across layers ensures consistent denoising behavior.
Frequency-Prompt Personalization (Rater-Level Adaptation):
- Goal: To represent rater-specific styles (e.g., boundary precision, texture sensitivity) which manifest as high-frequency cues.
- Mechanism: This module operates in the spectral domain. It decomposes features using a Discrete Wavelet Transform (DWT) to separate low-frequency structural outlines from high-frequency texture and edge information.
- Operation: A Rater-Aware Prompt Projection (RAPP) block encodes annotator-specific preferences into learnable tokens. These tokens are adaptively weighted and combined with high-frequency spectral features via an attention mechanism (Large Kernel Attention). The refined high-frequency features are then fused with low-frequency maps and transformed back via Inverse DWT (IDWT) to produce a rater-adaptive latent vector ( $z'$ ). This allows the model to synthesize expert-specific segmentations without retraining the backbone.
Diversity-Preserving Regularization:
- The framework employs a Generalized Energy Distance (GED)-based regularization. This loss aligns the model's generative distribution with the empirical annotation distribution. It minimizes the discrepancy between predicted samples and expert annotations (fidelity) while enforcing diversity among generated samples to prevent collapse to a single consensus mask.
Training Strategy:
- Phase 1: The backbone and Noise Harmonizer are trained jointly to learn artifact-invariant latent features. The rater-specific frequency adaptation head is excluded.
- Phase 2: The backbone and harmonizer are frozen. Only the Personalization Module is trained to align spectral features with rater-specific patterns.

Key Contributions

Disentanglement of Variability: The paper introduces a method to explicitly separate scanner-induced noise from genuine inter-rater variability, ensuring that uncertainty reflects anatomy rather than acquisition artifacts.
Frequency-Domain Personalization: Unlike previous spatial-domain approaches, this work introduces High-Frequency Prompt Modules that operate in the spectral domain to encode annotator-dependent boundary precision and textural sensitivity.
Lightweight Architecture: The framework adds minimal parameters (approx. 0.21M additional parameters to a 30M backbone) while achieving state-of-the-art performance, making it computationally efficient.
Calibrated Uncertainty: The model produces uncertainty estimates that rise in ambiguous regions (where experts disagree) and decline in consensus regions, supporting its use as an interpretable tool.

Experimental Results
The method was evaluated on the LIDC-IDRI (thoracic CT) and NPC-170 (nasopharyngeal carcinoma MRI) datasets, as well as the Kvasir dataset for noisy label robustness.

Distributional Fit: The Harmonizer Network achieved the lowest Generalized Energy Distance (GED) scores on both LIDC-IDRI (0.1048) and NPC-170 (0.1758), outperforming baselines like Probabilistic U-Net and D-Persona. This indicates a superior alignment with the empirical annotation distribution.
Personalized Accuracy: The model achieved the best personalized Dice scores (e.g., 92.65% $Dice_{max}$ on LIDC-IDRI), outperforming D-Persona by approximately 1.6 percentage points in mean Dice.
Robustness: Under synthetic noise (Gaussian noise, blur, gamma jitter) and real-world acquisition domain shifts (cross-scanner evaluation), the Harmonizer Network demonstrated superior stability, retaining over 95% of its clean-data performance under severe noise conditions where other methods degraded significantly.
Uncertainty Calibration: The model's pixel-wise uncertainty was shown to correlate strongly with prediction correctness and inter-rater disagreement, with low Expected Calibration Error (ECE) values across all raters.

Significance and Claims
The paper claims that the Harmonizer Network provides a lightweight yet effective solution for harmonized and personalized multi-rater segmentation. Its primary significance lies in its ability to:

Improve Reliability: By removing scanner artifacts from the latent space, the model ensures that uncertainty is driven by anatomical ambiguity rather than noise.
Enhance Interpretability: The frequency-domain personalization allows the model to replicate specific expert styles (e.g., sharp vs. smooth boundaries) without architectural redundancy, making it suitable for clinical workflows requiring multi-expert consensus or individualized second opinions.
Generalize: The framework demonstrates robustness to both synthetic perturbations and real-world domain shifts (different scanner manufacturers), addressing a critical gap in current multi-rater segmentation literature.

The authors conclude that their approach supports the use of probabilistic models as reliable tools for clinical decision-making by providing diverse, anatomically consistent, and rater-specific segmentations that respect the inherent uncertainty of medical imaging.