🤖 AI

Mutual Distillation of Dual-Foundation Models for Semi-Supervised PET/CT Segmentation

This paper proposes MuDuo, a mutual distillation framework that leverages structural (SAM-Med3D) and functional (SegAnyPET) foundation models to guide a lightweight student network for state-of-the-art semi-supervised PET/CT organ segmentation using only five labeled cases.

Original authors: Fuyou Mao, Beining Wu, Yanfeng Jiang, Bohan Xu, Lixin Lin, Naye Ji, Hao Zhang, Yan Tang

Published 2026-06-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Fuyou Mao, Beining Wu, Yanfeng Jiang, Bohan Xu, Lixin Lin, Naye Ji, Hao Zhang, Yan Tang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Expensive Map" Dilemma

Imagine you are trying to draw a detailed map of a city (the human body) using two different types of cameras:

The CT Camera: Takes a clear, sharp photo of the buildings and streets (anatomy/structure).
The PET Camera: Takes a glowing photo showing where the traffic is heaviest (metabolism/energy).

To train a computer to draw this map automatically, you usually need a human expert to trace every single building and street on thousands of these photos. This is like hiring a cartographer to hand-draw 1,000 maps. It takes forever, costs a fortune, and requires two different types of experts (one for structure, one for traffic) to agree on the lines.

Because getting these "hand-drawn maps" (labeled data) is so hard, researchers usually only have a tiny handful of them (maybe just 5 or 10) and a massive pile of photos with no labels at all.

The Solution: The "Master-Apprentice" Team

The authors of this paper, Fuyou Mao and his team, built a system called MuDuo. They didn't try to teach the computer from scratch. Instead, they set up a "Master-Apprentice" training camp.

1. The Masters (The Foundation Models)

They hired two "Master" AI experts who are already famous for being great at drawing maps, but they specialize in different things:

Master CT (SAM-Med3D): An expert who knows exactly how buildings look but doesn't care much about the glowing traffic.
Master PET (SegAnyPET): An expert who knows exactly where the glowing traffic is but sometimes gets confused by the building shapes.

These Masters are "frozen," meaning they are already trained and won't change. They are the teachers.

2. The Apprentice (The Student)

They have a lightweight "Student" AI (a small, fast computer program) that is trying to learn how to draw the map using both cameras at once. The Student is cheap and fast, but it needs help because it hasn't seen enough examples.

3. The Training Process: Mutual Distillation

Here is where the magic happens. Instead of the Masters just lecturing the Student, they work together in a loop:

Step A: The Guess: The Student looks at an unlabeled photo and makes a rough guess at where the organs are.
Step B: The Refinement:
- The Student's rough guess is shown to Master CT, who uses their knowledge of building shapes to clean up the lines.
- The Student's rough guess is also shown to Master PET, who uses their knowledge of glowing traffic to highlight the active areas.
Step C: The Agreement Check (The Filter): Sometimes the Masters might disagree. Maybe Master CT thinks a spot is a wall, but Master PET thinks it's empty space.
- The system has a special rule: Only keep the guesses where both Masters agree. If they both say "Yes, this is an organ," the system marks it as a "High-Quality Label." If they disagree, the system throws that guess away.
Step D: The Lesson: The Student learns from these "High-Quality Labels" created by the Masters. Over time, the Student gets better and better at drawing the map on its own.

Why is this special?

Most previous methods tried to use just one Master (either CT or PET) or relied on the Student to guess its own labels (which often leads to mistakes).

MuDuo is unique because it forces the two Masters to cross-check each other.

Analogy: Imagine two detectives trying to solve a crime. Detective A looks at the footprints (CT), and Detective B looks at the fingerprints (PET). If Detective A says "The suspect was here" and Detective B says "No, the fingerprints are there," they argue. But if both point to the same spot, they are 100% sure. MuDuo only trusts the spots where both detectives agree.

The Results

The team tested this on a dataset called AutoPET. They tried to teach the system using only 5 labeled cases (5 hand-drawn maps) and 953 unlabeled cases (photos with no labels).

The Result: Their system (MuDuo) drew the maps much more accurately than any other method, even beating systems that tried to use more labeled data.
The "Boundary" Win: The system was particularly good at drawing the edges of the organs (the HD95 metric). It reduced errors in the boundaries by about 41% compared to the next best method. This is like drawing a circle that is perfectly round, rather than a wobbly one.

Summary

The paper claims that by using two specialized "Master" AIs to teach a "Student" AI, and by only trusting the parts where the Masters agree, they can create a highly accurate medical imaging tool using very few labeled examples. They call this Mutual Distillation, and it works like a team of experts double-checking each other's homework to ensure the student learns the right answer.

Technical Summary: Mutual Distillation of Dual-Foundation Models for Semi-Supervised PET/CT Segmentation

Problem Statement
Accurate organ segmentation in whole-body PET/CT is essential for quantitative metabolic analysis, disease staging, and radiotherapy planning. However, developing deep learning models for this task is hindered by two primary challenges:

Anatomical-Functional Mismatch: PET imaging offers high metabolic sensitivity but suffers from blurred boundaries and spill-over artifacts, while CT provides clear structural details. Fusing these modalities to resolve spatial discrepancies is difficult.
Data Scarcity: High-quality pixel-level 3D annotations require synchronized expertise from radiologists and nuclear medicine specialists, making labeled data expensive and scarce.
While Semi-Supervised Learning (SSL) and recent Foundation Models (like SAM) offer solutions, existing approaches are often limited to single-modality scenarios or fail to leverage the deep semantic interdependence between structural (CT) and functional (PET) imaging. Furthermore, standard SSL methods relying on pseudo-labeling often suffer from confirmation bias, reinforcing erroneous predictions.

Methodology: The MuDuo Framework
The authors propose MuDuo, a mutual distillation framework designed to synergistically exploit structural and functional foundation models to distill knowledge into a lightweight student network without manual prompting. The framework operates on dual-modal inputs ( $X_{CT}$ and $X_{PET}$ ) and consists of three main components:

Dual Foundation Model Teachers:
- SAM-Med3D: A frozen teacher specialized for CT, leveraging anatomical priors.
- SegAnyPET: A frozen teacher specialized for PET, leveraging metabolic priors.
- Student Network: A learnable 3D U-Net ( $S_\theta$ ) that processes dual-modal inputs.
- Prompt Generator: A lightweight UNet ( $G_\phi$ ) that bridges the student and the PET teacher.
Dual-Pathway Pseudo Annotation Generation:
For unlabeled data, the framework generates pseudo-labels through two distinct pathways adapted to modality characteristics:
- CT Pathway: The student's predictions (via the prompt generator) are downsampled and used as mask prompts for SAM-Med3D. The teacher refines these predictions based on CT anatomical priors.
- PET Pathway: The prompt generator produces coarse logits, which are converted into multi-point prompts (sampling top-K high-probability locations) rather than single centroids. This addresses the heterogeneous uptake typical in PET imaging. These points guide SegAnyPET to generate refined masks.
Mutual Distillation and Consistency:
- IoU-Based Consensus Filtering: To ensure reliability, the framework computes the Intersection-over-Union (IoU) between the CT and PET branch predictions. Only the top 50% of samples where both modalities agree (high consensus) are retained for training. Samples with invalid prompts are excluded.
- Dynamic Pseudo-Label Fusion: High-quality pseudo-labels are generated by dynamically mixing the CT and PET expert predictions using a random weighting factor ( $\alpha$ ). This introduces perturbation to prevent the model from memorizing its own predictions.
- Consistency Loss: The student network is trained to match these fused, high-quality pseudo-labels using Mean Squared Error (MSE), with a Gaussian ramp-up weight schedule to prevent early training instability.

Key Contributions

Dual-Modality Foundation Integration: Unlike prior single-modality SSL methods, MuDuo uniquely integrates SAM-Med3D (structural) and SegAnyPET (functional) to mutually distill knowledge, addressing the anatomical-functional mismatch inherent in PET/CT.
Automatic Prompting Mechanism: The framework eliminates the need for manual prompts or external prompt generators by utilizing the student model's own predictions to guide the foundation models, maximizing the utility of unlabeled data.
Consensus Filtering: The introduction of an IoU-based filtering mechanism ensures that only reliable, consensus-driven pseudo-labels supervise the student, mitigating the confirmation bias common in traditional pseudo-labeling.
Multi-Point Prompting for PET: The proposal of multi-point sampling for PET inputs better captures the spatial extent of organs with heterogeneous metabolic uptake compared to single-point prompting.

Experimental Results
The framework was evaluated on the AutoPET dataset (1,038 studies), utilizing a subset of 100 cases with expert annotations. Experiments were conducted under extreme data scarcity settings with 5, 10, and 20 labeled cases.

Performance: MuDuo achieved state-of-the-art performance across all settings. In the 5-labeled case scenario, it attained a Dice score of 46.93%, significantly outperforming the second-best method (CPS) and single-modality foundation approaches (SemiSAM+).
Boundary Accuracy: The method demonstrated superior boundary precision, reducing the 95th percentile Hausdorff Distance (HD95) to 8.50 voxels in the 5-labeled setting—a 41% reduction compared to SemiSAM+.
Comparison to Fully Supervised: With 10 labeled cases, the semi-supervised MuDuo method surpassed the dual-modality fully-supervised baseline by approximately 9% in Dice score, demonstrating the efficacy of mining unlabeled data via cross-modality consensus.
Ablation Studies: Experiments confirmed that using dual teachers outperforms single-modality variants, adaptive top-50% IoU filtering yields better results than fixed thresholds, and multi-point prompting (K=3) is superior to single-point prompting for PET segmentation.

Significance and Claims
The paper claims that MuDuo successfully bridges the gap between the task-specific precision of student models and the general segmentation priors of foundation models. By synergistically leveraging structural and functional priors, the framework achieves automatic, high-precision segmentation without the burden of expensive dual-modality annotations. The authors assert that their approach effectively breaks through the performance bottlenecks of limited annotations, offering a practical solution for quantitative analysis and radiotherapy planning in oncology where labeled data is scarce. The source code is made available to facilitate reproducibility.