Technical Summary: SynerMedGen

Problem Statement

The unification of multimodal understanding and visual generation represents a significant frontier in medical AI, particularly for clinical workflows requiring the interpretation and transformation of heterogeneous modalities (e.g., CT, MRI, PET). While recent unified Medical Multimodal Large Language Models (MLLMs) like HealthGPT and UniMedVL have begun to couple these capabilities, a critical misalignment persists. Existing frameworks typically treat understanding and generation as disjoint objectives, training the understanding branch on proxy tasks (e.g., global semantic recognition or lesion-centric VQA) that are weakly related to the specific requirements of medical image synthesis.

This "task misalignment" results in models that may excel at answering recognition-style questions but fail to provide the fine-grained, slice-level anatomical correspondences and modality-specific constraints necessary for faithful cross-modality synthesis. Consequently, unified models often struggle with preserving patient-specific pathology, maintaining pixel-level correspondence, and adhering to target-modality constraints, leading to hallucinations and structural inconsistencies.

Methodology: SynerMedGen

To address this, the authors propose SynerMedGen, a unified framework built on the principle of generation-aligned understanding. The core hypothesis is that understanding tasks must be derived directly from the generation dataset itself to ensure the learned representations are beneficial for synthesis.

1. Generation-Aligned Understanding Tasks

Instead of generic supervision, SynerMedGen constructs three specific tasks from paired synthesis data (source-target image pairs) to guide the understanding branch:

• Conditional Target Selection (CTS): Enforces slice-level correspondence. Given a source slice and a target modality request, the model must select the correct paired target slice from a set of candidates (including hard negatives from adjacent slices). This forces the model to learn fine-grained alignment rather than coarse semantics.
• Modality Identification (MI): Treats modality as an explicit, learnable control factor. The model predicts the specific imaging modality (e.g., CT, CBCT, PET, specific MRI sequences) of an input image, reducing reliance on superficial shortcuts and ensuring modality is disentangled from patient content.
• Transformation Instruction Alignment (TIA): Grounds the direction and constraints of the synthesis. Given a source-target pair, the model selects the text description that best aligns with the observed visual transformation (specifying what changes, e.g., texture/contrast, and what remains invariant, e.g., anatomy/lesions). This disambiguates the synthesis direction.

2. Two-Stage Training Strategy

SynerMedGen employs a two-stage optimization schedule to transfer these beneficial representations to the generation branch:

• Stage I: Generation-Aligned Understanding (GAU): The model is trained exclusively on the three generation-aligned tasks (CTS, MI, TIA) using the understanding pathway (Vision Encoder + Understanding Expert). This stage learns a "synthesis-sufficient" representation that preserves correspondence, disentangles modality, and encodes mapping constraints.
• Stage II: Understanding-Conditioned Generation (UCG): The model is initialized from Stage I and fine-tuned for conditional synthesis in the VAE latent space using flow matching. The generation expert conditions on the source evidence and the learned representations from Stage I to synthesize the target image.

3. Dataset: SynerMed

The authors release SynerMed, a large-scale dataset comprising:

• 1M paired synthesis samples covering diverse cross-modality tasks (CT↔CBCT, CT↔MRI, CT↔PET, multi-sequence MRI) across brain, pelvis, and whole-body regions.
• 2M generation-derived understanding instances constructed from the paired data to support the three proposed tasks.

Key Results

The paper evaluates SynerMedGen across 22 medical image synthesis tasks on benchmarks including SynthRAD2023, AutoPET, and BraTS, comparing against specialized synthesis models (e.g., ResViT, SynDiff, RCD) and recent unified MLLMs (HealthGPT, UniMedVL).

• Superior Synthesis Performance: SynerMedGen consistently outperforms all baselines. For instance, on the BraTS T2→T1CE task, it improves SSIM by 4.3% over the specialized model SynDiff. On whole-body PET→CT, it achieves an SSIM of 91.10, significantly higher than specialized and unified competitors.
• Zero-Shot Generalization: Remarkably, the model trained only on the understanding tasks (Stage I, GAU) achieves strong zero-shot performance across 22 synthesis tasks, demonstrating that the learned representations are inherently beneficial for generation even without explicit generation training.
• Robustness to Unseen Data: The framework demonstrates robust generalization to unseen datasets (MyoPS cardiac MRI, SynthRAD2025), outperforming baselines in both SSIM and MAE metrics.
• Task Alignment Validation: Ablation studies confirm that replacing the generation-aligned tasks with traditional medical understanding tasks leads to significantly lower performance, validating that the specific design of the understanding tasks is the primary driver of the improvement.
• Qualitative Improvements: Visual comparisons show that SynerMedGen produces outputs with better anatomical consistency, fewer hallucinated structures, and more accurate target-modality appearances compared to models trained with traditional supervision.

Significance and Claims

The paper claims that the primary bottleneck in unified medical MLLMs is the gap between understanding and generation objectives. By explicitly aligning understanding tasks with generation goals, SynerMedGen bridges this gap, proving that "what form of understanding truly benefits generation" is a function of how closely the understanding supervision mirrors the constraints of the synthesis task.

The authors position SynerMedGen not just as a performance improvement, but as a methodological shift: deriving understanding supervision directly from paired generative data allows the model to learn representations that are "synthesis-sufficient" by design. This approach reduces hallucinations and enhances the fidelity of cross-modality translation, offering a scalable path toward unified medical AI systems capable of both interpreting complex clinical semantics and transforming imaging data for diagnosis and treatment planning. The release of the SynerMed dataset and code aims to support further research into understanding-generation synergy in the medical domain.