Technical Summary: CRAFT – Clinical Reward-Aligned Finetuning for Medical Image Synthesis

1. Problem Statement

Foundation diffusion models have demonstrated remarkable success in generating photorealistic natural images. However, adapting these models to medical imaging remains challenging. In the medical domain, limited labeled data often exacerbates "hallucination-like" synthesis, where generated images appear visually plausible but lack pathology-specific findings or contain clinically implausible features.

Existing evaluation metrics, such as Fréchet Inception Distance (FID) and Inception Score (IS), measure distribution-level visual fidelity but fail to quantify per-image alignment with pathology-relevant criteria. Similarly, metrics like CLIPScore measure text-image alignment but do not directly evaluate whether a generated image satisfies the specific clinical criteria required for a given diagnosis. Consequently, there is a gap in both the generation of clinically useful synthetic data and the ability to evaluate it against expert-defined medical standards.

2. Methodology

The authors propose CRAFT (Clinical Reward-Aligned Finetuning), a framework designed to adapt general-purpose diffusion models toward clinically aligned medical image synthesis. The methodology operates in two distinct stages:

Stage 1: LLM-Driven Semantic Enrichment

To address the sparsity of clinical labels, the framework utilizes Multimodal Large Language Models (MLLMs) to generate two complementary textual artifacts for each image-label pair $(x_i, y_i)$:

1. Enriched Prompt ($t_{ep}$): An instance-specific visual description generated by the MLLM that captures fine-grained attributes (morphology, color, borders) grounded in the specific training image $x_i$. The ground-truth label $y_i$ is provided as a hard constraint to prevent diagnostic hallucination.
2. Natural Language Checklist ($t_c$): A class-level structured checklist encoding shared clinical criteria for the disease $y$ (e.g., required shape, texture, color). This serves as a stable clinical constraint to enforce consistency across generations.

Stage 2: Clinical Reward-Aligned Finetuning

The diffusion model is fine-tuned using a reward-based objective that combines the standard diffusion reconstruction loss with a differentiable clinical reward computed by a frozen Vision-Language Model (VLM) critic. The training objective is:
$$\mathcal{L}(\theta) = \mathbb{E}[\lambda_{diff}\|\epsilon - \epsilon_\theta(z_t, t, c)\|_2^2 - \lambda_{cam} r_{cam}(\hat{x}_0, x_r, c, t_c, y)]$$
Where $r_{cam}$ is the Clinical Alignment Score (CAS), computed as the sum of four differentiable reward components:

1. Visual Description Consistency (VDC): Maximizes cosine similarity between the generated image embedding and the enriched prompt embedding ($t_{ep}$), ensuring instance-specific visual fidelity.
2. Clinical Criteria Satisfaction (CCS): Maximizes cosine similarity between the generated image and the disease-specific checklist ($t_c$), enforcing class-level clinical consistency.
3. Diagnostic Discriminability (DD): Uses a linear probe trained on frozen VLM embeddings to maximize the log-likelihood of the correct disease class, ensuring the generated features are discriminative for diagnosis.
4. Semantic Feature Similarity (SFS): Maximizes cosine similarity between the generated image and its paired real reference image ($x_r$) in the embedding space, anchoring the generation to real medical representations to prevent semantic drift.

Optimization: The framework employs DRaFT-K to backpropagate reward gradients only through the final $K$ denoising steps, reducing memory costs. Only Low-Rank Adaptation (LoRA) parameters are updated, while the VLM encoders remain frozen.

3. Key Contributions

1. Clinical Alignment Score (CAS): The introduction of a foundation-model-based proxy for clinical alignment that decomposes evaluation into four primitives: VDC, CCS, DD, and SFS. Unlike generic metrics, CAS targets clinically meaningful failure modes without requiring manual preference labels.
2. CRAFT Framework: A reward-aligned finetuning method that transfers structured medical knowledge from MLLMs and VLMs into diffusion models via prompt enrichment and differentiable reward optimization.
3. Comprehensive Evaluation: The framework is validated across four diverse medical imaging modalities (dermatology, radiology, histopathology, and retinal fundus imaging), demonstrating improvements in average CAS, downstream classification performance, and the reduction of low-alignment "failure tails."

4. Experimental Results

The authors evaluated CRAFT on four datasets: Fitzpatrick17k (dermatology), CheXpert (chest X-ray), BreakHis (histopathology), and ORIGA (retinal fundus).

• Quantitative Performance: CRAFT achieved the highest CAS across all four datasets compared to strong baselines (Zero-Shot, Textual Inversion, TI+LoRA, and DPO). For instance, on Fitzpatrick17k, CAS improved from 0.376 (TI+LoRA) to 0.417.
• Low-Alignment Tail Reduction: Crucially, CRAFT significantly reduced the fraction of samples falling below the 25th percentile of the real-image CAS distribution. Relative to the strongest baseline, CRAFT reduced the low-alignment rate by 5.5–34.7 percentage points across datasets, corresponding to an average relative reduction of 20.4%. This indicates a substantial decrease in hallucination-like generations.
• Downstream Utility: In real+synthetic augmentation settings (20% synthetic data), CRAFT consistently yielded the highest downstream classification accuracy across all datasets and data scales (10-shot, 20-shot, and full data).
• Validation:
  • Evaluator Robustness: Results held when evaluated with out-of-family encoders (MetaCLIP2).
  • Checklist Auditing: CRAFT achieved the highest pass rate on structured clinical checklists using an external medical evaluator (Lingshu).
  • Physician Preference: In a blinded study on CheXpert involving two physicians, CRAFT samples were ranked first in 67% of cases, outperforming all baselines.
  • Memorization Analysis: Tests using SSIM, pHash, and DINOv2 nearest-neighbor analysis found no evidence of pixel-level or feature-space memorization of training images.

5. Significance and Claims

The paper claims that optimizing for clinically motivated reward components is a productive path toward reliable medical image generation. The significance of CRAFT lies in its ability to move beyond simple visual realism to ensure clinical plausibility.

• Beyond Average Metrics: The authors emphasize that average performance metrics can hide clinically critical failures. By explicitly analyzing and reducing the "low-alignment tail," CRAFT addresses the risk of injecting misleading pathology cues into downstream training.
• Proxy vs. Ground Truth: The authors maintain a modest stance, explicitly stating that CAS is a proxy for clinical alignment and not a substitute for expert review. They frame physician preference results as supporting evidence rather than definitive clinical validation.
• Generalizability: While focused on medical imaging, the framework is presented as extensible to other domains with expert-defined semantic criteria, such as satellite imagery or industrial inspection.

The work concludes that while synthetic data generation is a partial remedy for medical data scarcity, methods like CRAFT are necessary to ensure that the generated data is not only visually realistic but also clinically valid and useful for downstream tasks.