🤖 AI

DualDiT: A Conditional Dual-Output Diffusion Transformer for Joint OCT Image and Segmentation Mask Generation

The paper proposes DualDiT, a conditional dual-output Diffusion Transformer that outperforms traditional U-Net-based diffusion models in generating realistic OCT images and segmentation masks for mouse eyes, effectively addressing data scarcity through superior generative fidelity and downstream segmentation utility.

Original authors: Fernando García-Torres, Rocío del Amor, Sandra Morales, Álvaro Barroso, Peter Heiduschka, Björn Kemper, Valery Naranjo

Published 2026-08-03

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Fernando García-Torres, Rocío del Amor, Sandra Morales, Álvaro Barroso, Peter Heiduschka, Björn Kemper, Valery Naranjo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to recognize a specific type of fruit, like a strawberry, but you only have three photos of strawberries and thousands of photos of apples. The robot would get confused, thinking every red fruit is an apple. This is a huge problem in medical science, where doctors need thousands of labeled images to train computers to spot diseases, but getting those images is hard, expensive, and sometimes impossible because patients are rare or the data is private. To fix this, scientists use "generative AI," a type of computer program that acts like a super-creative artist. Instead of just copying existing pictures, it learns the rules of how things look and then draws brand-new, fake pictures that look so real they could fool a human. But here's the tricky part: if the AI draws a fake strawberry, it also needs to draw a perfect map showing exactly where the seeds are. If the map doesn't match the fruit, the robot learning from it will get even more confused.

This is exactly the challenge researchers faced with a special kind of eye scan called an OCT (Optical Coherence Tomography). These scans are like ultra-detailed cross-sections of the retina, the light-sensitive layer at the back of the eye. Doctors use them to spot diseases, but labeling the tiny layers inside these scans by hand takes experts hours of work, leaving them with very few examples to teach their computers. The paper you are about to read introduces a new, clever AI artist called DualDiT. Think of DualDiT not just as a painter, but as a "dual-wielding" wizard that holds a brush for the image in one hand and a stencil for the map in the other. Unlike older AI models that tried to paint the picture and the map separately (often resulting in messy mismatches), DualDiT learns to create both at the exact same time, ensuring the layers in the fake eye scan line up perfectly with the fake map. The researchers tested this on tiny, preserved mouse eyes, which are crucial for studying human eye diseases but are notoriously difficult to scan and label.

The Magic of the Dual-Output Artist

The team behind this study, led by Fernando García-Torres and colleagues, wanted to solve the "data shortage" problem for mouse eye scans. They built a new AI model called DualDiT (Conditional Dual-Output Diffusion Transformer). To understand why this is special, imagine you are trying to teach a child to draw a house. If you show them a picture of a house and a separate drawing of the roof, they might draw a house with a roof floating in the sky. Older AI models worked a bit like that, trying to generate the image and the "mask" (the outline of the important parts) separately or in a clumsy way.

DualDiT, however, is like a child who learns to draw the house and the roof simultaneously, understanding that the roof must sit on the walls. The researchers trained this AI on a dataset of mouse retinas that had been preserved in two different ways: one group was kept in a liquid (physiological) medium, and the other was embedded in hard resin. These two methods look slightly different, like taking a photo of a fish in water versus a photo of a fish on a plate. The AI was taught to recognize these differences and generate new, fake scans that matched the specific style of the liquid or the resin.

The Results: Fooling the Experts

The team put their new AI to the test against two older, more traditional methods: one that worked directly on the raw pixels (called DDPM) and another that worked in a compressed "latent" space (called LDM). They measured how realistic the fake images were using a score called FID (Fréchet Inception Distance), where a lower number means the fake images are more similar to real ones.

The results were clear: DualDiT was the winner.

DualDiT achieved a FID score of 56.14 and a spatial FID (sFID) of 114.35.
The older DDPM model scored 164.55 (FID) and 254.52 (sFID).
The LDM model scored 102.21 (FID) and 150.66 (sFID).

In plain English, DualDiT's fake images were much closer to the real thing than the other models. The older models often made mistakes, like drawing blurry layers or creating "hallucinations" where the retina looked like a shape it shouldn't be. DualDiT, however, produced sharp, detailed images that looked just like the real mouse eyes, complete with the tiny speckles and textures that make them look authentic.

But the real test wasn't just how pretty the pictures were; it was whether they could actually help a computer learn better. The researchers used the fake images generated by DualDiT to train a new computer program to segment (outline) the retinal layers.

When they used only the real data, the computer got a score of 0.908 (Dice score).
When they added the fake DualDiT images to the training, the score jumped to 0.927.
The older models actually made things worse or didn't help much; for example, the DDPM model caused the computer's performance to crash when dealing with the resin-embedded samples.

The Human Test: Can You Spot the Fake?

To see if the fake images were truly convincing, the researchers showed them to three human experts. The experts were asked to look at a mix of real and fake images and guess which was which. The results were surprising: the experts were wrong 46% of the time when trying to identify the fake images as real. They also mistook real images for fakes 42% of the time. This suggests that DualDiT created images so realistic that even trained eyes couldn't reliably tell the difference.

What This Means (and What It Doesn't)

The paper suggests that DualDiT is a powerful new tool for creating "synthetic data" to help train medical AI when real data is scarce. It proves that using a Transformer-based architecture (a type of AI known for understanding long-range connections) is better than the older U-Net based methods for this specific job of generating both images and maps at once.

However, the authors are careful to note that this isn't a magic cure-all yet. The study was done on a relatively small dataset (about 340 scanned slices from 10 mice), and the images were a specific resolution (512 × 256 pixels). While the results are promising, the authors state that more testing on larger datasets is needed to be sure this works for all types of eye scans. They also mention that the current setup requires significant computing power, which might be a hurdle for some clinics.

In short, DualDiT is a highly skilled "dual-wielding" artist that can paint fake mouse eye scans and their matching maps so well that they fool experts and help computers learn faster. It suggests that by teaching AI to create images and their labels together, we can overcome the biggest bottleneck in medical AI: the lack of labeled data. But like any new tool, it needs more practice and bigger test runs before it can be trusted to diagnose real patients.

Technical Summary: DualDiT – A Conditional Dual-Output Diffusion Transformer for Joint OCT Image and Segmentation Mask Generation

Problem Statement
The paper addresses the critical shortage of annotated medical imaging data, specifically in Optical Coherence Tomography (OCT) of ex vivo mouse retinas. While human OCT datasets are relatively abundant, mouse retinal OCT data is scarce, and manual segmentation of retinal layers (specifically the Upper Retinal Cell Layers, or URCL) is extremely labor-intensive due to tiny structures and the need for expert knowledge. Existing generative models, particularly those based on U-Net architectures (DDPM and Latent Diffusion Models), have shown promise but often struggle to jointly generate anatomically aligned images and segmentation masks with high fidelity. Furthermore, Diffusion Transformers (DiT), which leverage self-attention to model long-range dependencies, have remained largely unexplored in the context of joint image-mask synthesis for medical imaging.

Methodology
The authors propose DualDiT, a conditional dual-output Diffusion Transformer framework designed to simultaneously synthesize OCT B-scans and their corresponding URCL segmentation masks. The methodology involves three primary stages:

Joint Latent Representation: To improve computational efficiency, both the OCT image ( $x$ ) and the segmentation mask ( $m$ ) are encoded into a shared latent space using a pre-trained Variational Autoencoder (VAE). The latent representations ( $z_x$ and $z_m$ ) are concatenated along the channel dimension to form a joint latent tensor $z_0 = [z_x || z_m]$ . This allows the model to learn the structural correspondence between retinal appearance and layer segmentation within a compressed space.
Conditional Diffusion Process: The framework employs a Transformer-based backbone (specifically a DiT-XL/2 architecture) rather than the traditional U-Net denoisers. The model is conditioned on the embedding medium domain (physiological liquid vs. resin), which introduces distinct visual characteristics and noise profiles. During training, Gaussian noise is progressively added to the joint latent tensor. The Transformer blocks utilize adaptive Layer Normalization (adaLN-Zero) to condition the denoising process on both the timestep and the domain label. The network is trained to predict the added noise, effectively learning the joint distribution $p_\theta(x, m | y)$ .
Dual-Output Synthesis: During inference, the model starts from pure noise and iteratively denoises the joint latent tensor. The resulting clean latent is split back into image and mask components, which are then decoded by the frozen VAE decoder to produce the final synthetic OCT image and segmentation mask.

Key Contributions

Novel Architecture: The paper introduces the first DiT-based framework formulated specifically for paired image-mask synthesis, moving beyond single-output generation or U-Net-based denoisers.
Domain-Conditioned Synthesis: The model successfully synthesizes data conditioned on two distinct embedding media (resin and physiological), addressing the domain shift challenges inherent in ex vivo mouse retinal imaging.
Comprehensive Evaluation: The study provides a rigorous quantitative and qualitative assessment, comparing DualDiT against state-of-the-art baselines (Conditional DDPM and Conditional LDM) using Fréchet Inception Distance (FID), spatial FID (sFID), and downstream segmentation performance.
Expert Validation: The realism of the generated data is validated through a panel of domain experts, assessing both the visual fidelity of the B-scans and the anatomical consistency of the masks.

Results

Generative Quality: DualDiT significantly outperformed both DDPM and LDM baselines. It achieved the lowest FID (56.14) and sFID (114.35) scores globally. In contrast, DDPM scored 164.55 (FID) and 254.52 (sFID), while LDM scored 102.21 (FID) and 150.66 (sFID). Qualitative analysis indicated that DualDiT produced higher resolution layers and more accurate reproduction of OCT artifacts (e.g., speckle noise) compared to the baselines, which often suffered from hallucinations or excessive brightness.
Downstream Utility: When used as a data augmentation strategy for training a U-Net segmentation model, DualDiT-generated data consistently improved performance. The best configuration (augmenting the real training set with 1,200 synthetic pairs) increased the global Dice coefficient from 0.908 (Baseline) to 0.927 and the IoU from 0.838 to 0.868. Notably, DDPM augmentation degraded performance in the resin subset, whereas DualDiT maintained or improved performance across both embedding domains.
Perceptual Realism: In an expert evaluation, panelists incorrectly classified an average of 46% of synthetic samples as real. Conversely, 42% of real samples were misclassified as synthetic. Statistical analysis (AUC) showed no significant difference in the experts' ability to distinguish embedding media between real and synthetic samples, confirming the model's ability to capture domain-specific characteristics.

Significance and Claims
The paper claims that DualDiT demonstrates the efficacy of transformer-based diffusion models in learning the joint distribution of medical images and their annotations. The authors assert that this approach surpasses conventional U-Net-based diffusion models in generative fidelity, downstream segmentation utility, and perceptual realism.

The significance of this work lies in its potential to serve as a robust data augmentation strategy for annotation-scarce medical imaging applications. By generating anatomically consistent image-mask pairs, DualDiT can help overcome the bottleneck of manual labeling in specialized domains like ex vivo mouse retinal imaging. The authors conclude that while limitations exist regarding resolution and dataset size, the proposed framework highlights a promising direction for foundation models in medical imaging, enabling large-scale, cross-modality synthesis and clinically interpretable generative pipelines. The paper remains modest, acknowledging that further validation on larger, independent datasets is required to establish statistical significance and generalizability across broader clinical scenarios.

The Magic of the Dual-Output Artist

The Results: Fooling the Experts

The Human Test: Can You Spot the Fake?

What This Means (and What It Doesn't)

More like this