⚡ electrical engineering

2D Versus 3D Diffusion for In Silico Training of Interventional X-ray AI Models

This paper demonstrates that view-conditioned 2D diffusion models can generate synthetic X-ray images capable of training anatomical landmark detection models that generalize to real interventional X-ray data with performance rivaling models trained on real images, offering a promising alternative to data-constrained mechanistic simulation methods.

Original authors: Sampath Rapuri, Jeremy Ko, Benjamin D. Killeen, Russell H. Taylor, Mathias Unberath

Published 2026-06-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Sampath Rapuri, Jeremy Ko, Benjamin D. Killeen, Russell H. Taylor, Mathias Unberath

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to navigate a complex city using only 2D street maps (X-rays). The problem is, you don't have enough real maps with the correct "treasure hunt" clues (annotations) marked on them to teach the robot effectively.

This paper explores two different ways to fake these maps using computer simulations so the robot can learn without needing real human data. The researchers tested which "fake" method works best for teaching the robot to find specific landmarks (like bones) on X-ray images.

Here is the breakdown of their two approaches, explained with simple analogies:

The Problem: The "Real Map" Bottleneck

Usually, to make a fake map, you start with a perfect 3D model of a city (a CT scan of a real patient), take a picture of it from a specific angle to make a 2D map (a DRR), and then mark the landmarks.

The Catch: You need a real 3D model to start with. Getting these models is hard, expensive, and limits how many fake maps you can make.

The Two New "Fake Map" Generators

The researchers tried two new methods using Diffusion Models (a type of AI that creates images from scratch, similar to how you might imagine a painting and then paint it).

Method 1: The "3D Sculptor" (DiffCT-DRR)

How it works: Imagine an AI artist that can sculpt a brand new, fake 3D clay body (a synthetic CT scan) out of thin air. Once the clay body is made, the researchers use a virtual camera to take a picture of it (creating a DRR) and then mark the landmarks.
The Analogy: It's like a chef who bakes a completely new, fake cake from scratch, then slices it to show you what's inside.
The Result: This method worked well. The fake 3D bodies were realistic enough that the robot learned to find landmarks almost as well as if it had learned from real patient scans.

Method 2: The "2D Painter" (DiffXray)

How it works: Instead of making a 3D body first, this AI skips the middleman. It takes a simple outline of organs (like a coloring book page) and a specific camera angle, and it paints a realistic X-ray image directly.
The Analogy: Instead of baking a cake and slicing it, this is like looking at a simple line drawing of a cake and instantly painting a photorealistic photo of that cake from a specific angle.
The Result: This was the surprise winner. When the researchers used this method to create many different views (by slightly changing the camera angle), the robot learned better than it did with the "3D Sculptor" method.

The Big Discovery: "Cloud" vs. "Exact Match"

The researchers ran two types of tests:

The "Exact Match" Test: They forced the AI to create fake images that looked exactly like the real test images (same angle, same pose).
- Result: The "2D Painter" (DiffXray) did okay, but it struggled a bit compared to using real data. It was precise but not perfect.
The "Cloud" Test (The Real Winner): They let the AI generate thousands of new images from slightly different, random angles (like taking photos of the same object from a cloud of different viewpoints).
- Result: This is where the magic happened. By feeding the robot a massive, diverse "cloud" of fake X-rays generated by the 2D Painter, the robot became incredibly good at finding landmarks. In fact, for the hardest test cases, the robot trained on these fake 2D images performed better than the one trained on the "3D Sculptor" method.

The Bottom Line

The paper claims that you don't necessarily need real 3D patient scans to train AI for X-ray surgery.

You can use a 3D AI to make fake bodies, but it's a bit rigid.
You can use a 2D AI to paint fake X-rays directly from simple outlines. This method is more flexible. When you use it to generate a huge, diverse variety of training images, it teaches the AI to find bone landmarks just as well as (and sometimes better than) training it on real human data.

In short: If you want to teach a robot to read X-rays, you don't need a library of real patients. You can just give the robot a "paint-by-numbers" kit and let a 2D AI paint it a million different pictures of what those patients might look like. The robot learns just fine from those paintings.

Technical Summary: 2D Versus 3D Diffusion for In Silico Training of Interventional X-ray AI Models

Problem Statement
The development of AI models for interventional X-ray imaging is hindered by a scarcity of available annotated data, as routine care does not generate the necessary ground truth annotations. While mechanistic simulation of Digitally Reconstructed Radiographs (DRRs) from real CT volumes (RealCT-DRR) has proven effective for tasks like segmentation and landmark detection, this approach relies on the availability of annotated, high-resolution anatomical models derived from real patients or specimens. This dependency creates a bottleneck regarding data quantity and variability. The authors investigate whether generative diffusion models (DMs) can overcome this bottleneck by synthesizing training data without requiring real 3D anatomical models, comparing two distinct paradigms: 3D diffusion for CT generation versus 2D diffusion for direct X-ray synthesis.

Methodology
The study evaluates two novel data synthesis paradigms against established baselines (RealXray and RealCT-DRR) using pelvic anatomical landmark detection as the motivating task.

DiffCT-DRR (3D Diffusion): This framework utilizes a 3D latent diffusion model to generate synthetic CT volumes. These synthetic CTs are processed through automated segmentation tools to extract surface meshes (e.g., pelvis). A Statistical Shape Model (SSM) is used to propagate 3D landmark annotations to the synthetic CTs. Finally, DRRs are generated from these synthetic CTs using camera projection matrices matched to real X-ray poses.
DiffXray (2D Diffusion): This framework employs a view-conditioned 2D diffusion model. Instead of generating 3D volumes, the model is trained on real X-ray images and their corresponding segmentations. During inference, plausible organ segmentations (projected from a 3D patient model or computational phantom) condition the generation of synthetic X-ray images. Landmarks are obtained by projecting SSM landmarks registered to the patient model.
Experimental Setup: The authors used the DeepFluoro dataset (366 fluoroscopic images from 6 cadaveric specimens). Experiments included:
- Precisely Matched Cross-Validation: Synthetic data was generated to match the exact poses of real X-rays to isolate the impact of the data domain.
- "Cloud" Experiments: Synthetic data was sampled from arbitrary views (perturbed by ±2.5°) to test scalability and the benefits of increased data diversity.
- Model Architecture: A Swin U-Net with combined segmentation and heatmap outputs was trained using domain randomization techniques to facilitate sim-to-real transfer.

Key Results

Precisely Matched Performance: In data-poor, strictly matched scenarios, DiffXray and DiffCT-DRR generally underperformed compared to RealCT-DRR and RealXray, particularly at lower confidence thresholds (lower NCC). However, at high confidence thresholds (0.60 NCC), DiffXray achieved the lowest mean $\ell_2$ error (4.40 ± 1.87 mm), rivaling RealXray (4.89 ± 6.42 mm). DiffCT-DRR showed comparable performance to RealCT-DRR at high confidence but exhibited rapid performance degradation at lower thresholds.
Scalability ("Cloud" Experiments): When the DiffXray framework was used to generate additional training data from varied views (Cloud sampling), it significantly enhanced model performance. For a challenging specimen (Specimen 6), a model trained on DiffXray-Cloud achieved an average $\ell_2$ error of 4.47 ± 1.49 mm at 0.60 NCC, outperforming the RealCT-DRR-Cloud baseline (4.87 ± 1.75 mm) and demonstrating superior robustness in high-confidence predictions.
Generalization: The study demonstrates that synthetic 2D diffusion-based X-rays can train landmark detection models that generalize to real X-ray images with performance rivaling models trained on real data, provided sufficient data diversity is introduced.

Significance and Claims
The paper provides preliminary evidence that synthetic, 2D diffusion-based training data can serve as a substitute for real X-ray data in interventional imaging. The authors claim that the DiffXray paradigm offers a promising avenue for generating large, diverse datasets without the bottleneck of acquiring annotated real CTs or specimens. While acknowledging limitations—such as the current scarcity of high-resolution 3D CT synthesis models and the reliance on underlying simulation realism for 3D approaches—the work validates an entirely in silico paradigm for training robust AI models. The results suggest that conditional diffusion models offer a scalable and controllable alternative to mechanistic DRR simulation, potentially enabling the training of AI models for tasks where real annotated data is unavailable.