💻 computer science

DINO-3DRA: Leveraging 2D Foundation Model Semantics for 3D Cerebral Aneurysm Segmentation

DINO-3DRA is a novel dual-path framework that leverages frozen 2D foundation model (DINOv3) features injected into a 3D U-Net via spatial mixing and residual fusion to achieve state-of-the-art, robust cerebral aneurysm segmentation in 3DRA by overcoming class imbalance and preserving anatomical continuity without requiring large-scale 3D pretraining.

Original authors: Jiayang Lu, Fengming Lin, Alejandro F. Frangi, Ali Sarrami-Foroushani

Published 2026-08-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jiayang Lu, Fengming Lin, Alejandro F. Frangi, Ali Sarrami-Foroushani

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to spot a tiny, dangerous bubble on a complex web of pipes inside a human head. This is the world of medical imaging, specifically looking at 3D scans of blood vessels to find cerebral aneurysms. These aneurysms are like weak spots on a tire that could burst, and finding them early is a matter of life and death. However, teaching computers to do this is incredibly hard. The "bubbles" are tiny compared to the massive web of healthy pipes, and they look almost exactly like the pipes themselves. It's like trying to find a single, slightly swollen knot in a giant ball of yarn where every knot looks the same.

To solve this, scientists often use "foundation models." Think of these as super-smart students who have read billions of books (or in this case, looked at billions of photos) before ever seeing a medical scan. They know what edges, shapes, and structures look like in the real world. But there's a catch: these super-smart students are experts at looking at flat, 2D pictures (like a single page of a book), while the medical scans are 3D (like a whole book). If you just take the 2D expert's knowledge and try to paste it onto a 3D problem without thinking, the information gets chopped up and loses its continuity, like trying to understand a 3D sculpture by only looking at a stack of disconnected paper cutouts. This paper tackles the tricky question of how to take that 2D "super-student" knowledge and smoothly blend it into a 3D brain scan without losing the big picture.

The researchers behind this study, Jiayang Lu and colleagues, have built a clever new system called DINO-3DRA. Their main finding is that you can successfully transfer the knowledge from a frozen 2D vision model (specifically one called DINOv3) into a 3D medical scanner, but only if you use a special "translator" to make sure the information flows correctly. They discovered that simply pasting the 2D features onto the 3D model doesn't work well; instead, they created a dual-path system where the 3D model learns the shape of the volume, while the 2D model provides a "map" of what things should look like.

To make this work, they invented two key tools. First, they used something called a Room-Lite spatial mixer. Imagine you have a stack of 2D slices of an orange, and you want to turn them back into a whole 3D orange. If you just stack them, you might lose the curve of the peel between slices. The Room-Lite mixer is like a gentle hand that smooths out the connections between the slices, ensuring the 3D shape feels continuous and real. Second, they used calibrated residual fusion. This is like a smart filter that decides how much of the 2D "super-student's" advice to listen to. If the 2D advice is confusing or doesn't fit the 3D picture, the filter ignores it; if it's helpful, it blends it in perfectly. This prevents the system from getting confused by the difference between the 2D training data and the 3D medical reality.

When they tested this new system on data from multiple hospitals, the results were impressive. The DINO-3DRA system achieved a segmentation score (called Dice) of 0.758 for finding aneurysms, which is a 13% improvement over the previous best standard method (nnU-Net). It also reduced the error in measuring the distance to the edge of the aneurysm to just 2.75 mm. Perhaps most importantly, the system was much more stable. In tests where other models completely failed to find the aneurysm (scoring below 0.5), DINO-3DRA had zero failures. Even when they tested it on completely different datasets from other hospitals without retraining it, the system remained reliable, cutting the number of total failures from over 11% down to 0%.

The paper explicitly rules out the idea that you can just take a 2D model and use it directly on 3D data without special handling; their experiments showed that "naïve" approaches actually performed worse than the standard 3D models. They also found that the improvements didn't just come from changing the math (the loss function) but specifically from the way they bridged the 2D and 3D worlds. While the system is very good, the authors note it sometimes struggles with very large or oddly shaped aneurysms, tending to be a bit conservative and labeling the edges as "vessels" rather than "aneurysms." However, they suggest this trade-off is actually useful in a clinical setting, as it's better to reliably find the problem than to miss it or get the exact boundary wrong. Ultimately, this work suggests that by carefully mixing 2D foundation model knowledge with 3D anatomy, we can build medical AI that is not only more accurate but also more robust when facing real-world variations in how scans are taken.

Technical Summary: DINO-3DRA

Problem Statement
Accurate segmentation of intracranial aneurysms in 3D Rotational Angiography (3DRA) is hindered by three primary challenges: extreme class imbalance (aneurysms constitute <1% of voxels), morphological similarity between aneurysms and parent vessels, and significant inter-scanner variability. Furthermore, while 2D vision foundation models (VFMs) like DINOv3 have learned dense structural priors from ~1.7 billion natural images, volumetric 3D models lack comparable large-scale pretraining due to data scarcity. Naïve approaches that transfer 2D foundation models to 3D tasks via slice-wise processing often fragment anatomical continuity and destabilize optimization, failing to compete with optimized 3D architectures.

Methodology
The authors propose DINO-3DRA, a dual-path framework designed to achieve effective cross-dimensional semantic transfer without fine-tuning the foundation model. The architecture consists of:

Dual-Path Encoder:
- Trainable 3D Branch: A standard 3D U-Net backbone processes volumetric inputs, expanding channels from 32 to 256 across four encoder stages.
- Frozen 2D Semantic Branch: A frozen DINOv3-Small (ViT-S/16) model processes axial slices independently. Each slice is replicated to RGB, resized to 224×224, and processed to generate 384-dimensional semantic embeddings. These embeddings capture slice-level vascular context and are broadcast to spatial supports matching the U-Net's skip connection resolutions.
Integration Mechanisms:
- Room-Lite Spatial Mixers: To address the lack of inter-slice continuity inherent in independent slice processing, the authors introduce a parameter-efficient module. This module restores volumetric coherence using learnable depth-positional biases and depthwise-separable 3D convolutions, effectively mixing slice-wise tokens into an isotropic 3D volume.
- Calibrated Heterogeneous Fusion: To align the statistically different feature distributions of the frozen DINO branch and the trainable U-Net, a residual fusion strategy is employed at each skip connection. This involves a 1×1×1 projection and a learnable scaling mechanism, ensuring the U-Net features are preserved if the foundation features are unhelpful.
Training Objective:
Due to severe class imbalance, a composite loss function is used: $L = 0.3 L_{vessel} + 0.5 L_{aneurysm} + 0.2 L_{CE}$ . The vessel Dice loss operates on the union of vessel and aneurysm classes to maintain vascular continuity, while the aneurysm Dice isolates lesion segmentation. Class-weighted cross-entropy stabilizes per-voxel classification.

Key Contributions

Lightweight Cross-Dimensional Integration: The framework combines a frozen DINOv3-Small with a 3D U-Net, adding only ~120K parameters to the backbone. It achieves state-of-the-art performance with a total of 5.72M trainable parameters.
Room-Lite Spatial Mixer: A novel module that restores volumetric coherence from slice-wise features using depth-aware positional encoding and depthwise-separable convolutions.
Calibrated Heterogeneous Fusion: A residual injection method with learnable scaling that stabilizes the alignment between frozen foundation features and trainable domain-specific features.
Systematic Validation: The study demonstrates that performance gains stem from structured cross-dimensional transfer rather than loss design alone, showing that bridged foundation features improve the anatomical continuity between aneurysms and parent vessels.

Results
Evaluated on the multi-centre @neurIST dataset (233 patients), DINO-3DRA achieved:

Aneurysm Segmentation: Dice score of 0.758 and HD95 of 2.75 mm, representing a +13% improvement over the nnU-Net baseline.
Vessel Segmentation: Dice score of 0.897.
Stability: The method exhibited notably lower variance ( $\sigma = 0.033$ ) compared to baselines ( $\sigma = 0.097–0.138$ ).
Cross-Dataset Generalization: Without fine-tuning, the model was tested on CADA and SHINY-ICARUS datasets. It eliminated all catastrophic failures (Dice < 0.5) observed in baseline architectures, reducing the coefficient of variation by 52–70%. On SHINY-ICARUS, it improved the Dice score by 10.4% and reduced HD95 by 75.8%.

Significance and Claims
The paper claims that DINO-3DRA successfully leverages the dense structural priors of 2D foundation models to overcome the data scarcity limitations of 3D medical imaging. The authors assert that the key to this success is not merely the transfer of features, but the specific architectural mechanisms (Room-Lite mixers and calibrated fusion) that bridge the dimensional gap.

The method produces anatomically coherent predictions where aneurysms are recognized as pathological vessel dilations rather than isolated anomalies. This consistency provides a superior geometric starting point for downstream tasks such as surface meshing and hemodynamic modeling. The authors conclude that the framework offers improved robustness across heterogeneous protocols, prioritizing reliable localization and reduced failure rates over perfect volumetric boundary delineation in ambiguous cases.

Technical Summary: DINO-3DRA

More like this