💻 computer science

SAM+D: Parameter-Efficient Dimensional Lifting of SAM-Family Models via Depth-Routed LoRA and Depth Shifting

This paper introduces SAM+D, a parameter-efficient framework that adapts 2D SAM-family models to 3D and 4D volumetric segmentation tasks by integrating Depth-Routed LoRA and Depth Shift Modules, achieving competitive performance with minimal trainable parameters while preserving pre-trained weights.

Original authors: Yu Song, Hao Sun, Shiyu Teng, Ikuko Nishikawa, Yen-wei Chen

Published 2026-08-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yu Song, Hao Sun, Shiyu Teng, Ikuko Nishikawa, Yen-wei Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to see the world. For years, the best robots have been experts at looking at flat, two-dimensional pictures, like a photograph of a cat or a map of a city. They are incredibly good at this, trained on billions of images to recognize shapes and boundaries instantly. But the real world isn't flat; it's deep. It has thickness, layers, and time passing. When doctors look at a 3D scan of a human body or biologists watch cells move under a microscope over time, they aren't looking at a single flat picture; they are looking at a stack of them, a volume.

The problem is that these flat-image experts get confused when you hand them a 3D object. If you ask a robot trained only on photos to understand a 3D tumor, it might try to look at each slice of the tumor one by one, like flipping through a book without realizing the pages are connected. It misses the big picture of how the object exists in space. To fix this, scientists usually have to build entirely new, massive robots from scratch, which takes huge amounts of data and computing power. But what if we could just give our existing flat-image expert a pair of 3D glasses and a little bit of training, without rebuilding its whole brain? That is the big question this paper tackles: How do we take a model that knows 2D and teach it to understand 3D (and even 4D, which is 3D plus time) without starting over?

Enter SAM+D, a clever new framework that acts like a magical adapter for these "flat" vision models. The authors, Yu Song and his team, wanted to take the famous Segment Anything Model (SAM) and its video-savvy cousin, SAM2, and lift them up by one dimension. Think of SAM as a master painter who can perfectly trace the outline of a cat in a photo. SAM+D is the tool that lets that same painter suddenly start painting a 3D sculpture of a cat, or even a movie of a cat running, without needing to relearn how to hold the brush.

The secret sauce of SAM+D is that it doesn't touch the model's heavy, pre-trained brain. Instead, it slips in two tiny, lightweight "gadgets" into every layer of the model's thinking process. These gadgets are so small that the team only had to train about 2.8% of the model's parameters for the 3D version and 3.7% for the 4D version. It's like upgrading a car's engine by adding two small turbochargers rather than replacing the whole engine.

The first gadget is called Depth-Routed LoRA (DRLoRA). Imagine you are looking at a loaf of bread. The crust on the top slice is different from the soft middle slice, which is different from the bottom crust. A standard AI might try to treat every slice the same way. DRLoRA is like a smart guide who knows exactly which slice you are looking at. It has a team of tiny experts (called "LoRA experts") and a router that says, "Hey, we are looking at the crust, so Expert A should handle this," or "We are in the middle, so Expert B should take over." This allows the model to adapt its understanding based on the depth of the slice, capturing the unique features of different parts of the volume.

The second gadget is the Depth Shift Module (DSM). This is even cooler because it costs absolutely nothing extra in terms of computer power. Imagine a line of people passing a note. Usually, Person 1 only talks to Person 2, and Person 2 only talks to Person 3. But in a 3D volume, the slice above and the slice below are neighbors too. The DSM acts like a magical whisperer that lets a slice "borrow" a few features from the slice right above it and the slice right below it before it does its own thinking. It shifts a small fraction of the information sideways between slices. This happens instantly, with zero extra math calculations, allowing the model to understand that the object continues smoothly from one slice to the next.

The team tested this magic adapter in two very different worlds. First, they used it on medical CT scans (3D images of kidneys, pancreases, livers, and colons). They gave the model just a single dot (a "point prompt") on the image to say "start here," and SAM+D successfully traced the entire 3D organ. It performed as well as, or better than, other methods that required much more training and data, all while using a tiny fraction of the computing power.

Second, they took it to the microscopic world of cell tracking (4D, which is 3D space plus time). Here, they watched cells divide and move over time. By using the same adapter on SAM2 (the video version), they could track cells through time and space, handling cell divisions and new cells appearing, all with the same single-point prompt. The results showed that the model could keep track of cells much better than just looking at each frame separately.

The paper is very clear about what this does not do. It doesn't claim to be a perfect, universal solution for every possible 3D task yet, and it notes that it hasn't been tested on the newest "text-prompted" version of SAM (SAM3) just yet. However, the results are solid: by using these two lightweight modules, the team successfully lifted 2D models into 3D and 4D, proving that you don't need to throw away your old, smart models to understand the deep, moving world. You just need to give them the right glasses.

Technical Summary: SAM+D

Problem Statement
Existing foundation models for vision, such as the Segment Anything Model (SAM) and its video extension SAM2, are natively designed for 2D images or 2D+T (video) sequences. While these models demonstrate strong zero-shot capabilities in 2D domains, they lack genuine volumetric understanding required for 3D medical imaging (e.g., CT scans) and 4D spatiotemporal tasks (e.g., 3D time-lapse microscopy). Current approaches to adapt these models to volumetric data face significant limitations:

Slice-by-slice processing: Treating 3D volumes as independent 2D slices ignores inter-slice context, failing to distinguish overlapping structures.
Heavy retraining: Fully 3D architectures or methods that rebuild SAM/SAM2 with 3D components often require massive annotated datasets and discard valuable 2D pre-trained priors.
Inefficient adaptation: Existing adapter methods often apply uniform transformations across the volume, ignoring that different anatomical regions (e.g., boundaries vs. interiors) require distinct adaptations. Furthermore, modeling cross-slice dependencies typically introduces 3D convolutions or attention mechanisms that add substantial parameters and computational cost.
Lack of unification: Most methods target a single base model (either SAM or SAM2) and a single dimensionality, lacking a unified architecture to lift both to 3D and 4D respectively.

Methodology: SAM+D
The authors propose SAM+D, a parameter-efficient framework that "lifts" frozen 2D foundation models by one spatial dimension to handle 3D and 4D data. The core innovation lies in inserting two lightweight, model-agnostic modules into every frozen transformer block of the pre-trained encoder, keeping the vast majority of parameters frozen.

Depth-Routed LoRA (DRLoRA):
- Mechanism: Instead of a single LoRA adapter applied uniformly, DRLoRA maintains multiple parallel LoRA experts ( $E$ ) per transformer block.
- Routing: A lightweight router maps the normalized depth position ( $z = d/D_s$ ) of a slice to a softmax distribution over these experts.
- Function: This allows the model to learn depth-dependent adaptations. For instance, features near organ boundaries can receive different low-rank updates compared to homogeneous interior regions. The router is conditioned solely on depth position, eliminating the need for auxiliary load-balancing losses required by content-based Mixture-of-Experts (MoE) methods.
- Injection: DRLoRA is applied to the Query ( $Q$ ) and Value ( $V$ ) projections of self-attention layers, leaving Keys ( $K$ ) frozen to preserve the pre-trained feature space.
Depth Shift Module (DSM):
- Mechanism: Inspired by the Temporal Shift Module (TSM) for video, DSM is repurposed to shift feature channels along the spatial depth axis.
- Operation: Before self-attention, a fraction ( $\alpha$ ) of channels in slice $i$ are shifted forward from slice $i-1$ and backward from slice $i+1$ . The remaining channels remain unchanged.
- Efficiency: This operation involves zero learnable parameters and zero multiply-accumulate (MAC) operations. It incurs only a negligible memory traffic overhead (measured at ~1.7% of encoder block latency).
- Purpose: It enables cross-slice feature exchange, providing volume-level context without the cost of 3D convolutions or cross-slice attention.

Implementation Settings

3D Setting (SAM $\to$ 3D): DRLoRA and DSM are inserted into the frozen ViT-B encoder of SAM. The model processes a resampled volume of slices in a single forward pass. A learnable depth embedding is added to prompt embeddings to modulate predictions based on slice proximity to the prompt.
4D Setting (SAM2 $\to$ 3D+T): The same modules are inserted into the frozen Hiera encoder of SAM2. Temporal consistency is handled by SAM2's native memory architecture, while DRLoRA/DSM handle spatial depth. LoRA is additionally injected into the memory attention module to adapt cross-attention for volumetric data.

Key Contributions

Unified Framework: SAM+D is the first method to lift both SAM (2D $\to$ 3D) and SAM2 (2D+T $\to$ 3D+T) using the same lightweight components without redesigning the base architecture.
Novel Modules: Introduction of DRLoRA for spatially adaptive, depth-conditioned low-rank updates and DSM for zero-cost cross-slice feature exchange.
Parameter Efficiency: The method tunes only ~2.8% of parameters for SAM and ~3.7% for SAM2, significantly fewer than existing adapter-based or fully fine-tuned approaches.
Generalization: The approach demonstrates generalization across different SAM-family architectures, target dimensionalities (3D, 4D), and domains (medical CT, bio-scene cell tracking).

Experimental Results
The authors evaluated SAM+D on five benchmarks under a single-point prompt setting:

3D Medical Segmentation: Evaluated on four CT tumor segmentation datasets (KiTS, Pancreas, LiTS, Colon). SAM+D achieved competitive or superior results compared to fully supervised 3D methods (e.g., nnU-Net) and existing SAM-based adapters (e.g., 3DSAM-adapter, Med-SA), despite using far fewer trainable parameters.
4D Spatiotemporal Segmentation: Evaluated on the Fluo-N3DH-SIM+ dataset from the Cell Tracking Challenge. SAM+D outperformed slice-based SAM2 approaches and other tracking methods, demonstrating that temporal propagation can refine segmentation quality beyond the initial detector (Cellpose).
Ablation Studies: Confirmed that both DRLoRA and DSM provide complementary gains. Depth-based routing outperformed content-based MoE routers, and DSM provided better cross-slice context than parameter-rich 3D attention or convolution adapters.

Significance and Claims
The paper claims that SAM+D provides a practical and generalizable recipe for extending pre-trained 2D foundation models to higher-dimensional tasks. By treating the added dimension (depth) as a specific axis for routing and shifting, the method achieves volumetric understanding without discarding 2D priors or incurring heavy computational costs. The authors note that while the current work focuses on SAM and SAM2, the principle of "dimensional lifting" via parameter-efficient adaptation could naturally extend to other dimensions (e.g., lifting 2D image models to video by treating time as the added axis). The work is modest in scope, acknowledging that it has not yet evaluated SAM3 (text-prompted) and that fully automatic 4D tracking still requires external detectors for cell division events.

Technical Summary: SAM+D

More like this