Single-Slice-to-3D Reconstruction in Medical Imaging and Natural Objects: A Comparative Benchmark with SAM 3D

This paper benchmarks five state-of-the-art image-to-3D foundation models on medical and natural datasets, revealing that while all struggle with severe depth ambiguity in single-slice reconstruction, SAM3D best preserves topological similarity to medical shapes, ultimately demonstrating that reliable medical 3D inference requires domain-specific adaptation beyond current zero-shot capabilities.

The Gist

Imagine you are trying to build a detailed, 3D model of a house, but you only have one single photograph of the front door.

That is essentially the challenge this paper tackles. In the medical world, doctors often have 2D slices (like a single slice of bread from a loaf) from CT or MRI scans. They want to turn that single flat image into a full 3D volume to see tumors, organs, or bones in their full depth.

Recently, powerful AI models (called "Foundation Models") have been trained on millions of photos of everyday objects—cats, cars, chairs—to guess what the 3D object looks like behind the camera. The big question the authors asked was: "Can these AI models, which are experts at guessing 3D shapes from natural photos, do the same job for medical scans?"

Here is the breakdown of their findings, using some everyday analogies:

1. The "Flatland" Problem

The researchers tested five of the smartest AI models available (including one called SAM3D) on medical data.

• The Analogy: Imagine trying to guess the shape of a complex, crumpled piece of paper just by looking at its shadow on a wall.
• The Reality: Natural photos have shadows, textures, and objects blocking each other, which give our brains (and AI) clues about depth. Medical slices are different. They are often just flat, uniform gray shapes with no shadows or depth cues.
• The Result: Because the AI lacks these depth clues, it gets confused. Instead of building a deep, 3D "loaf of bread," the AI tends to build a very thin, flat "sheet of paper." It fails to guess how deep the object actually is.

2. The "Good News, Bad News" Report Card

The paper graded the models on two different things:

• The "Volume" Grade (Bad): If you ask, "How much of the 3D space did the AI fill correctly?" the score was terrible for everyone. The models were so bad at guessing depth that they barely overlapped with the real 3D shape. It's like trying to fill a swimming pool with a single drop of water.
• The "Shape" Grade (Better): If you ask, "Does the AI at least get the general outline right?" one model, SAM3D, did the best.
  • The Analogy: Even though the AI couldn't guess the depth of a tumor, SAM3D was better at guessing the width and height. It was like a sculptor who couldn't carve the depth of a statue but managed to get the silhouette (the shadow shape) mostly correct.

3. The "Smooth vs. Bumpy" Challenge

The researchers tested two types of medical targets:

• Anatomy (The "Smooth" Stuff): Things like spines or airways. These are relatively smooth and predictable.
  • Result: The AI did okay here. It's like guessing the shape of a smooth, round apple.
• Pathology (The "Bumpy" Stuff): Things like tumors or irregular lesions. These are jagged, weird, and non-convex (they have holes and bumps).
  • Result: The AI struggled immensely.
  • The Analogy: Trying to guess the shape of a tumor from one slice is like trying to guess the shape of a crumpled ball of tinfoil just by looking at one side of it. The AI's training on "smooth" natural objects (like cars or cups) didn't help it handle the messy, irregular shapes of diseases.

4. The "Natural vs. Medical" Gap

When the researchers tested these same AIs on photos of natural objects (like household items or animals), the models performed much better.

• The Takeaway: These AIs are like chefs who are masters at cooking Italian food (natural images) but have never tried to cook Thai food (medical images). They can guess the shape of a cat or a chair perfectly, but when you hand them a medical scan, they are out of their element.

The Bottom Line

The paper concludes that while these fancy AI models are impressive, we cannot just use them "out of the box" for medical 3D reconstruction.

• Why? Because a single 2D medical slice doesn't have enough information for the AI to guess the 3D depth.
• What's needed? We need to teach these models specifically about human anatomy. We need to give them "rules" about how organs and tumors usually look, rather than letting them guess based on photos of cats and cars.

In short: The AI is a talented artist who can draw a 3D car from a photo, but if you hand it a single slice of a brain scan, it will likely draw a flat, 2D blob. To fix this, we need to train the artist specifically on medical textbooks, not just art galleries.

Technical Summary

1. Problem Statement

The paper addresses the critical challenge of zero-shot single-slice-to-3D reconstruction in medical imaging. While 3D imaging (CT/MRI) is essential for clinical diagnosis and treatment planning, it is often costly and time-consuming. The authors investigate whether image-to-3D foundation models, trained on large-scale natural image datasets, can generalize to medical data to infer volumetric structures from a single 2D slice.

Core Hypothesis & Challenge:
The central problem is the domain gap between natural images and medical slices.

• Natural Images: Contain rich depth cues (shading gradients, occlusion boundaries, multi-object spatial relationships) that foundation models learn to exploit.
• Medical Slices: Are inherently planar, often have uniform depth, lack texture, and possess minimal shading cues.
  The authors hypothesize that models trained on natural data may suffer from severe depth ambiguity when applied to medical slices, leading to volumetric reconstruction failures despite potentially capturing global shape topology.

2. Methodology

The study employs a rigorous zero-shot evaluation framework to benchmark five state-of-the-art (SOTA) image-to-3D foundation models without any fine-tuning or domain-specific adaptation.

Datasets

The evaluation spans 8 datasets (6 medical, 2 natural):

• Medical Datasets:
  • Anatomical: AeroPath (Lungs/Airways), BTCV (Abdominal organs), Duke C-Spine (Spinal Cord).
  • Pathological: MSD Lung, MSD Brain, MSD Liver (Tumor segmentations).
  • Modality: CT and MRI.
• Natural Datasets:
  • Google Scanned Objects (GSO): Household items.
  • Animal3D: 40 mammal species.

Experimental Pipeline

1. Input Generation: From volumetric NIfTI scans, a midpoint slice is extracted along either the coronal or axial plane. A corresponding binary segmentation mask is applied to isolate the structure, creating a single masked 2D input image.
2. Ground Truth: Surface points are extracted from the full 3D segmentation mask using morphological erosion to define the boundary, scaled to physical units to create a ground-truth point cloud.
3. Reconstruction: The masked 2D slice is fed into the five models to generate a predicted 3D point cloud.
4. Evaluation: Predictions are aligned to ground truth via Iterative Closest Point (ICP) and evaluated using five metrics.

Benchmarked Models

1. SAM3D: Segment Anything Model adapted for 3D.
2. Hunyuan3D-2.1: Diffusion transformer for shape and texture.
3. Direct3D: 3D-native latent diffusion transformer.
4. Hi3DGen: Focuses on fine geometry via normal-bridging.
5. TripoSG: Large-scale rectified-flow generative modeling.

Metrics

• Voxel-Based (Volumetric Overlap): F1 Score, Voxel IoU, Voxel Dice (measures local volumetric accuracy).
• Point Cloud Distance (Global Shape): Chamfer Distance (CD), Earth Mover's Distance (EMD) (measures global topological similarity).

3. Key Contributions

• First Comprehensive Benchmark: The first systematic comparison of general-purpose image-to-3D foundation models on diverse medical datasets (anatomical vs. pathological, CT vs. MRI).
• Quantification of Domain Gap: Provides empirical evidence that geometric priors learned from natural images do not transfer effectively to medical slices, resulting in a "volumetric ceiling."
• Differentiation of Metrics: Demonstrates that while all models fail at local volumetric reconstruction (low voxel overlap), they differ significantly in capturing global shape topology (distance metrics).
• Pathological vs. Anatomical Analysis: Highlights that pathological structures (tumors) with irregular, non-convex morphologies pose a compounded difficulty compared to smoother anatomical structures.

4. Key Results

A. Volumetric Failure (Voxel Metrics)

• Uniformly Low Scores: All five models exhibited extremely low voxel-based scores across medical datasets (F1 < 0.10, Voxel IoU < 0.16, Voxel Dice < 0.26).
• Depth Ambiguity: The models consistently produced near-planar reconstructions with minimal volumetric extent. This is attributed to the lack of depth cues in 2D medical slices.
• Model Ranking: SAM3D and Hi3DGen slightly outperformed others, but the margins were small. TripoSG and Hunyuan3D-2.1 often failed to generate any meaningful volumetric overlap (near-zero F1) on complex datasets.
• Plane Dependency: Reconstruction quality varied significantly between coronal and axial inputs, indicating that the informativeness of the silhouette is plane-dependent.

B. Global Shape Fidelity (Distance Metrics)

• SAM3D Superiority: Despite volumetric failures, SAM3D consistently achieved the lowest Chamfer Distance (CD) and Earth Mover's Distance (EMD) across medical datasets.
  • Example: On the Duke C-Spine dataset, SAM3D's CD (0.153) was roughly half that of the next best model (Direct3D, 0.236).
• Interpretation: While models cannot recover accurate volume, SAM3D better preserves the global shape distribution and coarse morphological features.

C. Natural vs. Medical Performance

• Domain Gap: Natural datasets (GSO, Animal3D) yielded significantly better scores across all models.
  • GSO: SAM3D achieved Voxel IoU ~0.18 (vs. ~0.14 max in medical).
  • Distance: Natural datasets had CD/EMD in the 0.15–0.40 range, whereas medical datasets were 0.30–0.60.
• Conclusion: Foundation models operate effectively within their training distribution (natural objects) but degrade significantly when applied to the planar, texture-poor nature of medical slices.

D. Anatomical vs. Pathological Difficulty

• Pathological Structures are Harder: Tumors (MSD Lung, Brain, Liver) yielded significantly worse reconstruction quality than whole organs (Duke C-Spine, BTCV).
• Reasoning: Pathological structures have irregular, non-convex boundaries that deviate further from the "compact, smooth" shape priors learned from natural objects.
• Best Case: The Duke C-Spine (simple, elongated geometry) was the most amenable dataset, achieving the highest medical scores.
• Worst Case: MSD Lung (irregular tumors) was the most challenging, with some models failing to produce volumetric overlap entirely.

5. Significance and Implications

• Limits of Zero-Shot Inference: The study establishes a hard ceiling for zero-shot single-slice 3D reconstruction in medicine. Current foundation models cannot reliably infer volume from a single medical slice due to fundamental depth ambiguity.
• Need for Adaptation: Reliable medical 3D reconstruction requires domain-specific adaptation (training on medical data) and the incorporation of anatomical constraints to guide the geometry.
• Strategic Recommendations:
  • For geometrically simple structures (e.g., spine), zero-shot inference might offer coarse shape approximations.
  • For complex pathological structures, zero-shot approaches are insufficient; multi-view strategies or hybrid models combining foundation priors with anatomical knowledge are necessary.
• Metric Selection: The paper argues that relying solely on voxel overlap is misleading for foundation models; distance metrics (CD/EMD) are more informative for assessing global shape preservation, even when volumetric accuracy is low.

In summary, while foundation models like SAM3D show promise in capturing global topology, they currently fail to solve the depth ambiguity inherent in single-slice medical imaging, necessitating targeted domain adaptation for clinical viability.