PlaneCycle: Training-Free 2D-to-3D Lifting of Foundation Models Without Adapters

The paper introduces PlaneCycle, a training-free and adapter-free method that lifts pretrained 2D foundation models to 3D by cyclically distributing spatial aggregation across orthogonal planes, enabling strong 3D performance without architectural modifications or additional parameters.

The Gist

Imagine you have a brilliant, world-class 2D artist (a computer model) who has spent years learning to paint beautiful landscapes, portraits, and objects from flat pictures. This artist is incredibly talented, but they have never seen a real 3D object like a cube or a human body from all angles; they only know how to look at a single flat slice of it.

Now, imagine you want this artist to understand 3D medical scans (like CT scans of lungs or brains). Usually, to teach them, you'd have to:

1. Fire them and hire a new 3D artist (train a new model from scratch).
2. Give them expensive training wheels (add "adapters" or extra layers to their brain).
3. Make them relearn everything (retrain the model), which takes massive amounts of time and energy.

PlaneCycle is the paper's solution. It's a clever, "training-free" trick that lets your existing 2D artist understand 3D objects without firing them, without adding training wheels, and without making them relearn anything.

Here is how it works, using a simple analogy:

The "Rotating Table" Analogy

Think of a 3D object (like a loaf of bread) as a stack of many 2D slices.

• The Old Way (Slice-by-Slice): You take the 2D artist, show them one slice of bread, ask for a description, then show them the next slice, and so on. At the end, you try to glue the descriptions together. The problem? The artist never sees how the slices connect. They don't understand the "loaf" as a whole.
• The PlaneCycle Way: Instead of just looking at the slices one by one, you put the loaf of bread on a magic rotating table.
  1. First, you look from the Top (Axial Plane): You show the artist the top-down view of the whole loaf.
  2. Then, you rotate it 90 degrees (Coronal Plane): Now you show them the front view.
  3. Then, you rotate it again (Sagittal Plane): Now you show them the side view.
  4. You keep cycling: You rotate the loaf back to the top, then the front, then the side, over and over again as the data moves through the artist's brain.

By constantly rotating the view, the artist (the model) starts to naturally understand how the top, front, and side connect. They realize, "Oh, this bump on the top slice is the same bump on the side slice!" They build a 3D understanding just by looking at the same object from different angles repeatedly.

Why is this a Big Deal?

1. No Extra Weight: The artist doesn't need to grow a new brain part. The method adds zero new parameters (no extra memory or complexity). It's like giving the artist a new pair of glasses that rotate automatically, rather than giving them a new brain.
2. Instant 3D Superpowers: Even if you don't train the model at all (just use the "frozen" artist), this rotating trick makes the model suddenly good at 3D tasks. It's like the artist suddenly realizing, "Wait, I can see the whole loaf!" without ever taking a 3D class.
3. Saves Energy: Training 3D models from scratch is like trying to build a skyscraper from scratch every time you need a house. It takes huge amounts of electricity and time. PlaneCycle is like taking an existing house and just rearranging the furniture to make it a skyscraper. It's fast, cheap, and eco-friendly.
4. Works with Anything: Whether your "artist" is a classic CNN (the old school style) or a modern ViT (the new Transformer style), this rotating table trick works for both.

The Results

The researchers tested this on medical data (like finding tumors in lungs or segmenting organs).

• Without any training: Their method was already better than other methods that tried to force 2D models into 3D.
• With a little training: It matched or beat models that were specifically built for 3D from the ground up.

The Bottom Line

PlaneCycle is a "magic lens" that lets us reuse the massive, powerful 2D AI models we already have (like DINOv3) to solve 3D problems. It proves that you don't need to reinvent the wheel or spend millions of dollars training new models to get 3D intelligence; you just need to look at the data from the right angles, over and over again.

It's a simple, elegant, and free upgrade that turns flat 2D experts into 3D masters.

Technical Summary

1. Problem Statement

Large-scale 2D foundation models (e.g., DINOv3, ViT) possess robust, transferable representations but are inherently limited to 2D data. Extending these models to 3D volumetric data (such as CT, MRI, or electron microscopy) presents significant challenges:

• Current Approaches:
  • Slice-wise 2D: Processes slices independently and aggregates later. This is computationally efficient but fails to capture cross-slice dependencies.
  • Full 3D Conversion: Converts 2D kernels to 3D or uses adapters. This often requires architectural redesign, introduces new parameters, and necessitates extensive retraining to unlock 3D capabilities.
  • 3D Pretraining: Requires massive 3D datasets, which are scarce and computationally expensive compared to 2D natural image pretraining.
• The Core Question: Can the 3D capability of pretrained 2D foundation models be unlocked without modifying the architecture, adding parameters (adapters), or retraining the backbone?

2. Methodology: PlaneCycle

The authors propose PlaneCycle, a parameter-free, training-free operator that lifts 2D foundation models to 3D while preserving their pretrained inductive biases.

Core Mechanism

PlaneCycle operates by cyclically distributing spatial aggregation across three orthogonal planes (HW - Axial, DW - Coronal, DH - Sagittal) throughout the network depth.

1. Input Reshaping: Given a 3D feature map $X \in \mathbb{R}^{D \times H \times W \times C}$, the volume is reshaped into $P$ slices along a chosen axis (e.g., $D$ slices of size $H \times W$).
2. Token Processing:
   • The 3D volume is flattened into a sequence of tokens for the selected plane.
   • Global tokens (e.g., CLS and register tokens in ViTs) are adapted via adaptive average pooling to match the plane's token count.
   • These tokens are concatenated and passed through the frozen pretrained 2D layer ($F_\theta$).
3. Cyclic Aggregation:
   • The process is repeated across layers in a cycle: $HW \to DW \to DH \to HW$.
   • This allows the model to progressively fuse 3D information by viewing the volume from different orthogonal perspectives without changing the underlying weights.
4. Global Token Handling: To handle token length mismatches when switching planes, the method uses AdaptiveAvgPool1d to average and replicate global tokens, ensuring distribution consistency without learnable parameters.

Key Technical Features

• Architecture-Agnostic: Works with both CNN and Transformer (ViT) backbones.
• Zero Parameters: Introduces no additional weights; it only modifies the data flow (reshaping and re-indexing).
• Computational Efficiency:
  • 2D Baseline: $O(D(HW)^2)$ complexity (independent slice processing).
  • Full 3D Baseline: $O((DHW)^2)$ complexity (quadratic scaling with volume).
  • PlaneCycle: Maintains $O(D(HW)^2)$ complexity (similar to 2D slicing) because it processes plane-wise 2D tokens, offering a $D$-fold reduction in attention cost compared to full 3D flattening.

3. Key Contributions

1. Training-Free 2D-to-3D Lifting: Demonstrates that 3D capability can emerge from 2D foundation models without retraining or adapters.
2. Intrinsic 3D Fusion: Shows that simply cycling through orthogonal planes allows the model to learn coherent 3D representations immediately upon loading pretrained weights.
3. Unified Framework: Provides a single operator compatible with both CNNs and ViTs, addressing the gap left by previous methods (like ACS convolution) that were restricted to specific architectures.
4. Sustainability: Offers a green AI solution by reusing the massive computational investment (e.g., 9M GPU hours for DINOv3) of 2D models for 3D tasks, avoiding the need for new 3D pretraining.

4. Experimental Results

The method was evaluated on six 3D classification and three 3D segmentation benchmarks using DINOv3 backbones (ViT-S, ViT-B, ViT-L).

Classification Performance

• Zero-Training (Linear Probing): PlaneCycle significantly outperformed both slice-wise 2D baselines and full 3D baselines.
  • Example: Using ViT-B/16, PlaneCycle (PCg) achieved an average 86.7% AUC and 80.8% ACC, surpassing the fully fine-tuned 3D ResNet-50 (78.0% ACC) and the 3D baseline (74.8% ACC) by large margins.
  • It even outperformed ViViT (a video-based 3D transformer) by up to 2.6 AUC points, suggesting better preservation of pretrained semantics.
• Full Fine-Tuning: After fine-tuning, PlaneCycle matched or slightly exceeded standard 3D architectures while maintaining the computational efficiency of 2D models.

Segmentation Performance

• Zero-Training (FeatDice): PlaneCycle produced significantly more coherent 3D feature maps than 2D or 3D baselines, achieving higher Dice scores on lesion masks without any optimization.
• Full Fine-Tuning: PlaneCycle surpassed 3D flattening methods by up to 2.6 Dice points in some settings, while requiring >2x less training time and significantly less GPU memory.

5. Significance and Conclusion

• Paradigm Shift: PlaneCycle challenges the notion that 3D tasks require dedicated 3D pretraining or heavy architectural modifications. It proves that 3D understanding is latent within 2D foundation models and can be unlocked via simple geometric reorganization.
• Practicality: It is a "plug-and-play" solution. Researchers can take any pretrained 2D model and immediately apply it to 3D medical imaging tasks with high performance and low computational cost.
• Scalability: The method scales efficiently, making it feasible to apply massive models (e.g., DINOv3-7B) to 3D tasks, a scale previously unattainable due to the quadratic complexity of full 3D attention.

In summary, PlaneCycle is a highly efficient, parameter-free operator that bridges the gap between 2D foundation models and 3D volumetric data, offering a sustainable and high-performance alternative to traditional 3D deep learning approaches.