An Adapter-free Fine-tuning Approach for Tuning 3D Foundation Models

1. Problem Statement

The paper addresses the challenge of adapting 3D point cloud foundation models (e.g., Point-MAE, Point-BERT) to downstream tasks when training data is scarce (low-data regimes).

• Full Fine-Tuning (FFT): While effective with abundant data, FFT leads to severe overfitting and representation drift in low-data scenarios. As shown in the paper's Figure 1, FFT causes the model's internal representations to diverge significantly from the pre-trained state, particularly in deeper layers, destroying the model's generalization capabilities.
• Parameter-Efficient Fine-Tuning (PEFT): Existing methods (e.g., Adapters, Prompts, LoRA) attempt to mitigate overfitting by freezing the backbone and training only a small number of new parameters. However, the paper argues that:
  1. PEFT yields only marginal performance gains (often ~1%) in 3D low-data settings compared to 2D/NLP domains.
  2. PEFT introduces additional trainable parameters (e.g., IDPT adds 1.7M parameters), which increases memory usage and inference latency. This makes PEFT unsuitable for resource-constrained environments like mobile devices.

2. Methodology: Momentum-Consistency Fine-Tuning (MCFT)

The authors propose MCFT, an adapter-free approach that bridges the gap between Full Fine-Tuning and PEFT. It fine-tunes the model without adding any new parameters to the encoder, maintaining the original model's parameter count and inference efficiency.

Core Mechanism

• Selective Fine-Tuning: Instead of freezing the entire encoder (PEFT) or updating all weights (FFT), MCFT selectively fine-tunes a specific portion of the pre-trained encoder layers.
• Momentum-Based Consistency (Self-Distillation): To prevent overfitting during selective tuning, MCFT enforces a consistency constraint between the trainable "student" encoder and a "teacher" encoder.
  • Teacher Model: The parameters ($\theta_t$) are an Exponential Moving Average (EMA) of the student model's parameters ($\theta_s$).
  • Update Rule: $\theta_t = \alpha \cdot \theta_t + (1 - \alpha) \cdot \theta_s$, where $\alpha$ is a smoothing factor (e.g., 0.999).
  • Alignment Loss: The model minimizes the cosine distance between the [CLS] token embeddings of the student ($E_s$) and teacher ($E_t$):
    $$L_{align} = 1 - \frac{E_s^T \cdot E_t}{\|E_s\|\|E_t\|}$$
  • Total Loss: $L = L_{align} + \lambda \cdot L_{sup}$, where $L_{sup}$ is the standard cross-entropy loss for the task head.
• No Extra Parameters: The only added component is a standard task-specific prediction head (e.g., a classifier layer), which is required regardless of the fine-tuning method.

Variants

The authors extend MCFT with two variants to further enhance performance and efficiency:

1. Semi-Supervised MCFT (MCFT-SSL): Leverages abundant unlabeled data alongside limited labeled data. It adopts the AllMatch framework, combining entropy minimization with pseudo-labeling and contrastive learning to utilize unlabeled samples effectively.
2. Pruned MCFT: A structural pruning variant that removes less important encoder layers based on a salience scoring function (gradient-based importance). This reduces computational overhead (FLOPs) and model size while maintaining high accuracy.

3. Key Contributions

1. Adapter-Free Architecture: MCFT is the first approach to fine-tune 3D foundation models without introducing adapter modules or prompts, thereby preserving the original model's inference speed and memory footprint.
2. Momentum-Consistency Regularization: The novel use of EMA-based self-distillation allows the model to adapt to new tasks while preserving the generalizable features learned during pre-training, effectively solving the overfitting problem in low-data regimes.
3. Efficiency and Scalability: By avoiding extra parameters, MCFT is uniquely suited for deployment on edge devices. The pruning variant further optimizes this for extreme resource constraints.
4. Semi-Supervised Extension: Demonstrates that unlabeled data can significantly boost few-shot performance in 3D foundation models when combined with the MCFT framework.

4. Experimental Results

The authors evaluated MCFT on ModelNet40, ScanObjectNN, and ShapeNetPart across few-shot, full-few-shot, and fully-supervised settings.

• Few-Shot Performance (Object Recognition):
  • In the 5-shot setting (most data-scarce), MCFT achieved a 3.30% improvement over state-of-the-art (SOTA) methods (e.g., IDPT, DAPT, PointGST).
  • On ModelNet40 with Point-MAE, MCFT reached 78.36% accuracy vs. 75.06% for the next best method.
  • In n-way few-shot tasks, MCFT consistently outperformed existing PEFT methods.
• Semi-Supervised Gains:
  • The MCFT-SSL variant achieved 84.49% accuracy in the 5-shot setting on ModelNet40, a 6.13% improvement over the standard supervised MCFT and a massive 9.43% gain over the previous SOTA.
• Part Segmentation:
  • On ShapeNetPart, MCFT improved mean Intersection over Union (mIoU) by 2.6% (class-level) and 3.2% (instance-level) over SOTA.
• Efficiency & Inference:
  • Parameters: MCFT maintains the same parameter count as the base model (e.g., 22.7M for Point-MAE), whereas PEFT methods like IDPT increase this to 23.8M.
  • Throughput: MCFT maintains throughput comparable to the base model (~321 frames/s). The Pruned MCFT variant reduced parameters to 20.9M and FLOPs to 4.4G, increasing throughput to 350.75 frames/s with only a minor accuracy drop (83.5% vs 87.2%).
  • Latency: Unlike IDPT, which increased inference latency by 50%, MCFT introduces no additional inference overhead.

5. Significance

This paper presents a paradigm shift in adapting 3D foundation models. It challenges the prevailing assumption that parameter-efficient fine-tuning (via adapters) is the only solution for low-data scenarios. By proving that adapter-free methods with momentum-based consistency can outperform parameter-heavy PEFT methods while being more computationally efficient, MCFT offers a practical, deployable solution for real-world 3D applications (e.g., robotics, autonomous driving) where data is limited and hardware resources are constrained. The work establishes a new baseline for balancing model adaptability, generalization, and inference efficiency.