Technical Summary: Projector Is All You Train

Problem Statement

Multimodal Large Language Models (MLLMs) typically require a two-stage training process to adapt a pretrained language model (LM) backbone to a new modality (e.g., 3D point clouds). First, a projector aligns features from a modality-specific encoder with the LM's embedding space. Second, both the projector and the LM backbone are jointly fine-tuned on multimodal instruction data. The authors question the necessity of this second stage: Is fine-tuning the LM backbone required to achieve strong multimodal performance, or is training the projector alone sufficient?

Existing literature suggests that joint training is standard, but the authors hypothesize that it may introduce unnecessary computational overhead and cause "catastrophic forgetting" or drift in the LM's pre-existing capabilities (language, vision, and spatial reasoning).

Methodology

The paper investigates this hypothesis through a controlled comparison of two training regimes on 3D MLLMs:

1. Projector-Only Training: Only the parameters of the projector (a 3-layer MLP) are optimized. The 3D point cloud encoder (Point-BERT) and the LM backbone (Qwen3.5-4B/9B or Llama-3.1-8B-Instruct) remain frozen.
2. Joint Training: Both the projector and Low-Rank Adaptation (LoRA) adapters attached to the LM backbone are optimized, while the encoder remains frozen.

Experimental Setup:

• Backbones: Three different LM backbones were tested to ensure generalizability.
• Dataset: The PointLLM-V2 dataset (Objaverse objects), mixing Stage 1 (brief captions) and Stage 2 (complex instructions) data.
• Compute Budget: Experiments were matched by wall-clock time (16 hours on a single A100 GPU) rather than optimizer steps. This is a critical design choice because projector-only training is approximately twice as fast as joint training due to the lack of backpropagation through the backbone.
• Evaluation:
  • 3D Understanding: Generative zero-shot 3D object classification (ModelNet40, Objaverse, OmniObject3D) and 3D object captioning.
  • LM Capability Drift: Standard benchmarks for language (MMLU-Pro, GSM8K), vision (MMMU-Pro, BabyVision), and spatial reasoning (ERQA, LingoQA) were run on the frozen backbones of projector-only models (representing the base LM) versus the LoRA-adapted backbones of joint-trained models.

Key Contributions and Findings

1. Projector-Only Training is Sufficient for 3D Understanding

The authors demonstrate that training only the projector yields 3D MLLMs with performance comparable to, and often exceeding, existing baseline models (such as PointLLM and PointLLM-V2) and jointly trained variants.

• Performance: On 3D classification and captioning benchmarks, projector-only models (P-) consistently achieved scores comparable to or higher than joint-trained models (J-) across all backbones.
• Efficiency: Because the backbone is frozen, projector-only training processes training samples at approximately twice the throughput of joint training. When evaluated against wall-clock time, projector-only models generally reach competitive or superior accuracy faster.
• Sample Efficiency: While joint training sees fewer samples in the same time window, the projector-only approach's higher sample throughput compensates, and in some cases (e.g., ModelNet40), projector-only training proves more sample-efficient.

2. Joint Training Causes Capability Drift

A significant finding is that joint training leads to a degradation in the LM backbone's existing capabilities, a phenomenon the authors attribute to the specific distribution of the 3D training data.

• Language Drift: Jointly trained backbones showed severe degradation in language benchmarks. For instance, the Llama-3.1-8B-Instruct backbone dropped from 37.32 to 11.21 on MMLU-Pro and collapsed to a 0.00 score on HumanEval (code generation), failing to follow instructions or generate coherent text.
• Vision and Spatial Drift: While some vision benchmarks (e.g., MMMU) showed score increases, the authors argue these are misleading artifacts of the evaluation mechanism (multiple-choice likelihood renormalization). On open-ended vision and spatial reasoning tasks (e.g., BabyVision, RefSpatialBench), jointly trained models exhibited instruction-following failures, such as providing descriptive captions instead of required coordinates.
• Conclusion: The drift is "guaranteed to be absent" in projector-only training because the backbone weights are never updated.

3. The Role of the Projector

The results suggest that the projector carries the majority of the burden for adapting the model to the new modality. The authors hypothesize that optimizing the projector is akin to "prompt engineering," where the model learns to elicit correct token distributions from the frozen backbone using the projected features, without needing to alter the backbone's internal representations.

Significance and Claims

The paper claims that adapting the language model backbone is unnecessary for building effective 3D MLLMs. The authors argue that the standard practice of joint fine-tuning is suboptimal because:

1. It does not yield superior 3D understanding performance compared to projector-only training.
2. It incurs a significant computational penalty (lower throughput).
3. It introduces a risk of degrading the model's general capabilities (language, vision, spatial reasoning).

The authors propose a more modular and scalable approach: a single, frozen LM backbone can serve as the foundation for multiple projectors and encoders across different modalities, with each projector trained independently. This avoids the "catastrophic forgetting" associated with joint fine-tuning and allows for more efficient, specialized MLLM development. The paper concludes that for the specific case of 3D point clouds, "Projector Is All You Train."