PnP-U3D: Plug-and-Play 3D Framework Bridging Autoregression and Diffusion for Unified Understanding and Generation

Technical Summary: PnP-U3D

Problem Statement
While large multimodal models have successfully unified 2D understanding and generation, extending this paradigm to 3D remains underexplored. Existing attempts to unify 3D tasks under a single autoregressive (AR) paradigm, such as ShapeLLM-Omni, face significant limitations. These approaches require the discretization of continuous 3D signals into tokens, leading to information loss due to quantization, prohibitive training costs, and a failure to effectively leverage pretrained Large Language Model (LLM) priors. Furthermore, purely AR frameworks struggle to maintain the high fidelity required for 3D generation while simultaneously supporting understanding tasks. The core challenge lies in bridging the asymmetry between the rich priors of large-scale language models for understanding and the data-scarce nature of 3D generation, without forcing a unified paradigm that compromises the inherent strengths of specialized models.

Methodology
The authors propose PnP-U3D, a Plug-and-Play framework that decouples 3D understanding and generation into their respective optimal paradigms while enabling effective cross-modal interaction.

1. Dual-Paradigm Architecture:

   • 3D Understanding (Autoregression): The framework adopts an autoregressive next-token prediction paradigm for understanding. It extends a pretrained 2D Vision-Language Model (VLM), specifically QwenVL-2.5, to process 3D shapes. A 3D shape $x$ is encoded into continuous latents $u$ using a pretrained 3D VAE (Hunyuan3D 2.1). A lightweight MLP projector maps these latents into the VLM's token embedding space. The VLM is kept frozen, and only the projector is trained to predict text descriptions $t$.
   • 3D Generation (Diffusion): For generation, the framework utilizes a continuous diffusion paradigm. Given a text prompt $t$, a fixed-length set of learnable query tokens $Q$ is appended to the input. These queries pass through the frozen VLM to extract enriched high-level features. A lightweight transformer connector then projects these features into the conditional embedding space of a pretrained 3D diffusion model (DiT). The diffusion model generates the 3D latent $z$, which is decoded back to mesh geometry.
   • 3D Editing: The same mechanism supports instruction-driven editing. The input sequence includes the text instruction, the latent representation of the source 3D shape, and learnable queries. The VLM extracts joint text-shape features, which the connector aligns with the diffusion model to produce an edited latent $z_{edit}$.

2. Training Strategy:

   • The framework employs a "Plug-and-Play" strategy where the heavy backbone models (VLM and 3D Diffusion) remain frozen.
   • Training is conducted in three stages using distinct datasets: (1) 3D Understanding (40K shapes), (2) Text-to-3D Generation (320K shapes), and (3) 3D Editing (14K shape-prompt pairs).
   • Only the lightweight projector, learnable queries, and the connector transformer are optimized. This significantly reduces training costs and preserves the priors learned by the standalone models.

3. Data Curation:

   • Text-3D Dataset: Created from G-Objaverse, pairing 3D meshes with multi-granularity captions (ranging from concise to detailed) generated by GPT-4o.
   • 3D Editing Dataset: Derived from Diffusion4D, utilizing 4D sequences where different frames represent pre- and post-edit states. Vision-language models generate concise editing instructions describing the transformation.

Key Contributions

• First Unified AR+Diffusion Framework: PnP-U3D is presented as the first framework to unify 3D understanding and generation by combining autoregression for understanding with continuous diffusion for generation, avoiding the pitfalls of forced quantization.
• Lightweight and Compatible Design: By freezing pretrained backbones and training only lightweight adaptation modules (projector, queries, connector), the framework achieves high performance with significantly lower training costs compared to training from scratch or full fine-tuning.
• Effective Cross-Modal Bridging: The use of learnable query tokens and a transformer connector enables efficient information exchange between the LLM feature space and the 3D diffusion conditional space, preserving the strengths of both domains.
• Instruction-Driven Editing: The framework naturally extends to 3D editing, allowing users to modify input shapes via natural language while preserving the identity and geometry of the original object.

Results

• 3D Understanding: On the Objaverse dataset, PnP-U3D outperforms state-of-the-art methods like ShapeLLM-Omni and PointLLM in semantic accuracy (Sentence-BERT and SimCSE scores), particularly when augmented with multi-view 2D images to provide texture cues.
• 3D Generation: In text-to-3D generation, the method achieves the best Q-Align score among baselines (Trellis, ShapeLLM-Omni, SAR3D, GaussianAnything), indicating superior alignment between text prompts and generated shapes. Qualitative results show the model captures fine-grained details (e.g., "circular cut-outs") often missed by competitors.
• 3D Editing: The model successfully performs complex pose and appearance edits (e.g., changing a male figure to a female figure) while maintaining the original object's identity. Ablation studies reveal that increasing the query token length (up to 1024) significantly improves the preservation of input attributes and semantic alignment.
• Generalization: The framework demonstrates robustness by generalizing to longer latents (up to 4096) at inference despite being trained on 1024 tokens, and it adapts effectively to different 3D generative backbones (e.g., CraftsMan3D).

Significance
The paper claims that PnP-U3D highlights the potential of unified AR+diffusion models as a promising direction for building more general-purpose 3D intelligence. By demonstrating that effective unification does not require enforcing a single autoregressive paradigm, the work offers a scalable and cost-efficient path forward for 3D tasks. The authors posit that their approach sheds new light on developing unified 3D understanding-generation models and intelligent editing systems, encouraging the community to explore architectures that leverage pretrained priors rather than discarding them.