GAP3D: Generative Alignment of VLM Latents to Patch-Level Embeddings for 3D Generation

Technical Summary of GAP3D: Generative Alignment of VLM Latents to Patch-Level Embeddings for 3D Generation

Problem Statement
Recent efforts to integrate Vision-Language Models (VLMs) as prompt encoders for generative models face significant limitations. Standard approaches often rely on expensive end-to-end retraining or map VLM features to compressed, pooled representations (e.g., CLIP or T5 embeddings). These compressed vectors discard the dense, patch-level spatial structure essential for geometry-aware tasks like 3D asset generation. Consequently, existing methods struggle with fine-grained attribute binding, complex compositional structures, and precise geometric control. While tightly coupled frameworks offer multimodal reasoning, they lack compatibility with frozen pre-trained systems and require massive domain-specific datasets. Conversely, modular approaches that align VLMs to frozen diffusion models typically target only pooled tokens, failing to provide the spatially detailed conditioning signals required for high-fidelity 3D generation.

Methodology: GAP3D
The authors propose GAP3D (Generative Alignment of VLM latents to Patch-level embeddings for 3D generation), a modular, diffusion-based approach designed to bridge the representation gap between VLMs and dense image encoder spaces without retraining the downstream generative model.

• Generative Alignment via Rectified Flow: Unlike deterministic regression, GAP3D employs a diffusion transformer (DiT) based on the Rectified Flow framework. This generative approach is chosen because mapping abstract semantic features from a VLM to dense visual representations involves synthesizing geometry and appearance details not explicitly present in the text prompt, making the task non-deterministic.
• Target Space: The model aligns VLM-generated latents directly to the complete feature space of a pre-trained image encoder, specifically DINOv2. This target space includes:
  • A dense 2D grid of patch embeddings ($x_p$) encoding local appearance and geometry.
  • A global class token ($x_{cls}$) for high-level semantics.
  • Register tokens ($x_{reg}$) for aggregating intermediate global context.
• Architecture:
  • Input: A frozen VLM (Qwen2.5-VL-3B) encodes user prompts. Trainable "soft tokens" are appended to the VLM input to produce a sequence of latent semantic features ($C$).
  • Generator: A DiT accepts noisy states of the patch grid, CLS token, and register tokens. It uses cross-attention to inject the VLM latents ($C$) and predicts the velocity field required to denoise the inputs into the target DINOv2 embedding space.
  • Training Objective: The model is trained via flow matching to minimize the difference between the predicted velocity field and the true vector field connecting noise to ground-truth DINOv2 embeddings.
• Downstream Integration: The generated dense embeddings serve as conditioning signals for a frozen TRELLIS image-to-3D generative model. This effectively transforms the text-to-3D task into a pseudo-image-to-3D problem, leveraging the spatial priors of the image encoder.
• Domain Adaptation: To address the distribution shift between natural images (training data) and synthetic 3D renderings (target domain), the authors employ a two-stage curriculum:
  1. Pre-training: On large-scale general image-text pairs (BLIP3-o dataset) to learn a robust mapping between semantics and visual features.
  2. Fine-tuning: On a smaller dataset of 3D asset renderings (Objaverse-XL) to align the feature distribution with the frozen TRELLIS model's expectations, reducing artifacts.

Key Contributions

1. Dense Generative Representation Alignment: The paper demonstrates that VLM features can be aligned to high-dimensional, patch-level embedding spaces of pre-trained image encoders via flow matching, moving beyond compressed semantic vectors.
2. Modular 3D Asset Generation: The method enables text-to-3D generation and emergent multimodal conditioning without end-to-end retraining of the 3D backbone, extending VLM-based conditioning beyond the image domain.
3. Domain Adaptation Insights: The study highlights that domain-specific fine-tuning is critical for bridging the gap between natural images and synthetic 3D renderings, significantly reducing artifacts in modular pipelines.

Experimental Results
Evaluated on 3D asset generation using the Toys4K dataset and MS-COCO for general alignment:

• Alignment Quality: GAP3D achieves meaningful alignment between text prompts and DINOv2 patch embeddings. While the fine-tuned model shows superior alignment on 3D renderings (Toys4K), it exhibits "catastrophic forgetting" on general natural images (MS-COCO), indicating a trade-off between domain specificity and generalization.
• 3D Generation: The fine-tuned GAP3D model significantly outperforms the pre-trained version and approaches the performance of the native TRELLIS text-to-3D baseline. It successfully suppresses background artifacts (e.g., fused ground planes) and captures specific prompt details like color schemes ("yellow and black") and object shapes. However, geometric fidelity and fine-grained attribute binding (e.g., specific colors on complex shapes) remain lower than the native image-to-3D baseline.
• Emergent Multimodal Capabilities: Despite being trained exclusively on text, the model exhibits zero-shot capabilities for multimodal prompts. When provided with an image and a text instruction (e.g., "Make it red"), the model successfully extracts semantic concepts and applies changes, though it functions primarily as a semantic translator rather than a structure-preserving editor, often generating new geometry aligned with the semantics rather than strictly editing the input image.

Significance and Claims
The authors position GAP3D as a significant step toward the modular integration of foundation models. By demonstrating that VLM latents can be mapped to dense, spatially structured embedding spaces through generative alignment, the work suggests a pathway to connect independently pre-trained models without expensive retraining. The paper claims that while current limitations exist regarding fine-grained detail and geometric fidelity, the approach successfully bridges the representation gap between VLMs and image encoders. The authors emphasize that their method prioritizes high-level semantics and establishes the feasibility of connecting foundation models in high-dimensional spaces, paving the way for scalable, versatile generation systems where encoders and generators can be upgraded independently.