Text-Image Conditioned 3D Generation

1. Problem Statement

Current 3D generative models typically rely on a single conditioning modality, either text or image, which creates inherent limitations in satisfying complex user intent:

• Image-Conditioned Models: While they achieve high visual fidelity and preserve local appearance details, they suffer from viewpoint bias. If the input image is occluded, atypical, or lacks coverage of the object, the model hallucinates unobserved regions, often deviating from the intended semantics (e.g., generating a trophy with a missing base because the input view was a top-down shot).
• Text-Conditioned Models: These provide broad semantic guidance and global intent but lack low-level visual constraints. Consequently, they often produce 3D assets that are semantically correct but exhibit poor visual fidelity, ambiguous geometry, and lack pixel-aligned details.

The core problem addressed is how to combine these two modalities to create a 3D generator that is both semantically aligned with a textual description and faithful to the visual appearance of a reference image, even under ambiguous viewing conditions.

2. Methodology: TIGON

The authors propose TIGON (Text–Image conditioned GeneratiON), a minimalist dual-branch baseline designed for Text–Image Conditioned 3D Generation.

Core Architecture

TIGON employs a dual-branch Diffusion Transformer (DiT) architecture:

1. Dual Backbones: It maintains two separate, modality-specialized DiT backbones:

   • An Image-conditioned branch ($\mathcal{F}_{img}$) that processes visual cues (dense, view-grounded tokens).
   • A Text-conditioned branch ($\mathcal{F}_{txt}$) that processes semantic descriptions (sparse, abstract tokens).
   • Rationale: Keeping them separate prevents the "granularity mismatch" where mixing heterogeneous token types in a single backbone degrades performance.

2. Early Fusion (Cross-Modal Bridges):

   • To enable interaction between the branches, the authors insert zero-initialized linear projections (bridges) between corresponding blocks of the two DiT streams.
   • These bridges allow bidirectional feature exchange ($\mathbf{f}_{img} \leftarrow \mathbf{f}_{txt}$ and vice versa).
   • Zero-Initialization: Inspired by ControlNet, the bridges start with zero weights. This ensures that at the beginning of joint training, the branches behave exactly like their pre-trained single-modality counterparts, ensuring training stability. The network gradually learns when and how much to exchange information.

3. Late Fusion (Prediction Averaging):

   • At each denoising step $k$, the velocity fields predicted by both branches ($\mathbf{v}_{img}$ and $\mathbf{v}_{txt}$) are simply averaged to produce the final velocity field:
     $$\mathbf{v} = \frac{1}{2}(\mathbf{v}_{img} + \mathbf{v}_{txt})$$
   • The paper argues that sophisticated learnable fusion mechanisms (e.g., attention-based weighting) are unnecessary because the early fusion and joint fine-tuning allow the branches to implicitly learn the optimal weighting.

Training Strategy

• Two-Stage Process:
  1. Pre-training: The image and text branches are pre-trained separately on their respective modalities to ensure balanced learning capabilities.
  2. Joint Fine-tuning: The zero-initialized bridges are trained, and all parameters are jointly fine-tuned.
• Condition Dropout: To preserve the ability to generate from text-only or image-only inputs (free-form conditioning), the model is trained with a 50% dropout probability for each condition. This creates a uniform mixture of four regimes: unconditional, text-only, image-only, and text+image.

3. Key Contributions

1. Task Formulation: The paper formally defines Text–Image Conditioned 3D Generation, a task requiring the model to jointly reason over a visual exemplar and a textual specification to generate a consistent 3D asset.
2. Diagnostic Study: Through empirical analysis, the authors demonstrate that existing single-modality models fail significantly when input views are low-information (e.g., occluded). They show that even a naive late fusion of pre-trained models significantly recovers quality, proving the strong complementarity of the two modalities.
3. TIGON Baseline: They introduce TIGON, a simple yet effective architecture that leverages modality-specific backbones with lightweight cross-modal fusion (early bridges + late averaging).
4. Comprehensive Evaluation: Extensive experiments on Toys4K and UniLat1K benchmarks demonstrate that TIGON outperforms strong single-modality baselines and existing multimodal approaches.

4. Experimental Results

The authors evaluated TIGON on Toys4K and UniLat1K datasets using metrics like CLIP score, FDDINOv2 (Fréchet Distance), ULIP, and Uni3D.

• Quantitative Performance:

  • Text-Image Conditioning: TIGON significantly outperforms single-modality baselines. For example, on Toys4K, TIGON (I+T) achieved an FDDINOv2 of 61.59, compared to 84.62 for Image-only and 152.34 for Text-only (lower is better).
  • Robustness: TIGON maintains high performance even when the input image is a low-information view (e.g., bottom view), whereas image-only models degrade drastically.
  • Single-Modality Capability: Due to condition dropout training, TIGON remains competitive in pure text-only or image-only modes, matching or exceeding specialized single-modality models.

• Qualitative Findings:

  • Disambiguation: Text successfully resolves ambiguities in the image (e.g., specifying "with a long coiled tiger tail" when the image is a top-down view).
  • Visual Fidelity: Images anchor the geometry and texture, preventing the "hallucination" of incorrect shapes common in text-only generation.
  • Controllability: The model allows for diverse generation by fixing the image and varying the text (or vice versa), enabling fine-grained attribute editing while preserving object identity.

5. Significance and Impact

• Paradigm Shift: The paper moves the field beyond single-modality constraints, establishing that vision-language complementarity is crucial for high-quality, controllable 3D generation.
• Efficiency: The proposed method is "minimalist," avoiding complex attention mechanisms or heavy architectural changes, making it easy to integrate into existing 3D diffusion pipelines (like TRELLIS or UniLat3D).
• Practical Application: By solving the viewpoint bias of image-based generation and the lack of detail in text-based generation, TIGON offers a more robust tool for VR/AR, industrial design, and embodied AI, where precise control over both shape and semantics is required.

In conclusion, TIGON demonstrates that combining text and image conditions is not just additive but synergistic, enabling the generation of 3D assets that are simultaneously semantically precise and visually faithful.