Image-Guided Geometric Stylization of 3D Meshes

1. Problem Statement

Current 3D generative models and stylization methods face significant limitations:

• Limited Geometric Control: Most existing 3D stylization techniques focus on transferring high-frequency surface textures (colors, patterns) rather than altering the underlying geometry.
• Ambiguity of Text Prompts: Text-guided deformation often struggles to capture specific, complex geometric styles (e.g., the rigid structure of a fire hydrant or the expressive pose of a spider) because text descriptions are inherently ambiguous regarding spatial structure.
• Topological Integrity: Volumetric representations (like NeRFs) often lack rigorous topological structures, making them incompatible with standard 3D pipelines (UV maps, rigging, remeshing).
• Optimization Instability: Directly optimizing mesh vertices or using standard latent diffusion encoders for 3D geometry often leads to unstable gradients, fragmented deformations, or a failure to capture global structural changes.

The authors propose a framework to deform a valid 3D mesh to express the geometric style of a reference image while preserving the mesh's original topology and part-level semantics.

2. Methodology

The proposed pipeline, GeoStyle, transforms a source mesh into a stylized version through a coarse-to-fine optimization process guided by image-based style references.

A. Style Extraction (LoRA + DreamBooth)

Instead of relying on text prompts, the method extracts geometric style directly from reference images.

• Mechanism: It fine-tunes a pre-trained latent diffusion model (SDXL) using DreamBooth on a small set of style reference images.
• Efficiency: To reduce computational cost, the model is adapted using LoRA (Low-Rank Adaptation) weights.
• Result: The trained LoRA weights serve as a compact, abstract representation of the target geometric style (including shape, proportions, and structural traits), which acts as the optimization target.

B. Deformation via Score Distillation Sampling (SDS)

The core deformation mechanism uses Score Distillation Sampling (SDS) to guide the mesh geometry toward the style distribution.

• Per-Face Jacobians: Instead of optimizing vertex positions directly (which causes fragmentation), the method optimizes per-face Jacobian matrices ($J_i \in \mathbb{R}^{3\times3}$). This allows for coherent, large-scale shape changes.
• Vertex Recovery: The actual vertex positions are recovered by solving a Poisson equation based on the optimized Jacobians.
• Approximated VAE Encoder: A critical innovation is the use of an approximated linear encoder for the VAE in SDXL. Standard VAE encoders are computationally heavy and provide unstable gradients for geometry optimization. The authors fit a linear matrix $A$ to approximate the mapping from rendered images to latent space ($z \approx A\bar{x}$), significantly stabilizing and accelerating the optimization.

C. Coarse-to-Fine Deformation Pipeline

To handle both global structural changes and fine details without destroying the mesh identity, the authors employ a two-stage strategy:

1. Coarse Stage (Global Structure):

   • Auxiliary Mesh & Cages: An auxiliary mesh composed of spheres is generated from the source mesh vertices. Semantic parts are segmented (using PartField), and Oriented Bounding Boxes (OBBs) are fitted to these parts.
   • Cage Coefficients: The deformation is driven by transforming these OBBs (scale, rotation, translation). The source mesh vertices are moved based on "cage coefficients" that interpolate the OBB transformations.
   • Loss: This stage minimizes an SDS loss on the auxiliary mesh and a Cage Loss ($L_{cage}$) to ensure the target mesh follows the semantic structure of the auxiliary mesh, preventing topology collapse.

2. Fine Stage (Local Details):

   • The cage regularization is gradually decayed, and the optimization focuses on refining the per-face Jacobians.
   • This stage captures fine-grained geometric details and surface variations while maintaining the global pose established in the coarse stage.

D. Regularization

• Symmetry Loss: If the source mesh has reflective symmetry, a symmetry loss ($L_{sym}$) is applied to maintain internal consistency during deformation.
• Jacobian Regularization: A term ($L_{reg}$) encourages Jacobians to stay close to the identity matrix to prevent excessive deviation from the source mesh.

3. Key Contributions

1. Image-Guided Geometric Stylization: A novel framework that transfers complex geometric styles from 2D images to 3D meshes, overcoming the ambiguity of text prompts.
2. Approximated VAE Encoder: A crucial technical contribution that enables stable and efficient gradient propagation from SDXL to 3D mesh geometry, solving a major bottleneck in latent diffusion-based 3D optimization.
3. Coarse-to-Fine Deformation Strategy: A pipeline combining cage-guided semantic deformation (for global structure) and per-face Jacobian optimization (for local details), ensuring valid topology and semantic consistency.
4. LoRA-based Style Extraction: Utilizing DreamBooth and LoRA to create a compact, style-specific optimizer that captures holistic geometric traits rather than just surface textures.

4. Experimental Results

• Qualitative Comparison: The method outperforms baselines (Paparazzi, Neural 3D Mesh Renderer, MeshUp, Text2Mesh, TextDeformer) in generating expressive geometric deformations. Baselines often fail to change global structure or produce noisy artifacts, whereas GeoStyle successfully transfers poses, silhouettes, and structural rigidity.
• User Study: In a study with 32 participants, GeoStyle achieved the highest rankings for Geometry Alignment (matching the style reference) and Aesthetic Style Transfer. While some baselines scored higher on "Content Preservation," this was attributed to their inability to deform the mesh significantly.
• Ablation Studies:
  • Without Approx. Encoder: The model fails to deform global shapes meaningfully, staying close to the source mesh.
  • Without Cage/Auxiliary Mesh: Deformations become distorted and lose semantic coherence.
  • Without Symmetry Loss: Symmetric objects lose their internal consistency.
• Flexibility: The system supports additional conditioning, such as combining text prompts with image styles (e.g., "make it a giraffe" with a specific artistic style) and localized deformation (stylizing only specific parts of the mesh).

5. Significance

This work bridges the gap between 2D image aesthetics and 3D geometric structure. By enabling image-driven geometric deformation, it allows artists and creators to generate distinctive 3D assets that retain the valid topology required for animation, rendering, and manufacturing. The method moves beyond simple texture transfer, addressing the challenge of "how to make a 3D object look like a specific artistic style" by manipulating the object's very shape and pose, all while maintaining compatibility with standard 3D workflows (UVs, rigging). The introduction of the approximated VAE encoder provides a practical solution for integrating powerful latent diffusion models into geometry optimization tasks.