DogWeave: High-Fidelity 3D Canine Reconstruction from a Single Image via Normal Fusion and Conditional Inpainting

DogWeave is a novel framework that reconstructs high-fidelity 3D canine models from a single RGB image by refining parametric meshes into detailed SDF representations via diffusion-enhanced normal optimization and generating view-consistent textures through conditional inpainting, thereby overcoming challenges like self-occlusion and fur detail to outperform existing state-of-the-art methods.

The Gist

Imagine you have a single, beautiful photograph of a dog. Maybe it's your golden retriever, Buster, sitting in the park. Now, imagine you want to turn that flat 2D photo into a 3D model that you can spin around, zoom in on, and even put into a video game.

The problem is, the photo only shows you the front of Buster. It doesn't show his back, his tail, or the fur on the other side of his ears. If you try to guess what's on the back, you might end up with a model that looks like a blob, or one where the fur patterns don't match the front.

DogWeave is a new computer program designed to solve this exact puzzle. It takes a single photo of a dog and builds a hyper-realistic 3D model, filling in all the missing parts so perfectly that it looks like you took a 360-degree photo of the dog yourself.

Here is how it works, broken down into three simple steps using some creative analogies:

1. The Clay Sculptor (Coarse Geometry)

First, the program needs a basic shape. It doesn't start from scratch; it uses a "digital skeleton" of a dog (a pre-made template) that fits the general shape of the dog in your photo.

• The Analogy: Think of this like a potter throwing a lump of clay on a wheel. They get the general shape of a vase, but it's still smooth and lacks details like handles or ridges.
• What DogWeave does: It takes this rough clay shape and starts smoothing it out, making sure the legs are the right length and the head is in the right spot, based on the shadows and lines in your photo.

2. The Detail Artist (Normal Fusion)

Now the shape is right, but it's still too smooth. Real dogs have wrinkles, fur tufts, and tiny bumps. The program needs to add these tiny details.

• The Analogy: Imagine the clay pot is now covered in a layer of "magic dust" that tells the computer exactly how the light should hit every single tiny bump. The program uses a special AI (called a Diffusion model) to predict what these tiny bumps look like on the parts of the dog you can't see in the photo.
• What DogWeave does: It looks at the photo from many different angles (virtually) and uses AI to guess the "surface texture" of the invisible parts. It fuses these guesses together to create a highly detailed 3D surface that looks like real fur, not just smooth plastic.

3. The Painter with a Memory (Conditional Inpainting)

Finally, the model needs color and patterns. This is the hardest part. If you paint the front of a dog with black spots, the back needs to have black spots too, in the right places. If the dog is a "Blue Heeler," the back needs to look like a Blue Heeler, not a Dalmatian.

• The Analogy: Imagine a painter who is blindfolded for half the canvas. They have to paint the back of the dog. To do this, they have a "memory book" (the breed information) and a "style guide" (the photo). They don't just guess; they look at the front of the dog and say, "Okay, the front has a white patch here, so the back must continue that pattern logically."
• What DogWeave does: It uses a technique called "inpainting." It paints the missing parts of the dog one view at a time, but it constantly checks its "memory" (the breed type) and the "style" (the original photo) to make sure the fur patterns match perfectly. It ensures that if the dog has a specific spot on its left ear, the model knows exactly what that spot looks like from the back.

Why is this a big deal?

Before DogWeave, computers trying to do this usually made mistakes:

• The "Blob" Problem: The 3D shape was too smooth and looked fake.
• The "Drift" Problem: The back of the dog looked like a different dog entirely (e.g., the front was a Poodle, the back looked like a Beagle).
• The "Blurry" Problem: The fur looked like a fuzzy mess instead of individual strands.

DogWeave fixes all of these. It creates a model that is so realistic, you can spin it around, and the fur patterns, wrinkles, and colors stay consistent, just like a real dog.

In a nutshell: DogWeave is like a master sculptor and painter working together. The sculptor builds the perfect body shape, and the painter uses a "breed memory" to paint the missing sides of the dog so perfectly that you can't tell where the photo ends and the computer's imagination begins.

Technical Summary

Here is a detailed technical summary of the paper "DogWeave: High-Fidelity 3D Canine Reconstruction from a Single Image via Normal Fusion and Conditional Inpainting."

1. Problem Statement

The paper addresses the challenge of monocular 3D animal reconstruction, specifically for canines. Existing methods struggle with three main issues:

• Complex Articulation & Self-Occlusion: Animals have complex poses and self-occluding parts (e.g., legs, tails) that make inferring 3D geometry from a single 2D image difficult.
• Fine-Scale Details: Capturing high-frequency details like fur patterns, wrinkles, and specific breed markings is challenging.
• Data Scarcity: Most public animal datasets contain only single-view images (frontal/lateral), lacking the multi-view 3D supervision required for training robust models.
• The Trilemma: Current approaches face a trade-off between:
  • Few-shot inference methods: Fast but produce over-smoothed geometry, washed-out textures, and incorrect topology due to a lack of anatomical priors.
  • Model-based methods: Better alignment but often suffer from low-fidelity geometry and texture misalignment due to inherent shape ambiguities in single-view supervision.

2. Methodology

DogWeave is a model-based optimization framework that reconstructs photorealistic 3D dog models from a single RGB image without ground-truth 3D data. It operates in a three-stage pipeline:

Stage 1: Coarse Geometry Initialization

• Base Model: Starts with BITE (a dog-specific parametric model derived from D-SMAL) to establish a base mesh with correct skeletal proportions.
• Refinement: The mesh is refined using monocular depth and normal supervision from Marigold (a diffusion-based depth/normal estimator).
• Loss Function: A progressive refinement minimizes a total loss comprising:
  • Depth Loss: Scale-invariant loss to align body proportions.
  • Perceptual Normal Loss: Uses VGG-16 features to match surface style and curvature (Gram matrix) rather than just pixel-wise alignment, preserving local surface patterns.
  • Silhouette & Smoothness Losses: Standard regularization.

Stage 2: SDF-Based Geometric Refinement

To recover fine-scale details (wrinkles, fur) that the fixed-topology BITE mesh cannot capture:

• Conversion: The coarse mesh is converted into a volumetric Signed Distance Field (SDF) ($256^3$ resolution).
• Diffusion-Enhanced Normals: A custom ControlNet model (fine-tuned on Stable Diffusion 1.5) enhances the normal maps predicted by Marigold.
• Multi-View Normal Fusion: Instead of relying on a single view, the system renders normals from multiple azimuth views, enhances them via diffusion, and fuses them into a target normal field using a view-weighted averaging strategy. This resolves 3D inconsistencies and reduces noise.
• Optimization: The SDF is optimized to match the fused target normals using a volumetric consistency loss (cosine distance), a mask loss (for occupancy), and an Eikonal penalty (to ensure valid distance field properties).

Stage 3: Geometry-Aware Texture Synthesis

This stage generates photorealistic, view-consistent textures, particularly for unobserved regions:

• Volumetric Color Field: A $256^3$ color grid is initialized on the optimized SDF.
• Conditional Inpainting: A sequential inpainting scheme is used. For each view in a circular camera array:
  1. Render the current partial texture to identify missing regions.
  2. Use a diffusion inpainting model (Stable Diffusion 1.5 + ControlNet + IP-Adapter) to generate the missing texture.
  3. Inputs to the Diffusion Model: Rendered normals (structure), current partial texture, mask of missing regions, the input image (style reference), and Breed Information (text prompt).
• Volumetric Projection: Generated textures are projected onto the 3D volume using "volumetric tubes" (cylindrical neighborhoods) with confidence weighting based on view direction and surface normal alignment to ensure smooth blending and continuity.

3. Key Contributions

1. Novel Framework: Introduced DogWeave, the first model-based single-image-to-3D framework for high-fidelity canines trained solely on single-view 2D images without 3D ground truth.
2. Diffusion-Enhanced Geometry: Proposed a method to convert coarse parametric meshes into detailed SDFs using multi-view normal fusion guided by a fine-tuned diffusion model, effectively recovering fine surface details.
3. Identity-Preserving Texturing: Developed a sequential geometry- and style-conditioned inpainting scheme. By integrating IP-Adapter and breed-specific text prompts, the method ensures that unobserved regions (e.g., the back of a dog) maintain the correct fur patterns and identity, preventing "identity drift."
4. Self-Supervised Training: Demonstrated a pipeline to construct paired training data (coarse/fine normals, normals/textures) from a dataset of ~7,000 single-view dog images.

4. Experimental Results

The method was evaluated on the Stanford Dog Dataset (cross-dataset from the training set) against state-of-the-art baselines (Wonder3D, TRELLIS, CRM, TripoSR, Fauna, SAM3D, Hunyuan3D).

• Quantitative Performance: DogWeave achieved State-of-the-Art (SOTA) results across all perceptual metrics:
  • FID (Realism): 176.4 (Best), significantly lower than the next best (Hunyuan3D at 194.3).
  • CLIP Score (Identity): 0.9081 (Best), indicating superior preservation of breed-specific features.
  • LPIPS & DreamSim: Lowest scores, indicating the highest perceptual similarity to the input image and consistent appearance across novel views.
• Qualitative Performance:
  • Successfully reconstructed fine details like fur patterns and wrinkles.
  • Maintained anatomical correctness and consistent coloring across different viewpoints.
  • Baselines exhibited issues such as color drift, topology errors, over-smoothing, or blurred textures in occluded areas.
• Ablation Studies: Confirmed that removing any component (Perceptual Loss, Normal Enhancer, IP-Adapter, or Breed Conditioning) degraded performance, with IP-Adapter being the most critical for identity consistency.

5. Significance and Limitations

Significance:

• DogWeave bridges the gap between parametric models (good structure, low detail) and generative models (high detail, poor structure) for quadrupeds.
• It proves that style-controlled diffusion combined with explicit geometric constraints can solve the "unobserved region" problem in single-image reconstruction without needing expensive 3D scans.
• The framework offers a scalable path for digitizing animals for animation, gaming, and VR.

Limitations & Future Work:

• Inference Time: The optimization-based pipeline takes ~8 minutes per model (vs. seconds for feed-forward methods), though future work could explore distillation.
• Domain Specificity: Currently relies on the BITE dog prior; extending to other species (cats, horses) requires corresponding parametric priors.
• Lighting: The method "bakes" lighting and shadows into the texture (albedo), making relighting difficult.
• Extreme Occlusion: Highly curled poses or extreme self-occlusion can still lead to incomplete textures where geometric cues are too ambiguous for the diffusion model to hallucinate plausible details.