🤖 machine learning

DVD: Discrete Voxel Diffusion for 3D Generation and Editing

This paper introduces Discrete Voxel Diffusion (DVD), a framework that treats voxel occupancy as a native discrete variable to enable efficient generation, uncertainty estimation, and single-round editing of sparse voxel scaffolds for 3D pipelines without requiring continuous-to-discrete thresholding.

Original authors: Zhengrui Xiang, Jiaqi Wu, Fupeng Sun, Heliang Zheng, Yingzhen Li

Published 2026-05-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhengrui Xiang, Jiaqi Wu, Fupeng Sun, Heliang Zheng, Yingzhen Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Building 3D Worlds with LEGO Bricks

Imagine you want to build a 3D object, like a toy car or a house, but instead of using smooth clay, you have to build it out of tiny, invisible LEGO bricks (called voxels). Some of these bricks are there (occupied), and some are empty space.

Most current 3D AI tools try to build these objects by first imagining a smooth, continuous blob of clay and then trying to chop it into LEGO bricks later. The authors of this paper argue that this is like trying to cut a smooth cake into perfect cubes—it often leads to messy edges, missing pieces, or "holes" in the final structure.

Instead, they propose DVD (Discrete Voxel Diffusion). This method skips the "smooth clay" step entirely. It treats the LEGO bricks as discrete items from the very beginning: a brick is either ON or OFF. There is no in-between.

The Problem with the Old Way (Continuous Diffusion)

Think of the old method like a sculptor working with wet clay. They smooth out the surface, and then, at the very end, they try to snap the clay into a grid of cubes.

The Issue: If the sculptor makes a tiny mistake near the edge of a cube, the whole cube might get snapped off or left empty. This creates "holes" in the 3D object that look like missing teeth or broken legs.
The Paper's Claim: The authors found that trying to force a smooth, continuous process to create a blocky, binary structure (ON/OFF) causes these errors.

The New Solution: The "Binary Switch" Approach

The DVD method is like having a master builder who only thinks in terms of switches: "Is this brick ON? Yes. Is this one OFF? Yes."

Direct Modeling: Instead of guessing a smooth shape and then cutting it, the AI predicts the state of every single brick directly. It asks, "Is this specific spot empty or full?"
The Result: Because it doesn't have to guess a smooth curve first, it avoids the "cutting" errors. The paper shows this creates cleaner shapes with fewer holes and better details, even though the AI model is actually smaller and trained on less data than the previous giants.

Feature 1: The "Uncertainty Meter" (Knowing What You Don't Know)

One of the coolest features of DVD is that it doesn't just guess; it knows how confident it is.

The Analogy: Imagine a weather forecaster. A continuous model might say, "It's 50% likely to rain," which is vague. The DVD model acts like a forecaster who says, "I am 99% sure it will rain here, but I am only 51% sure about this tiny patch over there."
How it helps: The paper uses this "confidence score" (called entropy) to find the tricky parts of a 3D shape. If the AI is confused about a specific area (like the thin handle of a fire extinguisher), it flags it. This allows the system to filter out bad data or focus on improving the difficult parts of the shape.

Feature 2: Editing with "Block-Structured Perturbations"

Sometimes you want to change a part of a 3D object, like turning a flat roof into a pointed one, without rebuilding the whole thing.

The Old Way: To edit a 3D object, previous methods often had to resample the whole thing over and over again, which is slow and computationally expensive. It's like trying to fix a hole in a wall by tearing down the whole house and rebuilding it.
The DVD Way: The authors introduced a trick called Block-Structured Perturbation.
- The Analogy: Imagine you are painting a mural. Instead of repainting the whole wall to fix one flower, you put a stencil (a block) over just the flower area and repaint only that block.
- How it works: The AI is trained to handle these "blocks" of change. When you want to edit a part of the object, the AI treats that area as a "corrupted block" and fills it in while keeping the rest of the object perfectly intact. This happens in a single pass, making editing fast and seamless.

Summary of Claims

Better Quality: By treating 3D voxels as simple ON/OFF switches rather than smooth clay, the method creates fewer holes and sharper details.
Efficiency: It achieves these results with a smaller model and less training data than previous methods.
Transparency: It provides a built-in "confidence meter" that tells you exactly where the 3D shape is ambiguous or difficult to generate.
Easy Editing: It allows for fast, single-step editing of specific parts of a 3D object without needing to regenerate the whole thing.

The paper concludes that this "discrete" approach is a practical and powerful new way to build the skeleton (scaffold) of 3D objects before adding the final details.

Technical Summary: Discrete Voxel Diffusion (DVD) for 3D Generation and Editing

1. Problem Statement

Recent advancements in 3D generative modeling have increasingly adopted SLAT (Structured LATent) pipelines, which operate in two stages: first generating a sparse voxel scaffold to define the coarse shape, and second predicting continuous latent features attached to active voxels for high-fidelity decoding into meshes, 3DGS, or NeRFs.

However, the first stage of these pipelines typically relies on continuous diffusion models operating on VAE latents. This approach forces a continuous-to-discrete conversion where voxel occupancy is generated continuously and then discretized via thresholding. The authors identify several critical issues with this paradigm:

Artifacts: Small errors near the binarization threshold can flip voxels from occupied to empty (or vice versa), creating local holes or disconnected fragments that persist through meshing.
Misalignment: The native structure of voxel occupancy is sparse and binary, yet it is modeled as a continuous distribution, reducing interpretability and complicating downstream editing.
Uncertainty: Continuous models approximate Gaussian posteriors, making it difficult to explicitly quantify uncertainty or identify ambiguous geometric regions.

The paper posits that modeling sparse voxel occupancy directly as discrete data is a more natural and effective approach for this specific sub-task.

2. Methodology

2.1 Discrete Voxel Diffusion (DVD)

The core contribution is a Discrete Diffusion framework applied to the first stage of SLAT pipelines. Instead of generating continuous latents, DVD treats the voxel grid $X \in \{0, 1\}^{N \times N \times N}$ as a sequence of discrete tokens.

Uniform State Discrete Diffusion (USDM): The authors adopt USDMs, where the prior distribution is uniform across all possible states (occupied or empty). The forward process randomly reassigns each voxel to 0 or 1 based on a time-dependent probability.
Reverse Process: The model predicts the categorical distribution $q(x_0 | x_t)$ over the clean states given a noisy sample. Unlike continuous diffusion which estimates a mean, the discrete model explicitly outputs a probability distribution for each voxel's state.
Sampling: Generation proceeds via ancestral sampling, drawing samples from the predictive posterior at each timestep.

2.2 Uncertainty Estimation

A unique feature of the discrete formulation is the explicit access to the predictive categorical distribution. The authors leverage predictive entropy as a robust uncertainty metric:

Voxel-level: High entropy indicates ambiguous regions where the model is unsure if a voxel should be occupied.
Shape-level: By aggregating voxel entropies (specifically at a near-zero timestep $t=\epsilon$ ), they derive a lightweight uncertainty score ( $\gamma_{\epsilon, \rho}$ ). This score correlates with geometric complexity and can be used for data filtering, quality assessment, and identifying "difficult" samples.

2.3 Block-Structured Perturbation (BSP) for Editing

Standard discrete models are trained with spatially homogeneous noise, making them ill-suited for inpainting or editing where specific regions are masked (known) and others are corrupted. To bridge this training-inference gap, the authors propose Block-Structured Perturbation (BSP):

Strategy: During fine-tuning, instead of perturbing individual voxels independently, the model is trained on axis-aligned hypercubes (blocks) of varying scales.
Mechanism: Random blocks are selected and corrupted, while the rest of the volume remains largely intact. This simulates local editing scenarios, forcing the model to learn to generate structures consistent with a clean context.
Efficiency: This allows for single-round editing and inpainting without the need for iterative resampling strategies (like RePaint) or additional network evaluations.

3. Key Contributions

Discrete Diffusion for Sparse Voxel Generation: The paper introduces a discrete diffusion formulation for the first stage of SLAT pipelines. It avoids continuous-to-discrete thresholding artifacts, achieving improved performance on most metrics with a smaller model size and less training data compared to continuous baselines.
Explicit Uncertainty Modeling: The authors demonstrate that discrete diffusion provides native predictive probabilities, enabling the derivation of a lightweight, entropy-based uncertainty score. This score effectively localizes ambiguous geometric regions and serves as a proxy for shape complexity.
Efficient Voxel Editing via BSP: A novel fine-tuning strategy using block-structured perturbations is proposed. This enables seamless voxel inpainting and editing within a single sampling trajectory, preserving structural integrity without auxiliary computation.

4. Experimental Results

The authors evaluated DVD on image-to-3D and text-to-3D tasks using the TripoSR and Toys4K datasets, comparing against the continuous baseline TRELLIS and other methods like Direct3D-S2 and TripoSR.

Quantitative Performance: DVD outperformed the continuous baseline (TRELLIS) across most evaluation metrics, including:
- FID (both image rendering $FID_D$ and point cloud $FID_{PC}$ ).
- Chamfer Distance (CD and $CD_V$ for voxel-to-mesh alignment).
- CLIP Score (prompt alignment).
- Notably, these gains were achieved with a ~400M parameter model (vs. ~550M for TRELLIS) and a slightly smaller training set.
Qualitative Improvements: Visualizations showed that DVD significantly reduced "holes" and disconnected fragments common in continuous methods, particularly in thin-layer structures.
Uncertainty Utility: The uncertainty score successfully identified complex regions (e.g., handles of fire extinguishers, roof details of mushroom houses). Using this score for data augmentation (oversampling high-uncertainty shapes) further improved generation quality.
Editing Performance: In inpainting tasks, the BSP-fine-tuned model successfully reconstructed missing geometry with high spatial consistency, whereas the non-fine-tuned model and continuous baselines (even with RePaint) produced fragmented or misaligned structures.

5. Significance and Claims

The paper claims that Discrete Voxel Diffusion is a practical and effective first-stage prior for SLAT-based 3D generation. Its significance lies in:

Native Modeling: Aligning the generative process with the discrete, binary nature of voxel occupancy, thereby eliminating thresholding artifacts.
Interpretability: Providing explicit categorical modeling that allows for the inspection of generation dynamics and the derivation of uncertainty metrics, which are difficult to obtain in continuous diffusion.
Efficiency: Enabling efficient editing and inpainting through simple fine-tuning (BSP) rather than complex iterative sampling.

The authors conclude that while discrete diffusion requires hundreds of sampling steps (a limitation for real-time applications), it offers a superior inductive bias for sparse voxel scaffolds, paving the way for more robust and controllable 3D asset creation. Future work is suggested to focus on sampling acceleration and extending these principles to joint discrete-continuous modeling of features and voxels.