💻 computer science

RADmesh: Remesh-Aware Mesh Deformation

RADmesh introduces a remeshing-aware method for generative mesh deformation that overcomes the challenges of coupling discrete remeshing with noisy visual supervision by periodically updating the triangulation while preserving optimization states, thereby enabling robust, high-quality, and locally controllable shape deformations with isotropic elements.

Original authors: Nam Anh Dinh, Itai Lang, Oded Stein, Rana Hanocka

Published 2026-08-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Nam Anh Dinh, Itai Lang, Oded Stein, Rana Hanocka

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of three-dimensional computer graphics, artists and engineers often work with digital models built from a mesh of tiny triangles. Think of this mesh as a flexible net draped over a shape; the points where the strings cross are vertices, and the triangles between them define the surface. For decades, the standard way to change these shapes has been to simply pull the vertices around. This works well for gentle adjustments, like bending a finger or softening a curve. However, when the goal is to create something entirely new—growing a long tail from a smooth sphere or turning a simple block into a detailed castle—pulling the vertices becomes a struggle. The triangles stretch thin and distort, losing their shape and ruining the model's quality. To create truly complex new forms, the underlying net itself needs to change, adding more triangles where detail is needed and removing them where the surface is smooth. But changing the net while simultaneously pulling the shape is a difficult mathematical puzzle, especially when the only guide is a computer program that judges the result based on a text description.

A team of researchers at the University of Chicago, the University of Southern California, and the Technion has developed a new method called RADmesh to solve this problem. Their approach allows a computer to take a simple 3D shape and, guided by a text prompt, grow large, detailed parts or completely restructure the object while keeping the digital mesh clean and high-quality. Instead of just pulling the existing points, their system periodically reorganizes the entire web of triangles as the shape evolves. This process, known as remeshing, ensures that the triangles remain uniform and well-proportioned, even as the object grows wings, horns, or intricate architectural details. The result is a system that can turn a basic sphere into a pegasus unicorn with a long horn and feathered wings, or transform a simple chair into one with a backrest shaped like a blooming flower, all while maintaining a smooth, efficient structure.

The core innovation lies in how the researchers handle the connection between the shape's movement and the changing mesh. In previous methods, if the computer tried to add new triangles while the shape was moving, the optimization process would often break down or produce noisy, jagged results. The RADmesh team solved this by treating the deformation not as a simple shift of points, but as a set of instructions for rotation and scaling attached to each vertex. When the system decides it is time to reorganize the mesh, it does not discard the progress made so far. Instead, it carefully transfers the "memory" of the optimization—the momentum and direction of the growth—from the old triangles to the new ones. This allows the computer to continue refining the shape seamlessly, as if the mesh had never been touched, even though the underlying structure has been completely rebuilt.

The researchers demonstrated this capability with a wide variety of examples, showing that the method works for both small, local changes and massive, global transformations. In one instance, they started with a basic cow model and selected a small region on its back. The system then grew a long, lizard-like tail and a pair of pegasus wings from that tiny spot, adding fine details like feathers and scales without distorting the rest of the animal. In another experiment, they took a simple sphere and, guided by a text prompt, evolved it into a complex medieval stone fireplace or a tall pagoda. Throughout these dramatic changes, the system maintained a high level of detail at the extremities, such as the branching tines of reindeer antlers or the folds of a skirt, while keeping the triangles in the middle of the shape neat and uniform.

To test the effectiveness of their method, the team compared it against other recent techniques that generate 3D shapes from text. Many of these existing methods rely on intermediate steps, such as creating a blurry 3D cloud or painting multiple 2D images before assembling them into a model. These approaches often result in noisy surfaces or shapes that lack structural integrity. Other methods that work directly on the mesh without changing the triangle count struggle to create large appendages or fine details, often resulting in stretched or broken geometry. The RADmesh approach outperformed these competitors in both visual quality and the efficiency of the triangle count. The models produced were not only more faithful to the text descriptions but also featured cleaner surfaces with better-shaped triangles, proving that changing the mesh structure is essential for high-quality generative design.

The process is not without its limitations. The system relies on a visual supervision signal that can be slow to compute, meaning that generating a single complex shape can take over an hour on a powerful graphics card. Additionally, the current method preserves the overall topology of the object, meaning it cannot create holes or merge separate parts together, though the researchers note this could be a direction for future work. Despite these constraints, the study suggests a significant step forward in how computers manipulate 3D space. By combining the ability to reshape the mesh with a robust method for tracking the optimization state, the researchers have created a tool that can grow complex, detailed geometries from simple beginnings, bridging the gap between a text prompt and a tangible, high-fidelity 3D object.

Technical Summary: RADmesh: Remesh-Aware Mesh Deformation

Problem Statement

Recent generative 3D methods often optimize mesh deformations using visual losses (e.g., Score Distillation Sampling from text-to-image diffusion models) to guide vertex displacements. However, these approaches typically maintain a fixed mesh connectivity (triangulation). The authors argue that sufficiently drastic deformations—such as growing large appendages, thin sheet-like volumes, or reconfiguring structural elements—cannot be achieved without compromising element quality. Keeping connectivity constant forces triangles to stretch or shrink excessively, leading to poor fidelity, bad triangle quality, and inefficient resolution distribution.

While remeshing (changing connectivity) is a standard tool in geometry processing, it is rarely integrated into generative deformation tasks. This is because remeshing is a discrete operation involving changes to indices and pointers, making it difficult to couple with the noisy, continuous supervision signals provided by visual losses. Existing visual-supervised remeshers are often designed for inverse rendering with vertex positions as variables, whereas generative tasks benefit more from differential deformation quantities (e.g., per-face Jacobians or per-vertex rotations).

Methodology

RADmesh proposes a remeshing-enhanced optimization pipeline that integrates isotropic remeshing directly into the generative deformation loop. The method operates on a base mesh $M_{ri}$ and optimizes a deformation quantity $Q$ rather than raw vertex positions.

1. Deformation Quantity ( $Q$ )

The core innovation is a vertex-based deformation quantity $Q = \{q_k \in \mathbb{R}^6\}$ , extending the normal-based approach of Geometry in Style. For each vertex $k$ , $q_k$ consists of:

Direction ( $q^{dir}_k \in \mathbb{R}^3$ ): A target direction vector.
Scale ( $q^{scale}_k \in \mathbb{R}^3$ ): A scaling vector for the three world axes.

The deformation function $D$ computes new vertex positions via a two-step process:

Local Step: Computes a local transform $T_k$ for each vertex, combining a rotation matrix $R_k$ (solved via a Procrustes problem aligning local edge vectors to the target direction) and a diagonal scaling matrix $S_k$ derived from $q^{scale}_k$ .
Global Step: Solves for global vertex positions that best satisfy these local transforms using a differentiable As-Rigid-As-Possible (dARAP) solver.

This vertex-based representation allows for straightforward interpolation of optimizer states during remeshing, unlike face-based Jacobians which are harder to map across changing connectivity.

2. Optimization Loop with Remeshing

The optimization proceeds in epochs. At regular intervals (every $N$ epochs), the process interrupts to perform remeshing:

Deformation: The current deformation quantity $Q_i$ is applied to the base mesh $M_{ri}$ to generate a deformed mesh $(V'_i, F_{ri})$ .
Supervision: The mesh is rendered, and a Cascaded Score Distillation (CSD) loss is computed against the text prompt. Gradients update $Q_i$ .
Remeshing: Every $N$ epochs, an isotropic remesher (based on Botsch & Kobbelt) is applied to the deformed geometry. This involves edge collapses, splits, flips, tangential smoothing, and projection to the surface.
State Interpolation: Crucially, the optimizer state (e.g., Adam's gradient accumulators) is interpolated from the old mesh vertices to the new mesh vertices using barycentric coordinates. This allows the optimization to resume coherently despite the discrete change in topology.

3. Coarse-to-Fine Schedule

The target edge length for the remesher follows a coarse-to-fine schedule. It starts with a larger target length (coarse resolution) and gradually decreases to a finer resolution over the course of the optimization. This schedule is critical for allowing large geometric growths to form before high-frequency details are added.

4. Localization

The method supports both local and global deformations. For local growths, a binary selection mask restricts deformation and remeshing operations to specific regions. The selection flag is treated as a vertex attribute and propagated through remeshing operations (collapses, splits, flips) to ensure only selected areas are modified.

Key Contributions

Remesh-Aware Optimization: A framework that couples continuous gradient-based optimization with discrete remeshing operations, enabling expressive geometric changes that fixed-connectivity methods cannot achieve.
Vertex-Based Deformation with Scale: An extension of vertex-normal deformation that includes optimizable scale parameters, enabling the growth of large parts and appendages while maintaining sharp details.
Optimizer State Interpolation: A mechanism to carry forward optimization history (gradient statistics) across remeshing events, ensuring stability and preventing the loss of momentum or symmetry during topology changes.
Coarse-to-Fine Remeshing Strategy: A specific scheduling of target edge lengths that facilitates the formation of large structures before refining details, avoiding the pitfalls of early high-resolution remeshing.

Results and Evaluation

The authors evaluate RADmesh on a variety of shapes (organic and manufactured) and text prompts, covering both local growths (e.g., adding wings to a cow, antlers to an owl) and global detailing (e.g., transforming a sphere into a pagoda or a chair into a balloon-themed seat).

Qualitative Performance: The method produces shapes with high geometric detail, symmetry, and clean boundaries. It successfully grows large parts from small selections and handles drastic structural changes (e.g., shrinking a car back into a pickup bed) without the artifacts seen in baseline methods.
Quantitative Metrics: On a dataset of 64 shape-prompt pairs, RADmesh achieves superior CLIP similarity scores and VQA scores compared to baselines like MagicClay, Instant3dit, Geometry in Style, and MeshUp.
Triangle Quality and Efficiency: RADmesh maintains higher average triangle face quality (FQ) than methods that do not remesh or those that reconstruct from proxy representations. Furthermore, it achieves these results with fewer faces (higher efficiency) than other remeshing-based approaches, demonstrating that the generated triangulation is tailored to the geometry rather than being a uniform upsampling.

Significance and Claims

The paper claims that RADmesh demonstrates the effectiveness of a purely mesh-based generative optimization approach. By integrating remeshing directly into the deformation loop with a robust vertex-based quantity and state interpolation, the method overcomes the limitations of fixed-connectivity editing. It enables the creation of complex, detailed 3D shapes with new appendages and structural reconfigurations that were previously unattainable with standard visual-loss-driven deformation. The authors emphasize that their approach avoids the lossy intermediate representations (like implicit fields or multi-view inpainting) used by other generative 3D methods, achieving desirable results directly on the mesh.

The work is modest about its limitations, noting that the runtime is dominated by the slow SDS supervision and that the current implementation requires manifold meshes due to the Poisson solve used in the global step. Future work is suggested to explore non-manifold operations and quad meshing.