Deep Feature Deformation Weights

1. Problem Statement

Handle-based mesh deformation is a fundamental task in computer graphics, allowing users to edit 3D shapes using sparse control points (handles). However, existing approaches suffer from a trade-off between control/fidelity and semantic intelligence/speed:

• Traditional Methods (e.g., ARAP, Biharmonic): Offer fast, precise, and local control but rely on manually placed handles and energy minimization (e.g., Laplacian or rigidity energies). They lack semantic understanding, often failing to preserve global symmetries or structural coherence (e.g., deforming one chair leg without affecting the others). Furthermore, they require solving expensive optimization problems for every new handle configuration, hindering real-time interactivity.
• Data-Driven Methods (e.g., DeepMetaHandles, APAP): Provide semantic-aware edits (preserving structure and symmetry) but often sacrifice fine-grained control, speed, or require complex re-optimization for new handles. They also struggle with high-resolution meshes due to computational bottlenecks.

The Core Challenge: How to achieve a deformation framework that combines the speed and fine-grained control of traditional methods with the semantic awareness and global structure preservation of data-driven methods, while supporting real-time interaction on high-resolution meshes without re-optimization.

2. Methodology

The authors propose Deep Feature Deformation (DFD), a framework that learns deformation weights directly from deep visual features rather than solving energy minimization problems.

A. Core Concept: Feature Proximity Weights

Instead of optimizing weights via energy functions, DFD defines deformation weights based on visual similarity derived from pre-trained 2D models (specifically DINOv2).

• Mechanism: For a given mesh, a neural field $\Phi$ is trained to map 3D surface points to deep feature vectors.
• Weight Calculation: The weight $W_{ij}$ between a handle vertex $j$ and a target vertex $i$ is determined by the similarity of their feature vectors $Z_i$ and $Z_j$:
  $$W_{ij} = \max(F(Z_i, Z_j), 0)$$
  where $F$ is a similarity function (e.g., $1 - ||Z_i - Z_j||^2$).
• Result: Vertices with similar visual semantics (e.g., both are "chair legs") receive high weights, enabling automatic co-deformation without explicit constraints. This function is closed-form and requires no optimization during inference.

B. Barycentric Feature Distillation

A major bottleneck in prior data-driven methods is the computational cost of training on high-resolution meshes. The authors introduce Barycentric Feature Distillation to decouple training complexity from mesh resolution.

• Problem: Standard distillation only supervises features at pixels containing mesh vertices, wasting visual signal from triangle interiors.
• Solution: The method leverages the known geometry of the mesh to supervise features at every pixel intersected by a triangle using barycentric coordinates.
• Resolution Independence: The authors first downsample high-resolution meshes (e.g., >1M faces) using Quadric Error Metrics (QEM) to a coarse resolution. They distill the neural field on these coarse renders. Because the distillation is based on render resolution rather than vertex count, the resulting feature field is robust and can be applied to the original high-resolution mesh during inference. This reduces training time from hours to minutes.

C. Advanced Control Mechanisms

To ensure the method remains practical and controllable, DFD incorporates several extensions:

1. Locality Weighting: A user-defined parameter $\lambda$ modulates weights based on geodesic distance, allowing users to restrict deformations to local regions if global semantic changes are undesired.
2. Feature Space Constraints: Fixed points (anchors) can be applied. The system constrains not just the anchor vertex, but all vertices with similar deep features, preventing unwanted twisting or stretching of semantically similar parts.
3. Automatic Visual Symmetry Detection: The neural field allows the evaluation of candidate symmetry planes based on visual features (not just geometry). If a plane is detected as a visual symmetry, deformations applied to one side are automatically reflected to the other, preserving structural symmetry.

3. Key Contributions

1. DFD Weights: A novel paradigm for mesh deformation that uses deep feature proximity to define weights, eliminating the need for iterative optimization during deformation.
2. Barycentric Feature Distillation: A technique that enables the rapid training of feature fields on coarse meshes while maintaining high-fidelity performance on million-face meshes, solving the scalability issue of prior methods.
3. Unified Control: The framework supports both local control (via geodesic weighting) and global semantic control (via feature similarity), bridging the gap between traditional and data-driven approaches.
4. Automatic Symmetry: A method to detect and enforce visual symmetry planes dynamically, enabling symmetric edits from single-sided manipulation.
5. Real-Time Performance: A proof-of-concept system capable of handling meshes up to 1 million faces in real-time on consumer hardware.

4. Results and Evaluation

The authors evaluated DFD against baselines including ARAP, Biharmonic coordinates, NeuralMLS, APAP, and DeepMetaHandles (DMH).

• Qualitative Performance:
  • DFD successfully preserves global structure and symmetry (e.g., elongating all chair legs simultaneously) where traditional methods fail.
  • It produces smoother, more visually plausible deformations than APAP, which often suffers from noisy score-distillation signals.
  • It handles non-manifold geometries where other methods (like Biharmonic) fail.
• Quantitative Performance (Timing):
  • Preprocessing: DFD scales robustly to high resolutions (up to $10^7$ faces) due to barycentric distillation, whereas methods like Biharmonic fail at resolutions $>10^5$.
  • Bind/Inference: DFD is orders of magnitude faster than methods requiring re-optimization (ARAP, APAP) for new handles. It achieves sublinear scaling in bind and pose times.
• User Study:
  • In a study with 37 users, 82% of participants preferred DFD-T (translation variant) and 79% preferred DFD-A (affine variant) over all baselines for "most desirable" deformations.
  • In a realism study (23 users), DFD was selected 64% of the time as the most realistic and detail-preserving method.

5. Significance

This work represents a significant shift in mesh deformation paradigms:

• From Optimization to Inference: It moves the computational burden from the interactive phase (solving linear systems) to the preprocessing phase (distilling a neural field), enabling true real-time interactivity.
• Semantic Awareness: It demonstrates that deep visual features can effectively replace hand-crafted energy functions for defining deformation behavior, allowing AI to "understand" shape structure.
• Scalability: By decoupling training resolution from inference resolution, it makes data-driven deformation feasible for high-fidelity assets used in film and gaming, which were previously too computationally expensive for such methods.

The paper concludes that DFD successfully synthesizes the best aspects of classical and modern deformation techniques, offering a robust, fast, and semantically intelligent tool for 3D editing.