Technical Summary: RegHead

Problem Statement

The creation of animatable 3D non-humanoid head avatars is critical for AR, social media, and gaming, yet remains underexplored. A practical animation interface requires semantic blendshapes: a fixed vocabulary of interpretable expressions that share vertex correspondence across diverse identities. This allows for low-dimensional control and cross-identity retargeting without solving new correspondence problems at runtime.

However, constructing these blendshape sets for non-humanoid heads faces three primary bottlenecks:

1. Scarce Supervision: Current image/video generation models struggle to produce the same predefined expression (e.g., "frown") consistently across different non-humanoid identities, making it difficult to obtain large-scale 2D observations with consistent semantic labels.
2. Lack of Correspondence: Modern 3D/4D generation methods produce high-quality expression-specific surfaces but typically lack vertex correspondence across expressions, rendering them unusable as a direct blendshape basis. Recovering correspondence via optimization-based non-rigid registration is computationally expensive and difficult to scale.
3. Localized Deformations: Non-humanoid facial expressions often involve highly localized, species-specific deformations (e.g., specific jaw articulation or eyelid movements) that are poorly captured by generic global deformation models.

Existing methods either rely on heavy optimization pipelines (slow) or parametric priors like 3DMM (which fail when geometry deviates significantly from human anatomy).

Methodology

RegHead proposes a feed-forward framework to convert unregistered expression meshes into a corresponded semantic blendshape basis in a single pass. The pipeline consists of three main stages:

1. Large-Scale Dataset Construction

To address the lack of consistent semantic labels, the authors define a fixed expression vocabulary $\mathcal{E}$ using a small library of high-quality, artist-rigged non-humanoid assets. They fine-tune an image editing model on paired renderings from this library to perform controlled edits from a neutral reference to target expressions. This model is applied to text-to-image generations to scale the vocabulary to approximately 20,000 non-humanoid identities. Expression-specific 3D surfaces are reconstructed from these images using image-conditioned 3D/4D generation pipelines, resulting in raw meshes $\{\tilde{M}_t\}$ that are semantically consistent but lack vertex correspondence.

2. Dense Stochastic Anchor Motion Representation

To handle diverse topologies without predefined skeletons or templates, RegHead introduces a dense stochastic anchor representation:

• Stochastic Anchors: Instead of optimizing fixed anchor coordinates, the model resamples a set of deformation anchors $\mathcal{A}$ on the neutral mesh at every training iteration. This encourages the model to learn anchor-layout-invariant predictions.
• Query Points: A denser set of query points $\mathcal{Q}$ is sampled on the neutral shape.
• Voxel Tokens: Expression meshes are voxelized to create multi-view unprojection-based voxel tokens $\mathcal{V}_t$, which include 8-channel latent codes and unprojected semantic segmentation masks (e.g., for eyes).
• Propagation: The model predicts per-anchor similarity transforms $\mathcal{T}_t$. These are propagated to the dense query points via Linear Blend Skinning (LBS) using precomputed skinning weights based on inverse Mahalanobis distances to the nearest anchors. This enables sparse-to-dense deformation that is efficient yet expressive.

3. Feed-Forward Registration Model

The core novelty is a coarse-to-fine correspondence-aware matching network that predicts the deformation from the neutral mesh to each target expression:

• Global Matcher: Uses cross-attention to aggregate long-range context from global voxel tokens into anchor tokens. It predicts a coarse deformation to bring anchors into the correct "basin" for local matching, addressing large-support motions (e.g., jaw articulation).
• Structured Local Matcher: For each coarsely warped anchor, the model gathers a structured, multi-radius stencil neighborhood from the voxel tokens (rather than simple kNN). It uses local cross-attention to refine the anchor token and predict a residual transform. This stage is designed to capture fine-grained, localized facial changes.
• Training: The model is trained without ground-truth point correspondences. It uses a differentiable rendering loss (comparing rendered RGB and segmentation of predicted points vs. target points) and an auxiliary Chamfer distance loss on the coarse anchor positions to ensure geometric consistency.

4. Real-Time Retargeting

For application, a small MLP maps human face tracking coefficients (from a standard 3DMM fit) to the weights of the non-humanoid blendshape vocabulary. This allows real-time retargeting from human performance to non-humanoid characters without per-frame optimization.

Key Contributions

1. Dataset: Construction of a large-scale dataset (~20k identities) of non-humanoid heads with a fixed semantic expression vocabulary, enabling scalable study of blendshape generation.
2. Motion Representation: Introduction of dense stochastic anchors, a topology-agnostic representation for localized non-humanoid facial deformation that does not rely on predefined bones, skeletons, or shared templates.
3. Feed-Forward Registration: A novel registration model that converts unregistered meshes into corresponded blendshapes in a single pass, trained without ground-truth correspondences or deformation fields.
4. Performance: Demonstration of higher fidelity and significantly faster inference (orders of magnitude) compared to optimization-based baselines, enabling real-time retargeting.

Experimental Results

The authors evaluate RegHead against ActionMesh (feed-forward), V2M4 (optimization-based), and T2Bs (optimization-based alignment).

• Quantitative Metrics: RegHead achieves superior performance in rendered appearance (PSNR: 23.84, SSIM: 0.9078, LPIPS: 0.0551) and visible geometry (Chamfer Distance: 0.00358, Silhouette IoU: 0.9865) compared to all baselines.
• Speed: RegHead runs in ~5 seconds per identity, whereas optimization-based methods (V2M4, T2Bs) take ~100s to ~1800s.
• Qualitative Analysis: RegHead better captures fine-grained, spatially localized expression changes (e.g., eyelid closure) while maintaining global structure. Baselines often underestimate localized motion or exhibit artifacts in subtle regions.
• Ablation Studies:
  • Removing stochastic anchors (using fixed anchors) degrades performance, confirming the benefit of layout invariance.
  • Reducing anchor density significantly lowers fidelity, proving the necessity of dense anchors for fine details.
  • Removing the local matching stage causes substantial degradation, validating it as the primary driver for fine expression deformation.
  • Using structured stencil neighborhoods outperforms standard kNN neighborhoods.

Significance and Claims

The paper claims that RegHead addresses the core challenge of generating semantically consistent, corresponded assets for non-humanoid heads in a scalable, feed-forward manner. By avoiding template-based anatomical constraints and heavy optimization, the framework enables the creation of high-fidelity, animatable avatars that support real-time control and cross-identity retargeting. The authors note that the method is specifically designed for fine-grained, species-dependent deformations, a gap left by existing articulated motion or whole-object generation methods.

Limitations: The authors acknowledge that RegHead may underfit identities with anatomy far removed from common head structures (e.g., insect-like or long-tail species with ambiguous facial regions) where semantic correspondence is weak. Additionally, the final blendshapes are subject to artifacts present in the upstream unregistered meshes.