PhysSkin: Real-Time and Generalizable Physics-Based Animation via Self-Supervised Neural Skinning

1. Problem Statement

Real-time physics-based animation is a critical goal for VR/AR, character animation, and interactive content. However, existing methods face significant limitations:

• Classical Subspace Methods: Traditional approaches (e.g., Linear Blend Skinning or FEM-based subspace) optimize a single mapping matrix tied to a specific mesh topology and resolution. They fail to generalize across different 3D shapes or spatial discretizations.
• Neural Subspace Methods: Recent neural approaches learn subspace mappings but typically require training a separate network for each individual object, preventing cross-object generalization.
• Neural Rigging/Skinning: Methods that infer skeletons and skinning weights from geometry often rely heavily on expensive, expert-annotated datasets and lack physical constraints, leading to deformations that are not physically plausible.

The Core Challenge: How to learn a physics-consistent deformation subspace mapping that generalizes across diverse 3D shapes, topologies, and discretizations directly from static geometries, without relying on annotated data or simulation trajectories.

2. Methodology

PhysSkin addresses these challenges through a three-part framework:

A. Skinning-Based Subspace Representation

Instead of optimizing a full-space simulation, PhysSkin models the 3D displacement field $\phi(X, z)$ as a weighted sum of affine transformations (handles), similar to Linear Blend Skinning (LBS):
$$\phi(X, z) = X + \sum_{i=1}^{m} W_i(X) Z_i \begin{bmatrix} X \\ 1 \end{bmatrix}$$

• Subspace Coordinates ($z$): Defined by handle transformations.
• Neural Skinning Fields ($W_i(X)$): Continuous functions learned by the network that output skinning weights for any point $X$ in space.
• Efficiency: By solving the physics equations (Newtonian dynamics) in the low-dimensional subspace ($m \ll n$) rather than the full space, real-time inference is achieved.

B. Neural Skinning Fields Autoencoder

To achieve generalization across shapes and discretizations, the authors propose a novel autoencoder architecture:

• Encoder (Shape Encoding): Uses a Transformer-based point cloud encoder (adapted from Michelangelo) to extract latent shape features ($F_s$) from surface point clouds. This allows the model to be mesh-free and discretization-agnostic.
• Decoder (Field Generation):
  1. Handle Latents: A cross-attention mechanism extracts handle-specific latent vectors ($F_h$) from the shape latents.
  2. Point-wise Features: Another cross-attention block aggregates information from handle latents to generate point-wise features for any spatial query point $X$.
  3. MLP Output: A ResNet-style MLP decodes these features into skinning weights $W(X)$.
  4. Orthogonalization: An Orthogonalization by Newton's Iteration (ONI) module is applied at the final layer to enforce orthogonality among skinning modes.
• Sampling Strategy: Unlike surface-only VAEs, PhysSkin samples cubature points both on the surface and inside the object's volume. This captures volumetric deformation cues essential for physical consistency.

C. Physics-Informed Self-Supervised Learning (PISSL)

Since no annotated data exists, the model is trained using a self-supervised strategy that enforces physical plausibility through three loss terms:

1. Potential Energy Minimization ($L_{pot}$): Minimizes the expected potential energy of random subspace coordinates to ensure the learned modes correspond to low-energy, physically plausible deformations. The training interpolates between linear elasticity and Neo-Hookean models for stability.
2. Spatial Smoothness ($L_{smooth}$): Penalizes high gradients in the skinning weight functions to ensure artifact-free, coherent deformations.
3. Orthogonality Constraint ($L_{orth}$): Enforces that different skinning modes are linearly independent.
   • Key Innovation: The authors introduce on-the-fly $\ell_2$-normalization of skinning modes to prevent numerical drift.
   • Conflict Resolution: They employ ConFIG (Conflict-aware Gradient Correction) to resolve destructive interference between the energy, smoothness, and orthogonality gradients, ensuring stable convergence.

3. Key Contributions

1. PhysSkin Framework: A generalizable, physics-informed framework for real-time animation that works across diverse 3D shapes and discretizations (Mesh, Point Cloud, Gaussian Splatting, NeRF) without retraining.
2. Neural Skinning Fields Autoencoder: A novel architecture combining Transformer encoders and cross-attention decoders to produce mesh-free, spatially continuous, and orthogonal skinning fields.
3. PISSL Strategy: A self-supervised learning approach that learns physics directly from static geometry using on-the-fly normalization and conflict-aware gradient correction, eliminating the need for annotated data or simulation trajectories.
4. Real-Time Performance: The method enables real-time physics-based animation by solving dynamics in a learned low-dimensional subspace.

4. Experimental Results

The authors evaluated PhysSkin on the ShapeNet and RigNet datasets, comparing it against baselines like Simplicits, RigNet, Anymate, and Puppeteer.

• Quantitative Metrics:
  • Orthogonality ($\Omega_{orth}$): PhysSkin achieved significantly lower values (e.g., 0.0033 vs. 0.5324 for RigNet baseline), indicating highly decorrelated and independent deformation modes.
  • Condition Number ($\kappa_{log}$): Demonstrated superior numerical stability (e.g., 1.0453 vs. 2.7997).
  • Spectrum Entropy ($H_{spec}$): Achieved near-perfect balance (0.9999), indicating expressive and uniform spectral energy distribution.
• Qualitative Results: Visualizations show that PhysSkin produces smoother, more physically consistent deformations compared to supervised methods that often suffer from artifacts or category-specific biases.
• Generalization: The single trained model successfully generalized to unseen objects, different topologies, and even 3D Gaussian Splatting (3DGS) representations without modification.
• Efficiency:
  • Animation Speed: PhysSkin operates at ~12–13 ms per step, whereas full-space FEM methods take ~1700–3000 ms and MPM methods take ~140–250 ms for similar complexity.
  • Training: The model is trained on 4x RTX 4090 GPUs, converging to stable, orthogonal skinning fields.

5. Significance

PhysSkin represents a major advancement in computer graphics and vision by bridging the gap between data-driven learning and physics-based simulation.

• Scalability: It removes the bottleneck of training per-object networks or requiring massive annotated datasets.
• Versatility: Its discretization-agnostic nature allows it to animate diverse representations (meshes, point clouds, neural fields) with a single unified model.
• Practicality: By enabling real-time, physically plausible animation on complex shapes without simulation overhead, it opens new possibilities for interactive applications in VR/AR and digital content creation.

The work demonstrates that purely geometric inputs, when processed through a physics-informed self-supervised lens, are sufficient to learn high-quality, generalizable deformation priors.