💻 computer science

PIAvatar: Physically Interactive Avatars via Deformation Gradient Decoupling

PIAvatar is a novel framework that enables physically interactive 3D human avatars with non-rigid deformable simulations by decoupling kinematic velocity from deformation gradients and integrating a skeletal framework for real-time pose tracking during complex avatar-avatar and avatar-environment interactions.

Original authors: Sang-Hun Han, Min-Gyu Park, Jisu Shin, Seunghyun Shin, Jin-Hwi Park, Hae-Gon Jeon

Published 2026-07-01

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Sang-Hun Han, Min-Gyu Park, Jisu Shin, Seunghyun Shin, Jin-Hwi Park, Hae-Gon Jeon

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a video game character. Usually, when that character runs, jumps, or gets hit by a ball, their body moves like a rigid statue or a rubber mannequin. If a heavy ball hits them, they might slide backward, but their belly doesn't jiggle, their hair doesn't fly, and their clothes don't flutter. They look real, but they don't feel real.

The paper PIAvatar introduces a new way to make these 3D characters feel like they have a real, squishy body that reacts to the world. Here is how they did it, explained simply:

1. The Problem: The "Stiff Rubber" Glitch

In traditional computer simulations, there is a conflict between what the character wants to do (like running a specific dance) and what physics wants to happen (like getting squished when hit).

Think of it like trying to walk through a crowd while wearing a suit made of stiff rubber. If you try to walk forward, the crowd pushes back. In old computer models, the "push back" (physics stress) fights against your "walk forward" (the animation). The result? The character gets stuck, wobbles weirdly, or can't reach the pose the animator intended. It's like trying to drive a car while someone is constantly pressing the brakes.

2. The Solution: The "Two-Track" System

The authors created a clever trick called Deformation Gradient Decoupling. Imagine the character's movement is split into two separate tracks:

Track A (The Driver): This is the "Kinematic Velocity." It's the pure instruction of "Move your arm here." The computer follows this instruction perfectly, ignoring any resistance.
Track B (The Passenger): This is the "Deformation." This is where the physics happens. When the character gets hit by a ball, the body squishes, jiggles, and bends.

By separating these two, the character can follow the dance moves perfectly (Track A) while simultaneously getting squished by a ball (Track B) without the squishing stopping the dance. It's like driving a car where the steering wheel (the pose) is connected to the engine, but the suspension (the squishy body) is connected to the road. You can steer exactly where you want, even if the road is bumpy.

3. The Skeleton: The "X-Ray" Tracker

When a character gets hit and their body deforms, it becomes hard for the computer to know where their joints are. It's like trying to find the bones inside a blob of jelly.

To solve this, the researchers put a hidden skeleton inside the character, like an X-ray view. Even when the jelly-like skin is squished and stretched, this internal skeleton keeps track of the joints. The computer uses this skeleton to instantly figure out the character's pose, even while they are being physically pushed around. This allows the system to know exactly how the character is moving in real-time.

4. The Engine: The "Snow and Sand" Simulator

To make all this squishing and hitting work, they used a physics engine called the Material Point Method (MPM). You might know this from video games that simulate snow or sand. It treats the character's body not as a solid block, but as millions of tiny particles (like a cloud of dust or a pile of sand) that can flow, stretch, and bounce.

Because of this, when the character jumps:

Their belly jiggles naturally.
Their hair flies around.
If they hit a heavy ball, the ball bounces off, and the character's body dents inward.
If two characters high-five, their hands actually deform on impact.

5. The "Smart Camera" Test

The researchers also tested if their simulation looked "real" to humans. They used an AI (a Vision-Language Model) that acts like a judge. They showed the AI videos of a character getting hit by a ball and asked, "Is this a light ball or a heavy ball?"

The AI learned to judge based on how much the character's belly squished. If the belly squished a lot, the AI said "Heavy." If it barely moved, the AI said "Light." The system then automatically adjusted the physics settings until the AI's judgment matched what a human would expect. This proved the simulation behaves in a way that feels physically correct to us.

Summary

PIAvatar is a new system that lets 3D characters have real, squishy bodies that react to the world (like getting hit by a ball) while still being able to perform specific actions (like dancing) without getting stuck or glitching. It does this by separating the "move" from the "squish" and using a hidden skeleton to keep track of the character's pose.

Technical Summary: PIAvatar

Problem Statement

Current 3D human avatar generation methods, while achieving high visual fidelity through pose-conditioned models (e.g., implicit models, 3D Gaussians, parametric bodies), fundamentally lack physical interactivity. These avatars are typically limited to kinematic animations and cannot interact physically with environments or other avatars. Recent attempts to incorporate physics face significant limitations:

Reinforcement Learning (RL) approaches often rely on simplified physics environments and primitive shapes (e.g., cylinders), failing to represent non-rigid deformations.
Simulation-driven methods (e.g., dressed avatar physics, Half-Physics) often support only one-way interactions (avatar affects object, but not vice versa) or fail to simulate surface deformations, limiting expressiveness.
Material Point Method (MPM) challenges: While MPM supports non-rigid deformation and bidirectional momentum transfer, applying it to user-defined avatar motions introduces two critical issues:
1. Stress Interference: In standard MPM, user-defined kinematic velocities are absorbed into the deformation gradient, generating unintended restorative stresses that hinder the avatar from reaching its target pose.
2. Pose Tracking: When an avatar undergoes non-rigid physical deformation, its pose is no longer directly trackable. Recovering the pose from a deformed shape is an ill-posed problem, typically requiring computationally expensive optimization or learning-based regression.

Methodology

The authors propose PIAvatar, a framework based on the Material Point Method (MPM) that enables physically consistent, bidirectional, and non-rigid interactions. The core innovation lies in decoupling kinematic control from physical deformation.

1. Kinematic Deformation Decoupling

To prevent user-defined motions from generating unwanted internal stresses, the authors decouple the kinematic velocity from the deformation gradient update:

Velocity Decomposition: The particle velocity $v_p$ is decomposed into a kinematic component $v^k_p$ and the physical component.
Separate Grid Transfer: A separate kinematic grid velocity field $v^k_i$ is computed via Particle-to-Grid (P2G) transfer using only the kinematic velocities.
Gradient Update: The kinematic deformation gradient $F^k_p$ is updated based on $v^k_p$ .
Stress-Free Update: The total deformation gradient $F_p$ is updated by extracting the kinematic part: $F_p \leftarrow F_p (F^k_p)^{-1}$ . This ensures that the user-defined motion drives the avatar without inducing restorative stress, while still allowing external forces to generate physical stress and deformation.

2. Skeleton-Based Pose Regression

To address the challenge of tracking poses during non-rigid deformation, the framework embeds a skeletal structure within the avatar:

Embedded Skeleton: A set of "bone particles" is embedded within the avatar's surface. External forces transfer to these particles, causing their positions to change.
Closed-Form Pose Estimation: The pose of each bone (rotation $R$ and translation $t$ ) is estimated in real-time using the Kabsch algorithm (least-squares optimization) to align canonical bone positions with their current deformed positions.
Kinematic Velocity Calculation: The kinematic velocity $v^k_p$ required for the next frame is derived from the incremental transformation between the current and next target poses, computed via Linear Blend Skinning (LBS) on the embedded skeleton. This allows for robust pose tracking even when the surface is physically deformed.

3. Implementation Details

Framework: Built upon the MPM framework (specifically extending PhysGaussian), supporting both 3D Gaussian and mesh-based avatars.
Materials: Supports heterogeneous material assignment (e.g., Neo-Hookean for bodies, corotated linear elastic for garments/hair) to simulate varying stiffness.
Optimization: Includes a module for automatic physical parameter optimization using Vision-Language Models (VLM) and Bayesian Optimization to tune parameters like mass and stiffness based on visual feedback.

Key Contributions

PIAvatar Framework: A novel MPM-based simulation system enabling both bidirectional physical interactions (avatar-avatar, avatar-object) and non-rigid deformations.
Deformation Gradient Decoupling: A method to separate user-defined kinematic velocity from the deformation gradient, ensuring intended motions are not hindered by stress while maintaining physical consistency.
Direct Pose Estimation: An embedded skeletal structure with closed-form pose regression (Kabsch algorithm) that allows for real-time pose tracking and kinematic velocity calculation during non-rigid physical interactions, avoiding expensive optimization or learning-based regression.
Comprehensive Evaluation: Demonstration of physically grounded interactions across diverse scenarios, including human-object collisions, human-human interactions, soft-tissue deformation, and loose garment simulation.

Experimental Results

The authors evaluated PIAvatar against a basic MPM baseline using Animatable Gaussians (AG) and SMPL-X avatars.

Quantitative Performance:
- Pose Accuracy: PIAvatar significantly outperforms the basic MPM in maintaining target poses. For SMPL-X, the Mean Squared Error (MSE) at 400 steps dropped from 0.248m (Baseline) to 0.039m (PIAvatar). Accuracy within 0.01m (Acc@0.01) improved from 3.6% to 58.3%.
- Pose Tracking: The Kabsch-based pose alignment errors remained negligible (floating-point precision level), confirming that the skeletal structure preserves pose consistency despite surface deformation.
Qualitative Results:
- Bidirectional Interaction: Demonstrated successful momentum transfer where heavy objects deform the avatar, and avatar actions (e.g., kicking, punching) alter object trajectories.
- Non-Rigid Deformation: Successfully simulated secondary effects like hair fluttering, belly jiggling, and soft-tissue deformation without explicit modeling of these specific features.
- Heterogeneous Materials: Showed distinct deformation behaviors for different body parts (e.g., stiff ankles vs. compliant hair) under collision.
- Loose Garments: Coupled with a thin-shell solver to simulate realistic clothing dynamics (fluttering, billowing) driven by body articulation.
VLM-Driven Optimization: A user study with 50 participants confirmed that the VLM-driven Bayesian Optimization framework successfully converged to physical parameters (mass, stiffness) that aligned with human perception of "heavy/light" and "stiff/soft."

Significance and Claims

The paper claims that PIAvatar resolves the fundamental trade-off between kinematic control and physical realism in avatar simulation. By decoupling the deformation gradient, the system allows avatars to follow precise user-defined motions while simultaneously exhibiting natural, non-rigid physical responses to external forces.

The authors emphasize that their approach achieves stress-free kinematic control and physically consistent interactions without relying on learning-based pose recovery or simplified rigid-body approximations. They position PIAvatar as a step toward bridging virtual and real-world experiences in AR/VR by enabling lifelike, bidirectional interactions where avatars can be physically pushed, deformed, and interact with complex environments in a unified simulation framework.

Limitations Acknowledged:
The authors note that the current system requires manual setup of avatar positions and pose sequences. They acknowledge that existing pose datasets (e.g., AMASS) lack torque control for external forces, and friction is not yet implemented, making posture maintenance under gravity difficult. They suggest future work could integrate PID control, RL, or large-scale physics engines (e.g., Genesis) to address these issues. Additionally, rapid collisions may cause sticking before stress develops, indicating a need for further learning frameworks to understand scene configurations.