Technical Summary: CONTACTMIMIC

Problem Statement

Current humanoid whole-body loco-manipulation systems primarily rely on tracking keypoint trajectories (kinematic targets) to imitate human motion. However, for many useful interaction tasks—such as sitting on a chair, wiping a whiteboard, or pushing furniture—success is defined not merely by the robot's pose, but by explicit physical contact with the environment.

Existing trackers often reproduce the shape of a motion but fail to establish the necessary physical contact (e.g., waving a hand near a board rather than wiping it). Consequently, these systems are insufficient for task execution and typically require fine-tuning with task-specific rewards to achieve success. Furthermore, current methods lack the ability to explicitly control whether contact should be made or suppressed during deployment, limiting their versatility in scenarios where the same motion geometry requires different contact behaviors (e.g., sitting on a chair vs. squatting without touching it).

Methodology

The authors propose CONTACTMIMIC, a learning framework that enables explicit, test-time controllable contact behavior alongside keypoint tracking. The system is built on three core pillars:

1. Contact-Conditioned Policy

The policy, $\pi_\theta(a_t | p_t, \bar{k}_t, \bar{c}_t)$, takes three inputs:

• Proprioception ($p_t$): The robot's internal state.
• Keypoint Targets ($\bar{k}_t$): Reference trajectories for body parts.
• Binary Contact Labels ($\bar{c}_t$): A per-time-step, per-body-part map indicating whether contact with specific object parts is desired (1) or suppressed (0).

The policy outputs target joint angles, which are converted to torques via a PD controller. Notably, the system does not use explicit contact sensors as inputs; it relies on proprioception to infer the runtime contact state.

2. Reward Design

To train the policy to adhere to contact commands, the authors introduce contact-aware rewards alongside standard tracking and regularization terms:

• Contact Label Matching Reward ($r^{lm}_t$): Encourages the policy to match the commanded contact state (True Positive Rate and True Negative Rate) between robot body parts and object surfaces. Two variants are used: Balanced Accuracy (default) and TP-FP (for sparse-contact motions).
• Contact Distance Reward ($r^{cd}_t$):
  • Positive term: Encourages body parts intended to contact an object to remain close to the surface.
  • Negative term: Penalizes body parts intended not to contact an object if they get too close (within a threshold $\delta$).

3. Data Augmentation to Break Correlations

A critical challenge is that in raw human-object interaction data, keypoint trajectories and contact patterns are strongly correlated (a specific motion usually implies a specific contact pattern). If trained naively, the policy might ignore the contact command and infer contact solely from the keypoints.

To mitigate this, the authors propose an augmentation scheme that generates motion pairs with similar keypoint structures but differing contact labels:

• Contact-Label Flipping: Retains the original trajectory but flips the contact labels, penalizing the robot for making contact with the object.
• Object Removal: Removes the interaction object from the simulation and forces contact labels to zero while keeping the keypoint trajectory intact.
• Inflated Geometry: Inflates the collision mesh of the object during retargeting to force the robot to route around it, creating a perturbed trajectory where contacts naturally do not occur, which are then labeled as zero.

These augmented pairs force the policy to attend to the explicit contact command rather than inferring contact from geometry alone.

Key Contributions

1. Explicit Contact Control: A framework that allows a single policy to toggle between making or suppressing physical contact for the same keypoint trajectory, enabling fine-grained control over interaction (e.g., sitting with back support vs. without).
2. Task-Free Manipulation: The ability to complete object manipulation tasks (e.g., lifting a box) using only keypoint and contact trajectory tracking, without requiring task-specific rewards.
3. Data Engineering Strategy: The demonstration that breaking the correlation between keypoints and contacts via specific augmentation (flipping, removal, inflation) is necessary for learning genuine contact-conditioned behavior.
4. Proprioceptive Inference: The finding that the policy implicitly learns to infer the runtime contact state from proprioception alone, eliminating the need for external contact sensors at inference time.

Experimental Results

The framework was evaluated on the Unitree G1 humanoid robot across 10 diverse human-object interaction motions (e.g., wiping, sitting, leaning, kicking, picking up boxes) in both simulation and the real world.

• Contact Controllability: In simulation and real-world tests, the policy successfully followed contact commands. When commanded to contact ($\checkmark$), the robot established physical contact (e.g., erasing a whiteboard, leaning on a backrest). When commanded to suppress contact ($\times$), the robot maintained the same pose geometry but avoided contact.
• Comparison with Baselines: Compared to BeyondMimic (a state-of-the-art keypoint-only tracker), CONTACTMIMIC achieved significantly higher contact metrics (contact bodies and impulse) while maintaining comparable keypoint tracking accuracy (MPJPE). BeyondMimic failed to establish necessary physical contact for manipulation tasks.
• Ablation Studies: Removing the proposed data augmentation strategies resulted in a significant drop in contact controllability, confirming that the augmentation is essential for decoupling contact behavior from keypoint geometry.
• Real-World Transfer: The policy successfully transferred to the real Unitree G1 robot, demonstrating robust contact controllability across five different motions (e.g., wiping, sitting, leaning) with high success rates.

Significance and Claims

The paper claims that CONTACTMIMIC represents a step toward humanoids that interact with the world purposefully rather than incidentally. By introducing explicit contact conditioning, the authors provide a simple and general interface for expressing contact-rich tasks.

The significance lies in:

• Decoupling Geometry and Contact: Proving that contact behavior can be controlled independently of the motion's geometric trajectory.
• Eliminating Task-Specific Rewards: Showing that complex manipulation tasks can be learned by simply matching contact and keypoint trajectories, removing the need for hand-crafted task rewards.
• Scalability: The approach suggests that a single policy architecture can handle diverse interaction modes (contact vs. non-contact) simply by toggling a label, offering a path toward more versatile and adaptable humanoid robots.

The authors acknowledge limitations, including the current need to train separate policies per motion (rather than a universal tracker), reliance on high-quality human-object interaction datasets (HUMOTO), and evaluation on a single robot platform. Future work is suggested to address joint training across motions and the use of in-the-wild video data.