💻 computer science

JointHOI: Jointly Generating Contact Maps Enhances Hand Object Interaction Generation

JointHOI is a single-stage diffusion framework that jointly generates 3D hand-object motions and dynamic contact maps from text descriptions, significantly improving physical plausibility and temporal stability by enforcing consistency between the generated contact cues and motion geometry.

Original authors: Mingyeong Song, Jungbin Cho, Jisoo Kim, Ananya Bal, Kartik Sharma, Youngjae Yu, Laszlo A. Jeni, Junhyug Noh

Published 2026-07-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Mingyeong Song, Jungbin Cho, Jisoo Kim, Ananya Bal, Kartik Sharma, Youngjae Yu, Laszlo A. Jeni, Junhyug Noh

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to pick up a coffee mug, open a jar, or put on glasses, but you can only do it by typing a sentence like "Pick up the mug with both hands." The goal is for the computer to generate a 3D video of a pair of hands doing exactly that.

The problem is that computers are terrible at "touch." They are great at moving arms around, but they often forget that hands need to actually touch the object. This leads to weird glitches: fingers floating in the air above the mug, or worse, fingers passing through the mug like ghosts.

JointHOI is a new computer program designed to fix this. Here is how it works, explained simply:

1. The Old Way: The Assembly Line vs. The Orchestra

Previous methods tried to solve this in steps, like an assembly line.

Step 1: The computer guesses where the hand should touch the object.
Step 2: It tries to move the hand to that spot.
Step 3: If the hand passes through the object, it tries to fix it later.

This is like trying to bake a cake by mixing the batter, then baking it, and then trying to frost it perfectly after it's already burnt. It often leads to messy results.

JointHOI changes the approach. Instead of an assembly line, it's like a conductor leading an orchestra. It doesn't just tell the hands where to go; it tells the hands and the "touch" what to do at the exact same time. It generates the movement of the hands and the "map of where they are touching" simultaneously.

2. The Secret Ingredient: The "Touch Map"

The paper introduces a clever trick. Instead of just guessing "touch" or "no touch" (like a light switch that is either on or off), JointHOI creates a dynamic distance map.

Think of this map like a heat map on a weather app.

Instead of just saying "It's raining," it shows exactly how far the rain is from the ground at every single second.
As the hand gets closer to the object, the map shows the distance shrinking smoothly.
As the hand pulls away, the distance grows.

By generating this "heat map" of touch alongside the hand movement, the computer learns the relationship between moving and touching much better. It learns that "when the hand moves this way, the touch distance must get smaller."

3. The "Self-Correction" (Contact Inner Guidance)

Even with a great plan, sometimes the computer makes a mistake during the final generation. Maybe the hand moves a tiny bit too far and starts to pass through the object.

JointHOI has a built-in self-check system called "Contact Inner Guidance."

Imagine you are drawing a picture of a hand holding a ball. As you draw, you constantly check: "Does the hand actually look like it's holding the ball, or is it floating?"
If the computer sees the hand is floating or passing through, it gently nudges the drawing back into place while it is still being created.
It does this without needing a second computer or a human to fix it later. It's like a GPS that reroutes you instantly if you take a wrong turn, rather than waiting until you hit a dead end to tell you.

4. The Results: No More Ghost Hands

The researchers tested this on two big datasets of people interacting with objects (GRAB and ARCTIC).

Better Touch: The hands actually touch the objects without floating or passing through them.
Better Meaning: If you type "pick up the scissors," the computer understands the action better than previous methods.
Faster: Because it does everything in one step (like a single orchestra rehearsal) rather than multiple steps (like a factory line), it is much faster to generate these videos.

Summary

JointHOI is a new way for computers to learn how to make realistic videos of hands interacting with objects. It stops treating "touch" as an afterthought and instead treats it as a core part of the movement, generating the motion and the touch map together. It also has a built-in "self-check" to ensure the hands don't float or phase through objects, resulting in much more natural and physically correct interactions.

Technical Summary: JointHOI

Problem Statement
Text-driven Hand–Object Interaction (HOI) generation is critical for robotics and immersive applications (AR/VR), yet producing physically plausible interactions remains a significant challenge. While recent methods can generate natural-looking motions, they often fail to maintain precise contact consistency. Small errors in contact timing or proximity manifest as conspicuous artifacts, such as fingers floating above objects or interpenetrating them. Existing approaches typically rely on multi-stage pipelines (e.g., predicting a grasp followed by motion refinement) or implicit latent priors. These methods often utilize static, binary contact cues that fail to capture the spatiotemporally structured nature of contact, which evolves continuously as hands approach, grasp, manipulate, and release objects. Consequently, these pipelines struggle to model time-varying contact evolution, leading to error propagation and a lack of physical grounding.

Methodology: JointHOI
The authors propose JointHOI, a single-stage joint diffusion framework designed to generate 3D hand–object motion and dynamic contact maps simultaneously from text prompts. The core innovation is treating contact not as a separate conditioning signal, but as an "inner modality" of the motion itself.

Dynamic Contact Map Representation:
Instead of binary or static contact maps, JointHOI represents contact as a continuous, distance-based field. The object surface is discretized using a fixed set of $N_c=1024$ anchor points (via Farthest Point Sampling). For each frame, the model predicts the minimum Euclidean distance from each anchor to the surface of the left and right hands. This yields a dense, temporally evolving signal ( $C \in \mathbb{R}^{L_{max} \times N_c}$ ) that tracks proximity frame-by-frame, avoiding threshold sensitivity and providing a smooth signal for gradient-based guidance.
Joint Diffusion Model:
The framework employs a Transformer-based diffusion model that operates on a unified sequence token $y_l$ . This token concatenates the kinematic states of the left hand, right hand, and object, along with the dynamic contact maps for both hands.
- Architecture: The model is conditioned on text embeddings (CLIP) and object geometry features (PointNet). It uses prefix conditioning to inject these signals into the sequence.
- Training: The denoiser is trained to reconstruct the clean joint sequence (motion + contact) from noisy inputs using a standard $L_2$ reconstruction loss. By generating both modalities simultaneously, the model learns the coupling between motion and contact (e.g., decreasing distances during approach, increasing during release) directly through self-attention mechanisms within a shared latent space.
Contact Inner Guidance (CIG):
To ensure physical plausibility at inference, the authors introduce Contact Inner Guidance (CIG), a training-free, classifier-based sampling strategy.
- Mechanism: During the denoising process, CIG enforces consistency between the contact maps explicitly generated by the model ( $\hat{C}$ ) and the contact maps implicitly derived from the synthesized geometry ( $\bar{C}$ ).
- Energy Function: A contact-consistency energy $\mathcal{L}_{cig}$ is computed as the $L_1$ distance between the log-scales of the generated and derived contact distances. The log-scale emphasizes fine-grained near-contact regions where physical violations are most critical.
- Guidance: The gradient of this energy with respect to the noisy variable is used to steer the denoising trajectory, reducing artifacts like penetration and floating without requiring secondary training stages or post-processing refinement.

Key Contributions

Single-Stage Joint Framework: JointHOI is the first text-driven HOI method to jointly generate 3D motion and dynamic, distance-based contact maps in a single diffusion step, eliminating the complexity and error accumulation of multi-stage pipelines.
Dynamic Contact Representation: The introduction of a temporally dense, distance-based contact map that captures the continuous evolution of hand–object proximity, serving as a more informative signal than static binary cues.
Contact Inner Guidance (CIG): A novel inference-time sampling strategy that leverages co-generated contact maps to guide motion generation. It enforces geometric consistency without external neural networks or post-processing, significantly reducing physical artifacts.
Comprehensive Evaluation: Extensive experiments and ablation studies validating the effectiveness of joint modeling and contact guidance.

Experimental Results
The method was evaluated on the GRAB (rigid objects) and ARCTIC (articulated objects) datasets.

Physical Plausibility: JointHOI achieved the lowest Interpenetration Volume (IV) and Depth (ID) among all synthesized methods. On ARCTIC, it reduced penetration volume by approximately half compared to the strong baseline Text2HOI.
Semantic Adherence: The model demonstrated superior text adherence, achieving the highest Top-3 action accuracy and the lowest Fréchet Inception Distance (FID) on both datasets, indicating high motion realism and semantic alignment.
Efficiency: JointHOI is significantly more efficient than multi-stage baselines (e.g., DiffH2O, LatentHOI), with inference times comparable to single-stage baselines like MDM, despite the added CIG step.
Ablation Studies: Incremental ablations confirmed that moving from a two-stage to a one-stage joint model, incorporating dynamic contact maps, and applying CIG each contributed distinct improvements in semantic fidelity and physical consistency.

Significance and Claims
The paper claims that JointHOI addresses the fundamental disconnect between motion synthesis and physical contact in HOI generation. By treating contact as an inner modality, the framework explicitly models the spatiotemporal evolution of hand–object proximity, allowing the model to internalize physically grounded co-variation patterns. The authors assert that this approach yields more natural, stable, and physically plausible interactions, effectively mitigating common artifacts like floating and interpenetration that plague prior methods. The work demonstrates that joint generation, coupled with inference-time consistency guidance, provides a robust path toward high-fidelity text-driven interaction synthesis for robotics and immersive applications.