💻 computer science

Pi-HOC: Pairwise 3D Human-Object Contact Estimation

Pi-HOC is a single-pass, instance-aware framework that achieves state-of-the-art accuracy and 20x higher throughput in dense 3D human-object contact estimation for multi-human scenarios by leveraging dedicated interaction tokens and a SAM-based decoder, while also enhancing 3D reconstruction and enabling referential prediction without additional training.

Original authors: Sravan Chittupalli, Ayush Jain, Dong Huang

Published 2026-05-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sravan Chittupalli, Ayush Jain, Dong Huang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you walk into a busy coffee shop. You see three people: one sitting at a table, one standing near the counter, and one pushing a stroller. They are all interacting with objects: cups, a counter, a stroller, and a chair.

If you asked a standard computer program to describe this scene, it might say, "There is a person touching a cup." But it wouldn't know which person is touching which cup, or if the person standing is actually holding the cup or just standing near it. It gets confused when there are multiple people and multiple similar objects.

This is the problem Pi-HOC solves. Think of Pi-HOC as a super-observant detective that doesn't just see "people" and "things," but sees specific pairs of people and things and figures out exactly where their bodies are touching.

Here is how it works, broken down into simple concepts:

1. The "Pairing" Game

Most old methods tried to guess contacts one person at a time, or they got overwhelmed when too many people were in the picture. Pi-HOC takes a different approach: It plays matchmaker.

The Setup: First, it spots everyone and everything in the photo (like a security camera tagging people and objects).
The Match: It then creates a specific "team" for every possible combination. If there are 3 people and 2 chairs, it doesn't just look at the chairs; it creates a specific team for "Person A + Chair 1," "Person A + Chair 2," "Person B + Chair 1," and so on.
The Token: For each of these teams, it creates a digital ID card (called an HO Token). This ID card holds the information about that specific pair.

2. The "Brain" (InteractionFormer)

Once the teams are formed, Pi-HOC uses a special brain called an InteractionFormer.

Imagine you have a group of detectives (the ID cards) sitting around a table looking at a giant photo of the coffee shop.
Instead of each detective looking at the whole photo, they focus on their specific team. The detective for "Person A + Chair 1" zooms in on Person A's legs and Chair 1's seat.
They talk to each other and look at the surrounding context to figure out the truth: "Is Person A actually sitting, or just standing near the chair?"
This happens all at once, very quickly, rather than asking one detective to solve the whole mystery alone.

3. The "Map" (SAM Decoder)

Once the brain decides who is touching what, it needs to draw the contact on a 3D model of the human body.

Think of the human body as a digital mannequin made of thousands of tiny dots (vertices).
Pi-HOC uses a tool called SAM (Segment Anything Model) to paint a "contact map." It doesn't just say "the hand is touching." It paints a specific red spot on the exact dot of the 3D mannequin where the hand meets the object.
It does this for every single pair in the photo simultaneously.

Why is this a big deal?

1. It's a Speed Demon
Previous methods were like a slow librarian who had to check every book one by one. If you added more people, the librarian got slower and slower.
Pi-HOC is like a high-speed scanner. It processes the whole scene in one pass. The paper claims it is 20 times faster than the best previous methods. It can handle a crowded room without getting confused or slowing down.

2. It Knows the Difference
If two people are holding identical red suitcases, old methods might mix them up, saying "a person is holding a suitcase" without knowing which person has which suitcase. Pi-HOC correctly assigns the contact to the specific person holding the specific suitcase.

3. It Fixes 3D Reconstructions
The paper shows a cool trick: If you use Pi-HOC to help rebuild a 3D scene from a photo, it can fix "floating hands."

The Problem: Sometimes 3D models look like people are hovering in mid-air because the computer didn't realize their hand was resting on a table.
The Fix: Pi-HOC says, "Hey, that hand is touching the table!" and pushes the 3D model down so the hand actually rests on the surface, making the scene look physically real.

4. It Listens to You
You can ask Pi-HOC a specific question in plain English, like "Show me the contact for the person sitting on the left." It will ignore everyone else and only show you the contact details for that specific person, without needing to be retrained for that specific question.

Summary

Pi-HOC is a new tool that looks at a photo, figures out every single person-object pair, and draws a precise map of exactly where they are touching in 3D space. It does this faster, more accurately, and with less confusion than anything before it, making it perfect for things like robotics, augmented reality, and understanding complex scenes.

Technical Summary: Pi-HOC

Problem Statement

Real-world human-object interactions (HOI) are inherently many-to-many, involving concurrent physical contacts between multiple humans and multiple objects. Existing methods for semantic contact estimation face three primary limitations:

Single-Human Assumption: Most state-of-the-art (SOTA) approaches, such as InteractVLM, are designed for single-human scenes and struggle to disambiguate multiple instances of the same object category or multiple interacting humans.
Dependency on Object Geometry: Many methods require access to object geometries (e.g., meshes or point clouds) in addition to the input image, limiting their applicability to real-world "in-the-wild" scenarios.
Scalability and Efficiency: Current SOTA methods leveraging large Vision-Language Models (VLMs) are computationally expensive, often requiring a separate forward pass for every human-object pair, resulting in poor inference throughput for complex scenes.

The core challenge is to perform dense, per-vertex 3D semantic contact prediction for all human-object pairs in a scene in a single pass, without relying on external object geometry or massive VLMs.

Methodology: Pi-HOC

Pi-HOC (Pairwise 3D Human-Object Contact) is a single-pass, instance-aware framework designed to predict dense contact for all human-object pairs simultaneously. The pipeline consists of four main stages:

1. Instance Detection and Pair Enumeration

Detection: A frozen DETR detector processes the input RGB image to identify human and object instances, outputting bounding boxes and query features.
Pairing: The system enumerates all candidate human-object pairs. Pairs with non-overlapping bounding boxes (IoU $\le$ 0) are pruned, as they cannot be in physical contact.
HO Token Construction: For each valid pair $(h, o)$ , a dedicated Human-Object (HO) token is constructed by concatenating the DETR query features of the human and the object, projected via a small MLP. This explicitly binds the specific human instance to the specific object instance.

2. InteractionFormer Refinement

Architecture: The core of the model is the InteractionFormer, a transformer encoder initialized with pretrained DINOv2 weights.
Mechanism: The HO tokens are jointly refined with image patch tokens through $L$ layers of self-attention.
Function: Through repeated self-attention updates, the HO tokens learn to attend to relevant image regions surrounding the specific human-object pair. This allows the tokens to implicitly encode the interaction type, involved body parts, and approximate contact location, making the representation "interaction-aware" without needing a large VLM.

3. Contact Decoding

Contact Presence Classification: A lightweight MLP classifier predicts the probability of contact for each pair. Pairs with low probability are filtered out to improve efficiency.
Dense Mask Prediction: For retained pairs, a SAM-based decoder predicts dense contact.
- Instead of directly decoding 6,890 SMPL vertices (computationally expensive), the model decodes HO tokens into multi-view 2D contact maps (pixel-wise likelihoods on 2D projections of the SMPL mesh).
- These 2D maps are lifted back to the 3D SMPL surface by pooling probabilities across visible views for each vertex, yielding a binary per-vertex contact mask $m_{h,o} \in \{0, 1\}^{N_v}$ .

4. Training Objectives

The model is optimized using a combination of losses:

2D Contact Loss: Focal and Dice losses on the projected 2D contact maps.
3D Contact Loss: Vertex-level focal loss and an $\ell_1$ sparsity term on the lifted 3D masks.
Contact Presence Loss: Binary cross-entropy for the contact classifier.
Auxiliary Losses: Optional scene and body-part segmentation losses (when ground truth is available) to sharpen representations.

Key Contributions

Multi-Human, Multi-Object Semantic Contact: The first framework to predict dense, per-vertex semantic contact for every human-object pair in a scene, addressing the many-to-many nature of real-world interactions.
Single-Pass, Instance-Aware Architecture: Pi-HOC forms HO pairs and refines them via an InteractionFormer and SAM-based decoder in a single forward pass, eliminating the need for repeated VLM queries.
State-of-the-Art Performance and Efficiency: The method achieves significant gains in accuracy and localization while operating at speeds approximately 20 $\times$ faster than previous SOTA methods (InteractVLM).
Downstream Applications: The paper demonstrates two applications:
- SAM-3D Refinement: Using predicted contacts as constraints to optimize test-time image-to-mesh reconstruction, correcting artifacts like "floating hands."
- Referential Contact Prediction: Enabling zero-shot prediction of contact for a specific human-object pair referenced by a natural language query, without additional training.

Experimental Results

Pi-HOC was evaluated on two datasets: MMHOI (multi-person, multi-object) and DAMON (single-person, multi-object).

MMHOI Performance: Pi-HOC achieved an F1 score of 61.09, outperforming the best InteractVLM variant (54.86) by 6.23 points (approx. 11.4% relative improvement). It also reduced the geodesic distance error by 13.3% (0.073 $\to$ 0.0633), indicating superior localization.
DAMON Performance: On single-person scenes, Pi-HOC improved F1 to 63.23 compared to InteractVLM's 59.48, while reducing geodesic error.
Inference Speed: Pi-HOC demonstrates superior scalability. For a scene with 11 human-object pairs, Pi-HOC runs at 2.3 FPS, whereas InteractVLM drops to 0.026 FPS (an 88.5 $\times$ speedup). Pi-HOC degrades gracefully as pair count increases, unlike the linear degradation of per-pair VLM approaches.
Ablation Studies:
- Initialization: DINOv2 initialization significantly outperformed CLIP and random initialization, attributed to DINOv2's stronger spatial features.
- Decoder: The SAM-based decoder significantly outperformed a standard MLP decoder, highlighting the importance of strong mask decoding for fine-grained localization.

Significance

The paper claims that Pi-HOC provides a practical, scalable solution for dense semantic contact prediction in complex, cluttered scenes. By moving away from single-human assumptions and heavy VLM dependencies, it enables real-time instance-level reasoning. The authors position this work as a structural prior for human-scene understanding, with immediate utility in augmented reality, robotics, and surveillance where accurate physical contact modeling is critical. The ability to perform referential contact prediction and improve 3D reconstruction without additional training further underscores the framework's versatility.