💻 computer science

Improving and Evaluating Hand-Object Interaction Detection

This paper introduces HOI-DETR, a state-of-the-art framework for Hand-Object Interaction detection that leverages an enhanced Co-DETR architecture and a comprehensive evaluation suite to achieve significant performance gains across multiple diverse datasets.

Original authors: Ahmad Darkhalil, Dima Damen, David Fouhey

Published 2026-06-17

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ahmad Darkhalil, Dima Damen, David Fouhey

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a cooking show. A standard computer vision program is like a guest who only knows the names of things: "That's a pan," "That's a cup," "That's a hand." It doesn't care what they are doing.

This paper introduces a new system called HOI-DETR that acts more like a human observer. It doesn't just name the objects; it understands the story of what is happening. It knows that the hand is holding the pan, and the pan is touching the stove. It even understands that if the hand is holding a spatula (the first object), and the spatula is flipping a burger (the second object), there is a chain of action connecting all three.

Here is a breakdown of what the authors did, using simple analogies:

1. The Problem: The "Name-Only" Detective

Previous computer programs were like detectives who only had a list of names. If they saw a pan, they said "Pan." They didn't realize that if the pan was just sitting on the counter, it was background noise, but if a hand grabbed it, it became a "tool."

The Flaw: They couldn't tell the difference between a pan being used and a pan just sitting there. They also often missed the connection between the hand and the tool, or the tool and the food.

2. The Solution: HOI-DETR (The Storyteller)

The authors built a new AI model called HOI-DETR. Think of this model as a director on a movie set who assigns roles to actors.

Role 1 (The Hand): The actor.
Role 2 (The 1st Object): The prop the actor is holding directly (like a knife).
Role 3 (The 2nd Object): The target the prop is acting on (like the food being cut).

The model looks at an image and asks: "Who is touching whom?" It draws invisible lines connecting the hand to the tool, and the tool to the target. It can even handle complex scenes, like one hand holding three cards at once, or two people shaking hands.

3. The "Training Camp" (Fixing the Data)

To teach this new model, the authors had to clean up the "textbooks" (datasets) they used.

The Messy Textbook: They found that the old dataset (called Hands23) had a lot of confusion. Sometimes, the same object was labeled twice by mistake (like drawing two boxes around the same apple).
The Cleanup: The authors acted like editors, going through thousands of images to remove duplicate boxes and fix the connections. This made the training data much cleaner, which helped the model learn faster and more accurately.
The New Challenge: They also created a new test based on high-definition videos (from the HD-EPIC dataset). This is like moving from a static photo quiz to a live-action movie test, where the model has to keep track of objects as they move and change over time.

4. The Results: Beating the Competition

The authors tested their new "Storyteller" against the old "Name-Only" detectives.

Accuracy: HOI-DETR was significantly better. On some tests, it improved accuracy by more than 20 percentage points. That's like going from getting a C to an A+ on a test.
Video Consistency: Even though HOI-DETR looks at one frame at a time (like a photo), it was better at keeping track of objects in videos than other models that were specifically trained on video. It didn't "flicker" or lose track of the object as easily.
Generalization: The model was so good that it could look at completely new types of videos (like biology experiments) it had never seen before and still understand what was happening.

5. Why It Matters (According to the Paper)

The paper explains that understanding these interactions is the first step for many other tasks.

3D Reconstruction: If you want to build a 3D model of a hand holding a cup, you first need to know exactly where the hand and cup are and how they touch.
Robotics: For a robot to pick up a tool and use it, it needs to understand the chain of interaction (Hand → Tool → Object).
Action Recognition: To understand what a person is doing, you need to see the relationship between their hands and the objects they are touching.

Summary

In short, the authors built a smarter AI that doesn't just see "objects"; it sees relationships. They cleaned up the training data, built a new model that understands chains of action (Hand → Tool → Target), and proved it works better than anything else currently available, even on difficult, moving video scenes. They made their code and data available for others to use and improve upon.

Technical Summary: Improving and Evaluating Hand-Object Interaction Detection

Problem Definition

Hand-Object Interaction (HOI) detection is a critical prerequisite for tasks ranging from 3D reconstruction and robotics to action perception. Unlike standard object detection, which identifies semantic classes (e.g., "pan") regardless of context, HOI detection requires understanding dynamic, state-driven roles. An object's classification depends on its interaction: a pan may be background, a "1st object" (directly grasped), or a "2nd object" (manipulated via a tool). Furthermore, existing literature suffers from uneven progress: some methods detect hands but ignore held objects, others track objects without identifying the holding hand, and many rely on outdated detectors. Additionally, evaluation has historically been limited to image-based metrics (mAP), neglecting spatiotemporal consistency despite the prevalence of video-based applications.

Methodology: HOI-DETR

The authors propose HOI-DETR, a framework that extends the Co-DETR architecture to jointly detect hands, 1st objects (directly interacted with), and 2nd objects (interacted with via tools), while predicting interaction links.

Architecture

Backbone and Encoder: The model utilizes a Vision Transformer (ViT) backbone and a multi-scale deformable attention encoder to map input images to latent feature tokens. It employs auxiliary heads for one-to-many supervision during training to stabilize the encoder.
Transformer Decoder: Unlike standard Co-DETR which predicts semantic categories, the decoder predicts role-based classes: {0: hands, 1: 1st object, 2: 2nd object, 3: background}. This allows the same physical object to change class based on its relationship to a hand. The decoder uses Hungarian matching to align predictions with ground truth.
Interaction Module: A novel component added to the decoder outputs. It takes pairs of decoder latents (representing bounding boxes) and classifies their interaction relationship via a Multi-Layer Perceptron (MLP).
- Valid Pairs: The model explicitly models hand → 1st object and 1st object → 2nd object chains. It excludes direct hand → 2nd object pairs by definition.
- Loss: The interaction module is trained using a focal loss on valid pairs, combined with standard detection losses (classification, L1 regression, GIoU).

Training Strategy

HOI-DETR is trained end-to-end. It is initialized from a Co-DETR model pre-trained on COCO and fine-tuned on a refined version of the Hands23 dataset. The training objective combines detection losses with the proposed interaction supervision, weighted by hyperparameters $\lambda_1, \lambda_2, \lambda_3$ .

Key Contributions

1. Methodological Advancement

HOI-DETR is the first transformer-based HOI detection model. By integrating an interaction module into a modern detection architecture, it achieves state-of-the-art performance. The model operates on single frames but demonstrates superior spatiotemporal consistency compared to video-based methods.

2. Data Refinement and New Benchmarks

The authors address data inconsistencies and introduce new evaluation standards:

Refined Hands23: The original Hands23 dataset contained duplicate annotations for objects involved in bimanual interactions. The authors built a pipeline to remove duplicates and remap interaction links, correcting inconsistencies in 56.2% of overlapping images in the validation set.
HD-EPIC-HOI: A new video-level benchmark derived from the HD-EPIC dataset. It focuses on challenging clips centered around object contact and release states, providing dense annotations for evaluating spatiotemporal consistency.
Comprehensive Suite: The evaluation spans four diverse datasets (Hands23, HD-EPIC-HOI, HOIST, FineBio), covering exocentric/egocentric views, images/videos, and out-of-distribution scenarios (e.g., biology experiments in FineBio).

3. Evaluation Metrics

Beyond standard frame-level mAP and F1 scores, the paper introduces rigorous spatiotemporal metrics:

Video-AP: Extends AP to the video level by treating trajectories as tubes and matching them via spatio-temporal IoU.
Long-Term Temporal Consistency (LTC): Measures the maximum uninterrupted duration a model tracks an object correctly, normalizing for video length to prevent bias toward longer clips.

Experimental Results

Frame-Level Performance

HOI-DETR significantly outperforms the previous state-of-the-art (Hands23 detector) and video-based methods (HOIST):

Hands23: Achieves an overall mAP of 86.1% (refined data) compared to 63.6% for the original Hands23 detector.
FineBio (Out-of-Distribution): Demonstrates strong generalization with an overall AP of 71.2% versus 50.2% for the baseline.
Interaction Accuracy: Reduces interaction error rates by 52% compared to the state-of-the-art.
Zero-Shot Hand Detection: Despite being optimized for HOI, HOI-DETR achieves the best pure hand detection performance on 4 out of 6 evaluated datasets, outperforming specialized detectors like WiLOR and MediaPipe in zero-shot settings.

Spatiotemporal Consistency

On the HD-EPIC-HOI benchmark, HOI-DETR achieves a Video-AP of 60.2 and an LTC of 61.0.

It outperforms the video-based HOIST model by 44.1 Video-AP points and 33.8 LTC points.
It outperforms the frame-based Hands23 detector by 33.4 Video-AP points.
These results indicate that video-level training alone does not guarantee temporal consistency; the representational capacity of HOI-DETR and the quality of the training data are the primary drivers of performance.

Ablation Studies

Pretraining: Critical for performance; removing pretraining drops AP by 33.4 points.
Interaction Module: Replacing the learned module with spatial heuristics (nearest box or max IoU) degrades interaction F1 scores significantly (e.g., -22.8% for 1st→2nd object), proving that spatial proximity is insufficient for contextual chain reasoning.
Training Strategy: End-to-end joint training significantly outperforms decoupled training, with interaction F1 scores dropping by ~14-19% when decoupled.

Significance and Claims

The paper claims that HOI-DETR establishes a new state-of-the-art for hand-object interaction detection by unifying detection and interaction modeling within a modern transformer framework. The authors emphasize that their work provides not just a better model, but a more rigorous evaluation ecosystem. By refining the Hands23 dataset and introducing the HD-EPIC-HOI benchmark, they highlight that previous evaluations may have been inflated by data inconsistencies and lacked temporal metrics. The results demonstrate that a frame-based approach, when properly architected and trained, can surpass video-based methods in both accuracy and temporal consistency, offering a robust foundation for downstream tasks like 3D reconstruction and robotic grasping.