💻 computer science

RegFormer: Transferable Relational Grounding for Efficient Weakly-Supervised Human-Object Interaction Detection

The paper introduces RegFormer, a transferable relational grounding module that enables efficient and accurate weakly-supervised Human-Object Interaction detection by leveraging spatially grounded signals to learn localized interaction cues, thereby eliminating the need for costly pair enumeration and achieving performance comparable to fully supervised models.

Original authors: Jihwan Park, Chanhyeong Yang, Jinyoung Park, Taehoon Song, Hyunwoo J. Kim

Published 2026-04-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jihwan Park, Chanhyeong Yang, Jinyoung Park, Taehoon Song, Hyunwoo J. Kim

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are looking at a busy photo of a park. You see dozens of people, dogs, frisbees, and benches. Your goal is to answer a simple question: "Who is doing what to whom?"

For example, you want to know: "Is that person feeding the dog?" or "Is that person riding the horse?"

This is a task called Human-Object Interaction (HOI) detection.

The Problem: The "Guessing Game" is Too Slow and Messy

In the past, computers tried to solve this by acting like a very confused detective who has to check every single possible combination in the photo.

"Is Person A holding the dog?" (Check)
"Is Person A holding the bench?" (Check)
"Is Person B holding the frisbee?" (Check)
"Is Person B holding the dog?" (Check)

If there are 10 people and 10 objects, the computer has to check 100 combinations. If the photo is crowded, it checks thousands. This takes forever (it's slow) and often leads to mistakes. The computer might think, "Oh, the person is holding the bench just because they are standing next to it," even though they aren't touching it. These are called false positives.

Furthermore, training a computer to do this perfectly usually requires a human to draw a box around every single person and object and label exactly what they are doing. This is incredibly expensive and time-consuming, like hiring an army of people to label every photo in the world.

The Solution: RegFormer (The "Smart Detective")

The paper introduces a new AI model called RegFormer. Think of it as a smart detective who doesn't need to check every single combination. Instead, it uses clues to figure out the answer quickly and accurately.

Here is how it works, broken down into three simple steps:

1. The "Grounding" Trick (Finding the Right Spot)

Usually, AI looks at the whole picture and gets confused. RegFormer is different. It uses a technique called "Spatial Grounding."

Imagine you are looking for a specific person in a crowd. Instead of looking at the whole crowd at once, you focus your eyes specifically on where that person is standing. RegFormer does this digitally. It looks at the image and asks, "Where is the human? Where is the object?" and then zooms in on just the space between them.

It creates a special "query" (a question for the AI) that is tied to that specific spot. This stops the AI from getting distracted by things in the background. It's like putting a spotlight on the interaction so the AI knows exactly what to look at.

2. The "Chemistry Check" (The Interactiveness Score)

This is the paper's secret sauce. Even if the AI finds a person and a dog, they might not be interacting. Maybe the dog is sleeping and the person is walking past.

RegFormer calculates a "Chemistry Score" (called an Interactiveness Score).

High Score: The person and the dog are close, looking at each other, or touching. The AI thinks, "Yes, they are interacting!"
Low Score: The person is far away, or the dog is looking the other way. The AI thinks, "Nope, ignore this pair."

This acts like a gatekeeper. It filters out the "fake" interactions before the AI even tries to guess what the action is. This solves the problem of the computer thinking a person is "holding" a bench just because they are standing near it.

3. The "Magic Transfer" (Learning Once, Using Twice)

Here is the coolest part.

Training: The AI is trained using only image-level labels. This means humans just tell the computer, "In this photo, there is a person riding a horse." They don't have to draw boxes or say which person or horse. It's like telling a student, "There is a story about a hero in this book," without pointing to the exact page.
Testing: When the AI is ready to work on a new photo, it can instantly switch to instance-level mode. It can now point to the exact person and the exact horse and say, "This specific person is riding this specific horse."

It's like a student who learns the concept of "riding a horse" from a storybook, and then, without any extra studying, can walk into a stable and immediately identify the specific rider and horse. This saves a massive amount of time and money because you don't need to retrain the AI for every new task.

Why is this a Big Deal?

It's Fast: Because it doesn't check every random combination, it works much faster than previous methods.
It's Accurate: By using the "Chemistry Score," it stops making silly mistakes about things that aren't interacting.
It's Cheap: It learns from simple, cheap labels (just the words "person riding horse") but can do the complex job of finding the exact people and objects.

The Bottom Line

RegFormer is like a smart, efficient detective that learns the concept of interactions from simple descriptions. It uses "spotlights" to focus on the right people and a "chemistry test" to ignore fake interactions. This allows it to understand complex scenes quickly and accurately, without needing expensive, detailed training data.

1. Problem Statement

Weakly-Supervised Human-Object Interaction (HOI) Detection aims to identify triplets $\langle \text{human}, \text{interaction}, \text{object} \rangle$ using only image-level annotations (i.e., knowing what interactions exist in an image, but not where the humans and objects are located).

Current approaches face two primary challenges:

Computational Inefficiency: Existing methods typically rely on off-the-shelf object detectors to generate thousands of candidate human-object pairs. They then process these pairs (often by cropping union regions) through a classification module. As the number of instances increases, the computational cost scales quadratically ( $N_h \times N_o$ ), making inference slow.
False Positives & Generalization: Processing union regions often includes irrelevant background or non-interacting objects, leading to spurious feature activation and false positives. Furthermore, many methods tightly couple the classifier with a specific detector, requiring retraining if the detector changes, and struggle to transfer reasoning from image-level to precise instance-level detection without additional training.

2. Methodology: RegFormer

The authors propose RegFormer (Relational Grounding Transformer), a lightweight, versatile module that unifies image-level HOI classification and instance-level HOI detection within a single framework. It operates in a sequential manner: first generating queries for human-object (HO) pairs, then predicting interactions.

Core Components

A. Pairwise Instance Encoder (Spatially Grounded Queries)
Instead of using static text embeddings for queries (as in standard Multi-Label Classifiers like ML-Decoder), RegFormer constructs Spatially Grounded Queries.

Mechanism: It calculates patch-level similarity between image features and text embeddings of human/object classes.
Aggregation: It applies a softmax to generate patch importance scores ( $\alpha$ ) and aggregates visual features based on these weights.
Result: This creates a query vector that implicitly encodes where the human and object are likely to be, bridging the gap between image-level semantics and spatial localization.

B. Interaction Decoder
The decoder performs cross-attention between the spatially grounded HO queries and the global image features. It predicts interaction scores for specific human-object pairs. This sequential approach (HO $\to$ I) avoids the need to enumerate all possible triplets simultaneously, significantly reducing computational overhead.

C. Interactiveness-Aware Learning (The Gating Mechanism)
To address the issue of non-interactive pairs causing false positives, RegFormer introduces an Interactiveness Score.

Local Interactiveness: Aggregates patch-level similarity scores within the region of a specific human-object pair to determine if they are interacting.
Masked Global Interactiveness: A novel component that combines local signals with global context. It uses the image-level patch importance scores but masks them with the specific instance regions. This prevents the model from assigning high scores to non-interactive instances that happen to have strong semantic alignment with the text (e.g., a person standing near a bike but not riding it).
Gating: The final interaction score is modulated by this interactiveness score, effectively suppressing predictions for non-interactive pairs.

D. Transfer to Instance-Level Detection (Zero-Shot Transfer)
A key innovation is the ability to transfer from image-level training to instance-level inference without additional training.

During inference, the model takes bounding boxes from an off-the-shelf detector.
It applies a Region-Aware Mask to the patch importance scores and interactiveness calculations, constraining the reasoning strictly to the detected instance regions.
This allows the model to perform efficient, precise instance-level reasoning using the weights learned during image-level supervision.

3. Key Contributions

Unified Framework: RegFormer is the first method to unify image-level HOI classification and instance-level detection in a single framework, eliminating the need for separate training pipelines.
Efficiency: By avoiding exhaustive enumeration of all instance pairs and using a single backbone forward pass, RegFormer is significantly faster (up to 128x faster in dense scenes compared to prior methods like ML-Decoder) while maintaining high accuracy.
Spatial Grounding & Interactiveness: The introduction of spatially grounded queries and the dual-component interactiveness score (local + masked global) effectively filters out non-interactive combinations, solving the false positive problem inherent in weakly supervised settings.
Detector Agnosticism: The model is trained independently of the object detector. It can be plugged into any detector (e.g., Faster R-CNN, DETR, H-DETR) without retraining, reducing detector bias and error propagation.

4. Experimental Results

The authors evaluated RegFormer on V-COCO and HICO-DET benchmarks.

Performance vs. Weakly Supervised SOTA: RegFormer significantly outperforms previous weakly supervised methods. On HICO-DET, it achieves 33.33 mAP (Full) with Faster R-CNN and 38.14 mAP with H-DETR, surpassing the previous state-of-the-art (Weakly HOI-CLIP) by a large margin (+2.19 to +15+ mAP depending on the backbone).
Competitiveness with Fully Supervised Models: In several configurations (e.g., using DINO-B and CLIP-B backbones), RegFormer achieves performance comparable to fully supervised models (e.g., 38.14 mAP vs. 34.69 mAP for HOICLIP), despite only using image-level labels during training.
Zero-Shot Generalization: In zero-shot settings (unseen HOI compositions), RegFormer demonstrates robust generalization, outperforming OpenCat (a large-scale pre-trained baseline) by 10.07 mAP in the challenging "Rare First Unseen Compositions" setting.
Efficiency: The method shows a marginal increase in inference time as the number of instance pairs grows, whereas baseline methods degrade significantly.

5. Significance

RegFormer represents a major step forward in scalable scene understanding. By decoupling the interaction reasoning module from the object detector and enabling direct transfer from image-level to instance-level reasoning, it offers a cost-effective, highly efficient, and highly accurate solution for HOI detection.

Its ability to achieve fully supervised-level performance with only weak supervision makes it highly practical for real-world applications where annotating bounding boxes and interaction triplets is prohibitively expensive. The "plug-and-play" nature of the module allows it to leverage improvements in object detection and vision-language models (like DINOv2 and CLIP) without requiring architectural retraining.