HOI-R1: Exploring the Potential of Multimodal Large Language Models for Human-Object Interaction Detection

Technical Summary of HOI-R1: Exploring the Potential of Multimodal Large Language Models for Human-Object Interaction Detection

Problem Statement
Human-Object Interaction Detection (HOID) aims to identify interactions between humans and objects in images, outputting instances as tuples of human bounding boxes, object bounding boxes, object classes, and interaction classes. Traditional HOID methods rely on a detector-centric paradigm, utilizing object detectors to generate proposals followed by a dedicated interaction head (often transformer-based) to predict labels. Recent state-of-the-art approaches increasingly incorporate Vision-Language Model (VLM) priors to handle long-tailed annotations. However, these methods introduce complex pipelines requiring additional modules, specific training objectives, and architectural coupling between detectors and interaction decoders, which hinders extensibility and reuse. Furthermore, the inherent reasoning capabilities of Multimodal Large Language Models (MLLMs) for structured HOID tasks—specifically the simultaneous reasoning about interactions and localization of pairs within a standardized protocol—remain under-explored.

Methodology
The authors propose HOI-R1, a framework that eliminates object detectors entirely, solving HOID end-to-end through natural language reasoning using MLLMs. The approach involves a two-stage training paradigm:

1. Language-Based Prediction & Prompt Design:
   Instead of bounding box regression, the model outputs HOI instances in a structured natural language format (JSON). The input prompt is designed with three components:

   • Task Instruction: Defines the model's role and provides exhaustive lists of valid object and interaction classes to constrain the output space and prevent hallucinations.
   • Reasoning Guidance: A "Thinking Process" section that decomposes the task into sequential steps (identifying humans, analyzing actions, determining interactions) to mirror human cognitive processes.
   • Format Example: A template requiring the output to include a reasoning chain (within <thought> tags) and final predictions (within <answer> tags), incorporating spatial coordinates directly into the text.

2. Supervised Fine-Tuning (SFT) with Thinking Distillation:
   To transfer task-specific knowledge, a teacher-student paradigm is employed. A powerful teacher model (GPT4o-mini) generates step-by-step reasoning traces for training images. The student MLLM is trained via SFT to predict both the teacher-generated reasoning traces and the ground-truth HOI triplets. This "thinking distillation" internalizes the logical process of HOI detection.

3. Reinforcement Learning (RL) with Custom Rewards:
   Following SFT, the model is aligned using Group Relative Policy Optimization (GRPO). The authors design a composite reward function consisting of four components to ensure structural, semantic, and geometric alignment:

   • Format Reward: Ensures the output contains required tags (<thought>, <answer>) and valid JSON keys (human, object, object class, verb class). It penalizes duplicate bounding boxes and incorrect key structures.
   • Object and Verb Label Rewards: Encourages accurate prediction of object and interaction classes by comparing predicted labels against ground truth using a drop-on-match strategy.
   • HOI IoU Reward: Uses the Hungarian algorithm to match predicted human-object pairs with ground-truth pairs based on the Intersection over Union (IoU) of both human and object boxes, providing a geometric alignment signal.

Key Contributions

• HOI-R1 Framework: The first MLLM framework to solve HOID end-to-end via natural language, removing the dependency on object detectors.
• Training Paradigm: Introduction of a two-stage training strategy combining SFT with thinking distillation (to inject HOI knowledge) and RL with specialized reward functions (to align with HOID metrics).
• Reward Design: Development of element-specific reward functions (format, label, and IoU) that guide MLLMs to produce structured, accurate, and spatially grounded predictions.

Experimental Results
Experiments were conducted on the HICO-DET dataset using multiple open-source MLLMs, including the Qwen-VL family (Qwen2.5-VL, Qwen3-VL) and Rex-Omni.

• Performance Gains: HOI-R1 demonstrates consistent improvements across models. Notably, applying HOI-R1 to Qwen2.5-VL-3B results in a 2x accuracy boost (from ~8.39 mAP to 18.33 mAP on the Full category under Default settings) with only 1 epoch of SFT and 40 steps of RL training.
• Comparison to SOTA: The HOI-R1 variant of Rex-Omni-3B achieves 27.00 mAP on the Known Object setting, outperforming the 72B-parameter Qwen2.5-VL-AWQ and traditional transformer-based methods like QPIC.
• Efficiency: The method converges significantly faster than traditional HOID methods, which often require hundreds of epochs, achieving strong performance with minimal training steps (1 epoch SFT + 40 RL steps).
• Ablation Studies: Removing the "thinking" component significantly degrades performance on rare categories, highlighting the value of reasoning. Removing the IoU reward causes a substantial drop (3.62 mAP), confirming the necessity of geometric alignment rewards.

Significance and Claims
The paper claims that HOI-R1 establishes new baselines for MLLM-based HOID, demonstrating that large language models possess the inherent potential to handle complex, structured visual reasoning tasks without specialized detection modules. By leveraging natural language reasoning and reinforcement learning, the framework simplifies the HOID pipeline while achieving competitive or superior performance compared to detector-centric approaches. The authors position this work as a step toward simplifying HOID architectures and unlocking the reasoning capabilities of MLLMs for fine-grained visual understanding.