A Unified Benchmark for HOI Evaluation across Vision-Language Models and HOI-Specific Methods

1. Problem Statement

Human-Object Interaction (HOI) detection has traditionally been dominated by task-specific models trained on datasets like HICO-DET. With the emergence of large generative Vision-Language Models (VLMs), a critical question arises: Can standalone VLMs perform HOI detection competitively against specialized HOI methods?

However, evaluating these two paradigms together is currently impossible due to fundamental flaws in existing benchmarks (e.g., HICO-DET, V-COCO):

• Incomplete Annotations & Exact-Match Penalty: Existing benchmarks rely on "exact match" evaluation under incomplete annotations. If a VLM generates a valid interaction description (e.g., "exiting an airplane") that is not explicitly labeled in the ground truth (which might only label "boarding"), the prediction is marked wrong. This unfairly penalizes the flexible, generative nature of VLMs.
• Ambiguity: Static images often lack sufficient visual evidence to distinguish between temporally ambiguous actions (e.g., catching vs. throwing a frisbee), yet existing benchmarks force a single ground truth, leading to systematic underestimation.
• Dataset Bias: Current test sets often mirror training distributions heavily, containing many simple, repetitive "head-class" cases (single person, single object, obvious action), which fails to challenge models on complex reasoning.

2. Methodology: CrossHOI-Bench

The authors introduce CrossHOI-Bench, a unified benchmark designed to fairly evaluate both VLMs and HOI-specific methods by reformulating the task.

A. Task Reformulation: Multiple-Choice Question Answering (MCQA)

Instead of open-ended generation or fixed-label classification, HOI detection is reformulated as a multiple-answer, multiple-choice task.

• Structure: For each image and target human-object pair, the model is presented with 4 options (A, B, C, D).
• Positives: Options include all valid interactions (ground truth + manually added valid unlabeled interactions).
• Negatives: Options include curated "hard negatives" (plausible but incorrect interactions).
• Benefit: This eliminates the "false negative" penalty. If a model selects a valid interaction that was previously unlabeled in HICO-DET, it is now a correct choice.

B. Dataset Construction & Curation

The benchmark is built upon HICO-DET but undergoes rigorous refinement to increase difficulty and reduce bias:

1. Removal of Simple Cases: Overly simple scenes (e.g., single person in a simple background) are removed to prevent models from exploiting dataset priors.
2. Coarse Screening (VLM-assisted): A multi-stage VLM pipeline (using GPT-4.1 and Qwen2.5-VL) filters out obviously incorrect negative candidates.
3. Manual Refinement: Human annotators verify the remaining choices to ensure:
   • Hard Positives: Multiple valid actions for the same pair (e.g., "holding" and "cutting" a knife) are included.
   • Hard Negatives: Interactions performed by other people in the scene (to test cross-person attribution) and fine-grained distinctions (e.g., "repair" vs. "type" on a laptop).
4. Distribution Shift: The test set is redistributed to have a higher KL divergence (0.629) from the training set compared to the original HICO-DET (0.088), ensuring the model cannot simply memorize the training distribution.

C. Evaluation Settings

The benchmark defines three distinct settings to isolate specific capabilities:

• Setting 1 (Full HOI Detection): The model must detect the target person (bounding box) and then recognize the interaction. This tests the full pipeline (Detection + Recognition).
• Setting 2 (Localized Recognition): The target person's bounding box is provided. This isolates the recognition capability, removing detection errors.
• Setting 3 (Image-Level Recognition): The model must identify interactions for all individuals in the scene. This tests global reasoning and the ability to disentangle multiple concurrent actions.

D. Complementary Sub-Benchmarks

To test generalization, two sub-benchmarks are constructed based on V-COCO (focusing on multi-person scenarios) and SWiG-HOI (focusing on human-human interactions).

3. Key Contributions

1. First Unified Benchmark: CrossHOI-Bench is the first dataset designed to jointly evaluate general-purpose VLMs and specialized HOI methods under a unified protocol, resolving the incompatibility caused by exact-match metrics.
2. Mitigation of Annotation Bias: By using curated negatives and allowing multiple correct answers, the benchmark prevents valid generative outputs from being penalized due to incomplete ground truth.
3. Comprehensive Analysis: The study provides a systematic comparison of state-of-the-art VLMs (e.g., Qwen2.5-VL, InternVL3) against HOI-specific models (e.g., ADA-CM, CMMP) across zero-shot and fine-tuned scenarios.
4. Identification of Paradigm Strengths/Weaknesses: The benchmark reveals that while VLMs excel at semantic understanding, they struggle with precise localization and multi-person attribution, whereas HOI-specific models are better at detection but weaker in open-vocabulary reasoning.

4. Experimental Results

The authors evaluated recent large VLMs (Qwen2.5-VL-32B, InternVL3-38B) and HOI-specific methods on the benchmark.

• Zero-Shot Performance:
  • Large VLMs (e.g., Qwen2.5-VL-32B) achieve state-of-the-art performance, often surpassing specialized HOI methods in Instance-F1 and Macro-F1 (e.g., +5.18% Instance-F1 over the best HOI method in Setting 1).
  • Small VLMs (e.g., 7B models) perform on par with HOI methods in recognition-only settings (Setting 2) but drop significantly when detection is required (Setting 1).
• Failure Modes of VLMs:
  • Incomplete Multi-Action Recognition: VLMs tend to predict only the most salient action, missing concurrent interactions (e.g., predicting "riding" but missing "hugging" a horse).
  • Cross-Person Misattribution: In multi-person scenes, VLMs frequently attribute the actions of surrounding people to the target person (approx. 20-25% of errors).
  • Detection Bottleneck: VLMs struggle with reliable object/person detection in crowded or occluded scenes, which is the primary cause of performance drops in Setting 1 compared to Setting 2.
• HOI-Specific Methods:
  • They maintain strong performance in detection and multi-action recognition but struggle with general HOI reasoning and out-of-distribution generalization (significant performance drops on V-COCO and SWiG-HOI sub-benchmarks).
• Fine-Tuning: When fine-tuned on the benchmark's training data, VLMs improve significantly, particularly in recall, though they still exhibit head-class bias.

5. Significance

• Paradigm Shift: The paper demonstrates that large generative VLMs are already competitive with, and sometimes superior to, specialized HOI detectors in terms of semantic understanding and zero-shot generalization.
• Evaluation Standard: It establishes a new standard for HOI evaluation that accounts for the ambiguity of real-world scenes and the flexibility of generative models, moving away from rigid exact-match metrics.
• Future Directions: The findings highlight that the primary bottleneck for VLMs in HOI is localization (detection) and disentangling multi-person interactions, rather than semantic understanding. This suggests future research should focus on improving VLM spatial reasoning and detection capabilities rather than just training more HOI-specific classifiers.

In summary, CrossHOI-Bench provides a robust, fair, and challenging platform to measure the progress of both specialized and generalist models in understanding human-object interactions, revealing that the gap between them is narrowing rapidly.