Technical Summary: MHPR – Multidimensional Human Perception and Reasoning Benchmark

1. Problem Statement

Current Large Vision-Language Models (LVLMs) and existing benchmarks face significant limitations in supporting real-world applications such as film analysis, virtual digital humans, and intelligent retail. While prior works like RefCOCO, HERM-Bench, and HumanRef have advanced human-centric detection and simple Visual Question Answering (VQA), they largely focus on single-task dimensions (e.g., detection or classification) and lack fine-grained, multidimensional annotations.

Specifically, existing datasets fail to comprehensively evaluate the joint perception and reasoning required for complex human-scene understanding. They often miss:

• Fine-grained attributes: Detailed descriptions of appearance, clothing, pose, and body parts.
• High-level semantics: Understanding of social relations, action semantics, spatial relations, and the intent or functionality behind human-human and human-object interactions.
• Multidimensional scope: The ability to simultaneously handle individual, multi-person, and human-object interaction scenarios.

This gap results in practical limitations, such as limited visual diversity in virtual digital human generation and the inability of film analysis systems to robustly track group relations or dynamic interactions across narrative threads.

2. Methodology

The authors introduce MHPR (Multidimensional Human Perception and Reasoning), a comprehensive benchmark designed to bridge these gaps through a systematic data design and an automated generation pipeline.

2.1 Multi-Level Data Design

MHPR employs a four-tiered data structure to support different stages of model development:

1. Captioned Raw Data (C-RD): Preserves openness and extensibility, allowing researchers to customize new tasks and evaluation dimensions.
2. Supervised Fine-Tuning Data (SFT-D): Consists of large-scale, task-related VQA samples strictly aligned in format with the test data. This layer is designed to learn fundamental instruction-following abilities and provide a stable alignment baseline.
3. Reinforcement Learning Data (RL-D): An advanced dataset constructed through systematic "bad-case" analysis of model errors on the test set. It targets specific failure modes (spatial relations, fine-grained perception, and reasoning depth calibration) to enhance perception and reasoning on difficult instances.
4. Test Data (T-D): Used for objective evaluation, ensuring consistency with SFT-D in composition and annotation paradigms.

2.2 Data Sources

The benchmark integrates three open-source datasets to ensure breadth, depth, and stability:

• COYO-700M: Provides a large-scale, open-domain, stylistically diverse collection of image-text pairs for general pretraining and cross-domain generalization.
• HumanCaption-10M: Offers broad-coverage, fine-grained, context-rich human-centric descriptions for training corpora.
• HumanCaption-HQ-311K: A high-confidence, low-noise refined subset used for alignment and calibration.

2.3 Automated Caption and VQA Generation Pipeline (ACVG)

To address the labor intensity of manual annotation, the authors propose ACVG, which utilizes a multi-model voting mechanism to ensure high-quality, scalable annotations with minimal human curation.

• Caption Auto-Annotation Pipeline (CAAP):
  • Filtering: A four-stage pipeline (Human Detection, Scale Filtering, Keypoint Completeness Check, and Collage-Image Filtering) removes low-quality or irrelevant images.
  • Generation & Fusion: Three local caption generators (Qwen, Intern, Llama) produce initial captions. A strong LLM orchestrator performs structured disagreement detection (GetDiff), conflict correction via multi-source voting (Vote), and schema-level fusion (Mix) to produce a single, accurate caption.
• VQA Auto-Generation Pipeline (VAGP):
  • Question Proposal: Generates candidate questions across three semantic dimensions (single-person, multi-person, human-object) based on high-quality captions.
  • Tri-MLLM Answering: Three heterogeneous MLLMs answer the questions in parallel.
  • Consistency Filtering: Answers are normalized and compared. Exact three-way agreement yields "easy" samples for training; disagreements are flagged as "hard" or ambiguous for human review or semi-automatic correction.

2.4 Evaluation Dimensions

The benchmark evaluates models across three complementary scenario dimensions:

• Single Person: Appearance, attributes (age, gender, occupation), and referential localization.
• Multiple Persons: Interpersonal relations, social ties, and comparative analysis.
• Person and Object: Interactions between specific persons and non-human objects.

These dimensions cover fine-grained attributes (appearance, clothing, pose, parts) and high-level semantics (social relations, action semantics, spatial relations, intent, and functionality).

3. Key Contributions

1. MHPR Benchmark: A human-centric, multidimensional benchmark that jointly evaluates perception and reasoning across individual, multi-person, and human-object interaction scenarios. It addresses key gaps in complex human-scene understanding by covering both fine-grained attributes and high-level semantics.
2. Multi-Level Data Design & ACVG Pipeline: The introduction of a structured data stack (C-RD, SFT-D, RL-D, T-D) and the ACVG pipeline. The pipeline employs multi-model voting and consistency correction to produce high-quality captions and VQA with minimal manual effort, significantly improving scalability and annotation quality.
3. Targeted RL Data Construction: The identification of three specific capability gaps in current LVLMs (spatial relations, fine-grained perception, and reasoning depth calibration) and the construction of a targeted RL dataset to address these specific weaknesses through bad-case analysis.

4. Experimental Results

The authors trained Qwen2.5-VL-7B using the MHPR dataset (specifically the SFT-D and RL-D components) to create Qwen2.5-VL-7B-SFT.

• Performance: The fine-tuned model achieved an average accuracy of 81.40% across multiple subsets, outperforming the base Qwen2.5-VL-7B-Instruct (79.33%) and InternVL3-8B (77.64%).
• Comparative Strength: Qwen2.5-VL-7B-SFT demonstrated significant gains in subsets requiring fine-grained reasoning and perception (e.g., Subject 4, 5, 6, 7, 8).
• Efficiency: The 7B parameter model trained with MHPR achieved near-parity with, and in some cases outperformed, considerably larger models, validating the effectiveness of the custom dataset and training methodology.

5. Significance and Claims

The paper claims that MHPR provides a necessary framework for advancing human-centric AI by moving beyond mechanical target verification toward deeper, context-aware comprehension.

• Joint Modeling: The results suggest that jointly modeling fine-grained perception and high-level semantic reasoning is critical for performance gains in human-centric tasks.
• Data Quality over Scale: The study demonstrates that format-aligned SFT data and challenge-focused RL data can substantially improve instruction following, stability, and reasoning capabilities, often yielding better results than simply increasing model size.
• Scalable Annotation: The ACVG pipeline proves that high-quality, scalable human-centric data can be generated with minimal manual curation through multi-model voting and automated consistency correction.
• Identified Challenges: The work highlights that spatial understanding (viewpoint modeling), fine-grained perception under occlusion, and the calibration of reasoning depth (knowing when not to infer) remain major challenges for current LVLMs, providing a clear direction for future research.

By releasing ACVG and MHPR, the authors aim to facilitate reproducible and extensible research in human-centric perception and reasoning, offering a practical path toward more robust and intelligent vision-language applications.