TAIHRI: Task-Aware 3D Human Keypoints Localization for Close-Range Human-Robot Interaction

1. Problem Statement

Current 3D human pose estimation methods are primarily optimized for whole-body reconstruction relative to a root joint (e.g., the pelvis) in a root-centric coordinate system. While effective for general benchmarks, this approach fails in close-range Human-Robot Interaction (HRI) scenarios for several reasons:

• Task Relevance: Robots need precise metric-scale localization of specific body parts (e.g., hands for handshaking, underarms for lifting) rather than just the global body pose.
• Egocentric Perspective: HRI often involves cameras mounted on the robot or worn by the user, leading to severe occlusion, truncation, and rapidly changing viewpoints.
• Metric Accuracy: Root-relative representations are insufficient for physical interaction, which requires absolute spatial coordinates in the camera frame to ensure safe and precise motion planning.
• Depth Ambiguity: Monocular close-range views suffer from significant depth ambiguity, making global pose estimation unreliable for distal keypoints (wrists, ankles).

2. Methodology: TAIHRI

The authors propose TAIHRI, the first Vision-Language Model (VLM) tailored for close-range HRI perception. It treats 3D keypoint localization as a next-token prediction task.

A. Dataset Construction: CloseHRI

To address the lack of close-range egocentric data, the authors created the CloseHRI dataset:

• Pipeline: Based on the WildHuman pipeline, they render normal maps from SMPL-X models in Blender, then use SD-XL to synthesize photorealistic images with diverse backgrounds.
• Viewpoints: Virtual cameras are placed 0.5–3 meters from subjects to simulate robot-mounted perspectives.
• Filtering: Low-quality samples are removed using SAM3 (IoU < 0.9) and VitPose (2D reprojection error < 15 pixels).
• Scale: The final dataset contains over 1 million images with diverse poses and interaction scenarios.

B. Discretized Interaction Space

To integrate with VLMs, continuous 3D coordinates are quantized into a discrete interaction space:

• The interaction volume (e.g., a cuboid in front of the robot) is divided into a $1000 \times 1000 \times 1000$ grid.
• Coordinates $(x, y, z)$ are converted to integer tokens $(X, Y, Z)$ via floor division, allowing the model to predict 3D locations as text tokens.
• Intrinsics Handling: To handle camera intrinsics without complex adapters, the focal length is unified (e.g., to 1000) via image resizing, and random cropping simulates principal point offsets.

C. Two-Stage Reasoning (Chain-of-Thought)

Instead of predicting 3D coordinates directly, TAIHRI employs a 2D-to-3D reasoning chain:

1. 2D Keypoint Reasoning: The model first predicts the 2D pixel coordinates of the relevant keypoints.
2. 3D Keypoint Prediction: Based on the 2D locations and the task context, the model reasons about the depth to output the final 3D voxel tokens.

• This approach leverages the visual clarity of 2D projections to guide the more ambiguous 3D depth estimation.

D. Training Strategy

• Supervised Fine-Tuning (SFT): The model (based on Qwen3-VL) is trained on the CloseHRI dataset using a mix of task-specific prompts and generic prompts.
• Reinforcement Finetuning (RFT): The model is further optimized using Group Relative Policy Optimization (GRPO).
  • Reward Function: A pose-aware reward is computed based on visible joints, combining a robust Huber loss and a PCK-style success term. This encourages the model to minimize error while maintaining structural consistency.

3. Key Contributions

1. Task-Aware Perception: TAIHRI is the first VLM designed to understand natural language commands (e.g., "lift from underarms") and dynamically focus on task-relevant keypoints rather than the whole body.
2. Metric-Scale Localization: By predicting coordinates directly in the camera frame via a discretized space, it solves the metric localization problem essential for physical robot control.
3. Novel Dataset: The introduction of CloseHRI, a large-scale, systematically filtered dataset specifically for close-range egocentric HRI.
4. Reasoning Architecture: The integration of 2D keypoint reasoning as an intermediate step significantly improves 3D localization accuracy in occluded scenarios.

4. Experimental Results

The model was evaluated on Harmony4D-Egocentric and EgoBody test sets using Global Mean Per Joint Position Error (G-MPJPE) in millimeters (without root alignment).

• Performance: TAIHRI significantly outperforms state-of-the-art methods (CameraHMR, PromptHMR, SAM 3D Body) and general VLMs (GPT-5.2, Gemini-2.5 Pro).
  • On Harmony4D, TAIHRI achieved 93.83 mm error for the Upper Body, compared to ~125 mm for the next best method.
  • It showed the most significant improvement on distal keypoints (wrists, ankles) which are critical for interaction but prone to error in root-relative methods.
• Ablation Studies:
  • Removing camera intrinsics caused a massive performance drop (error > 400 mm), proving the necessity of geometric priors.
  • Removing the 2D reasoning step degraded performance, validating the chain-of-thought approach.
  • RFT with GRPO outperformed standard MSE regression, demonstrating the benefit of reinforcement learning for robustness.
• Applications: The system was successfully deployed on a real bimanual robot for tasks like "handshaking" and "shoulder massage," demonstrating stable closed-loop control.

5. Significance

TAIHRI represents a paradigm shift in embodied AI perception. It moves away from generic whole-body reconstruction toward task-specific, metric-accurate localization. By bridging the gap between Vision-Language Models and robotic control, it enables robots to:

• Interpret natural language instructions regarding specific body parts.
• Perform precise physical interactions in close-range, occluded environments.
• Seamlessly integrate with downstream tasks like global human mesh recovery and robot motion planning.

This work establishes task-aware VLMs as a critical technology for the next generation of safe, intuitive, and collaborative human-robot systems.