💻 computer science

HandVQA: Diagnosing and Improving Fine-Grained Spatial Reasoning about Hands in Vision-Language Models

This paper introduces HandVQA, a large-scale benchmark of over 1.6 million questions derived from high-quality 3D hand datasets to diagnose and improve the fine-grained spatial reasoning capabilities of vision-language models, revealing their current limitations while demonstrating that training on this benchmark significantly boosts performance on downstream tasks like gesture recognition and hand-object interaction.

Original authors: MD Khalequzzaman Chowdhury Sayem, Mubarrat Tajoar Chowdhury, Yihalem Yimolal Tiruneh, Muneeb A. Khan, Muhammad Salman Ali, Binod Bhattarai, Seungryul Baek

Published 2026-03-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: MD Khalequzzaman Chowdhury Sayem, Mubarrat Tajoar Chowdhury, Yihalem Yimolal Tiruneh, Muneeb A. Khan, Muhammad Salman Ali, Binod Bhattarai, Seungryul Baek

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a brilliant but clumsy robot assistant how to use its hands. You show it a picture of a hand and ask, "Is the pinky finger bent a little bit, or is it completely straight?"

The robot, which is actually a very advanced AI called a Vision-Language Model (VLM), looks at the picture. It knows what a hand looks like generally. It knows what a "finger" is. But when you ask it about the specific angle of a tiny joint or whether one finger is slightly behind another, it starts to guess wildly. It might say, "The pinky is straight!" when it's clearly bent, or it might invent a finger that isn't there at all.

This paper, HandVQA, is like a giant, rigorous "finger gym" designed to fix this problem.

The Problem: The "Smart but Clumsy" Robot

Current AI models are like students who have read every book in the library but have never actually held a pencil. They are great at general conversation and recognizing objects ("That's a hand!"), but they are terrible at fine-grained spatial reasoning.

Think of it like this:

General AI: Can tell you, "That is a hand holding a coffee cup."
HandVQA's Goal: Can tell you, "The tip of the thumb is 2 millimeters to the left of the ring finger's middle joint, and the index finger is bent at a 45-degree angle."

Without this precision, these robots are dangerous in high-stakes jobs. If a robot surgeon misinterprets a "slightly bent" finger as "straight," the surgery could go wrong. If a robot on a chip factory line misjudges the angle of a fingertip, it could ruin a million-dollar microchip.

The Solution: HandVQA (The Finger Gym)

The researchers created a massive training and testing ground called HandVQA.

The Source Material: They didn't just take random photos. They used high-tech 3D datasets (like FreiHAND and InterHand2.6M) that act like a "X-ray vision" for hands. These datasets know the exact 3D coordinates of every single joint in a hand.
The Workout: Using this 3D data, they automatically generated 1.6 million multiple-choice questions.
- Question: "Is the tip of the middle finger closer to the ring finger or the index finger?"
- Question: "Is the thumb's tip above or below the ring finger's knuckle?"
- Question: "Is the pinky bent slightly inward or completely inward?"

This is like giving the AI a million flashcards where the answers are mathematically perfect, forcing it to stop guessing and start measuring the image.

The Training: From "Guesstimator" to "Surgeon"

The researchers took popular AI models (like LLaVA, DeepSeek, and Qwen) and "fine-tuned" them using this HandVQA gym.

Before Training: The AI was like a person trying to guess the weight of a watermelon by looking at it from across the room. It guessed "close" or "far" randomly.
After Training: The AI learned to look at the image and actually calculate the geometry. It learned that "close" means a specific distance, and "bent inward" means a specific angle.

The Magic Trick: Zero-Shot Transfer

Here is the most exciting part. The researchers didn't just teach the AI to answer their specific questions. They taught it a skill.

Imagine you teach a student how to solve complex geometry problems on paper. You then hand them a real-world task: "Build a bridge." You didn't teach them bridge building, but because they mastered the geometry, they can build a much better bridge than someone who never studied geometry.

In the paper, the AI trained on HandVQA was tested on tasks it had never seen before:

Gesture Recognition: Identifying if a hand is making a "peace sign" or a "thumbs up."
Hand-Object Interaction: Watching a video and figuring out if someone is grabbing a coffee cup or putting it down.

The Result? The AI that trained on the "finger gym" got significantly better at these new tasks (improving accuracy by over 10% in some cases). It proved that understanding the geometry of a hand helps the AI understand what the hand is doing.

The Takeaway

HandVQA is a wake-up call for the AI world. It shows that while our current AI models are smart, they are "spatially blind" when it comes to the tiny, complex details of human hands.

By building a specialized training ground that forces AI to understand the 3D skeleton of a hand, the researchers have given these models a new pair of glasses. Now, instead of just seeing a blurry hand, they can see the joints, the angles, and the relationships between fingers. This is a crucial step toward making robots safe enough to hold a scalpel, assemble a microchip, or interact with us naturally in the real world.

1. Problem Statement

While Vision-Language Models (VLMs) have achieved near-human performance on general visual benchmarks, they exhibit significant deficiencies in fine-grained spatial reasoning, particularly regarding the complex articulation of human hands.

Critical Gap: In high-stakes applications like robot-assisted surgery, chip manufacturing, and AR/VR, misinterpreting subtle hand gestures (e.g., a slightly bent finger vs. a fully extended one) can lead to catastrophic errors.
Specific Failure Modes: Current VLMs struggle with:
- Geometric Hallucinations: Predicting non-existent finger bends or incorrect joint configurations.
- Spatial Blindness: Inability to distinguish basic relative positions (left/right, front/behind, above/below) and distances between joints.
- Superficial Alignment: Models often rely on language priors or "safe" default answers (e.g., always predicting "close" for distance or "slightly bent" for angles) rather than genuine geometric understanding.

2. Methodology: The HandVQA Benchmark

The authors introduce HandVQA, a large-scale diagnostic benchmark designed to evaluate and improve VLMs' understanding of 3D hand anatomy through Visual Question Answering (VQA).

A. Data Construction Pipeline

HandVQA is built upon high-quality 3D hand datasets (FreiHAND, InterHand2.6M, and FPHA). The authors developed an automated, deterministic pipeline to convert 3D joint coordinates into natural language questions:

Pose Descriptor Extraction ( $F_{pose}$ ): Computes continuous geometric metrics from normalized 3D joint coordinates:
- Angles ( $\theta$ ): Bending degrees at specific joints (e.g., DIP, PIP).
- Distances ( $d$ ): Euclidean distance between joint pairs.
- Relative Positions ( $\Delta x, \Delta y, \Delta z$ ): Spatial offsets along Cartesian axes.
Discretization & Categorization: Continuous values are mapped to discrete linguistic categories using fixed thresholds (e.g., "bent slightly inward" for $150^\circ \le \theta < 170^\circ$ ; "close to" for $d < 0.1$ ).
Sentence Generation ( $F_{text}$ ): Fills deterministic templates to create natural language sentences (e.g., "The distal interphalangeal joint of the middle finger is [category] the distal interphalangeal joint of the ring finger").
MCQ Formation ( $F_{mcq}$ ): Generates multiple-choice questions where one option is the ground truth and others are distractors derived from incorrect categories.

B. Benchmark Scale and Scope

Scale: Over 1.6 million controlled multiple-choice questions.
Sub-tasks: The benchmark is divided into five distinct reasoning categories:
1. Joint Angles
2. Distances
3. Relative Position X (Left/Right)
4. Relative Position Y (Above/Below)
5. Relative Position Z (Front/Behind)

3. Experimental Setup

Models Evaluated: State-of-the-art 7B parameter VLMs: LLaVA-Mistral, DeepSeek Janus Pro, and Qwen-VL.
Training Strategy: Models were evaluated in their base (pre-trained) state and after fine-tuning using LoRA (Low-Rank Adaptation) on the HandVQA dataset.
Evaluation Metrics: Accuracy and Mean Absolute Error (MAE) for ordinal categories (angles/distances); Accuracy for binary classification tasks (relative positions).

4. Key Results and Findings

A. Diagnosis of Base Models

Poor Baseline Performance: Base VLMs performed at or below random guessing on most tasks.
- Distance: LLaVA and Qwen achieved <33% accuracy (worse than random 33.3% for 3 classes).
- Relative Position: Accuracy hovered around 50% for X, Y, and Z axes, indicating a lack of inherent spatial grounding.
- Bias: Models exhibited strong biases, such as predicting "bent slightly inward" for almost all angle questions or "close to" for all distance questions.

B. Impact of Fine-Tuning

Significant Improvement: Fine-tuning on HandVQA led to massive performance gains across all models and datasets.
- Distance: Accuracy jumped from ~13-45% (base) to 80-90% (fine-tuned).
- Relative Position: Accuracy improved from ~50% to >90% on X and Y axes.
- Angles: While improved, angle reasoning remained the most challenging, with fine-tuned models reaching only ~60-74% accuracy, suggesting that joint kinematics are harder to learn than simple spatial relations.

C. Zero-Shot Generalization (Transfer Learning)

A critical contribution of the paper is demonstrating that spatial reasoning learned from HandVQA is transferable to unseen downstream tasks without specific training on those tasks:

Gesture Recognition (HaGRID dataset): Fine-tuned models showed a +10.33% absolute accuracy gain over base models.
Hand-Object Interaction (H2O dataset): Fine-tuned models achieved a +2.63% gain in video-based interaction recognition.
Conclusion: The 3D-grounded spatial knowledge acts as a foundational skill that generalizes to dynamic and static hand-related tasks.

5. Key Contributions

HandVQA Benchmark: The first large-scale (1.6M+ questions), automated benchmark specifically for diagnosing fine-grained 3D hand spatial reasoning in VLMs.
Systematic Diagnosis: Revealed that current VLMs lack genuine geometric understanding, relying instead on superficial correlations and language priors, particularly failing on distance and angle estimation.
Training Resource: Demonstrated that HandVQA is not just a diagnostic tool but a robust training resource. Fine-tuning on this benchmark significantly boosts performance on novel, unseen downstream tasks (zero-shot transfer).
Anatomical Precision: Introduced a pipeline that maps 3D joint kinematics to structured natural language, enabling precise evaluation of part-whole spatial relationships.

6. Significance and Future Directions

Safety and Reliability: The work addresses a critical safety gap in embodied AI, robotics, and medical applications where precise hand pose interpretation is non-negotiable.
Beyond Object Detection: It shifts the focus from "what object is present" to "how is the object articulated," pushing VLMs toward true physical reasoning.
Future Work: The authors suggest extending the benchmark to video (motion dynamics), integrating with Vision-Language-Action (VLA) models for grasp planning, and developing adaptive mappings to handle continuous geometric reasoning beyond discrete categories.

In summary, HandVQA proves that while current VLMs are spatially blind regarding human hands, they can acquire robust, transferable 3D spatial reasoning capabilities through targeted, anatomy-grounded supervision.