A HamNoSys-Guided Dataset and Baselines for Fine-Grained Isolated Handshape Recognition in Sign Language

Technical Summary: A HamNoSys-Guided Dataset and Baselines for Fine-Grained Isolated Handshape Recognition in Sign Language

Problem Statement
Fine-grained handshape recognition is a critical component for computational sign-language transcription, recognition, and translation. However, the field lacks broad, phonetically defined visual inventories that include signer-aware evaluation. Existing resources often focus on fingerspelling (restricted alphabets), continuous signs (weak labels), or synthetic data, leaving a gap for a real-image benchmark that combines a language-independent, transcription-defined handshape inventory with systematic evaluation on unseen participants. Furthermore, while the Hamburg Notation System (HamNoSys) is widely used for linguistic annotation, it has not been previously utilized to define a bounded, visually reproducible class inventory for isolated handshape recognition datasets.

Methodology
The authors constructed a new dataset and established baseline models to address this gap.

• Dataset Construction:

  • Class Inventory: The study utilized the official, non-exhaustive HamNoSys 4 Handshapes Chart to define 160 distinct handshape classes. Each illustrated hand model in the chart was treated as a unique class, excluding blank cells or those containing only symbols. The classes were categorized into ten populated rows (e.g., Fist, One Finger, Two Fingers, Thumb opposition).
  • Data Acquisition: A balanced dataset of 144,000 RGB images was collected from 15 university participants. For each of the 160 classes, participants reproduced the target handshape while slowly rotating their hands to introduce viewpoint, overlap, and self-occlusion variations.
  • Processing: Videos were recorded at 30 fps (640 × 480 resolution). To reduce temporal redundancy while maintaining variation, every fifth frame was selected, yielding 60 images per 10-second clip. MediaPipe Hands was used to localize the dominant hand in 139,199 images, forming a common modeling subset.
  • Organization: The dataset is hierarchically organized by participant and HamNoSys category, with a naming convention linking images to specific chart-defined classes.

• Evaluation Protocols:
  Two complementary protocols were defined on the 139,199-image modeling subset:

  1. Subject-Dependent: A class-stratified 70:15:15 frame-level split (Train:Validation:Test) where all participants appear in every partition. This serves as a reference for recognition under seen-participant conditions.
  2. Subject-Independent (LOSO): A 15-fold Leave-One-Subject-Out cross-validation where one participant is held out for testing while the model is trained and validated on the remaining 14. This assesses generalization to unseen individuals.

• Baseline Models:
  Four models were evaluated to characterize the benchmark:

  • Appearance-Based (RGB): ResNet-18 and ViT-B/16, both fine-tuned on 224 × 224 hand crops.
  • Landmark-Based: A Graph Convolutional Network (GCN) and XGBoost, both operating on 21 hand keypoints (x, y, z, and joint angles) extracted via MediaPipe.

Key Contributions

1. HamNoSys-Grounded Dataset: The introduction of a balanced, real-image dataset of 144,000 images covering 160 fine-grained handshape classes defined by the HamNoSys 4 chart. This provides a reproducible resource for phonology-grounded research.
2. Comprehensive Evaluation Framework: The establishment of both subject-dependent and subject-independent (LOSO) protocols to distinguish between intra-participant recognition performance and generalization to unseen signers.
3. Baseline Performance: The provision of reproducible reference performance for four distinct model families (ResNet-18, ViT-B/16, GCN, XGBoost) on this specific 160-class inventory.
4. External Context: Matched-model evaluations on LSWH100 (synthetic) and ASL Fingerspelling Dataset A (real, 24 classes) to provide comparative context, though the authors note these are not direct quality rankings due to differing acquisition conditions and class counts.

Results

• Subject-Dependent Performance: On the 160-class dataset, ViT-B/16 achieved the highest top-1 accuracy (86.20%), followed closely by ResNet-18 (84.72%). Landmark-based models performed lower, with GCN at 72.44% and XGBoost at 69.57%. The authors note that appearance-based models benefit from visual details not fully captured by 21-point landmarks.
• Subject-Independent (LOSO) Performance: Generalization to unseen participants resulted in a significant performance drop. ViT-B/16 and ResNet-18 top-1 accuracies fell to approximately 45.22% and 45.38%, respectively. GCN achieved the highest top-3 (69.23%) and top-5 (78.58%) accuracies in this setting, suggesting it retains the correct class within its top predictions even when the top-1 prediction fails.
• Confusion Analysis: Visual analysis revealed that the most frequent errors occurred between handshapes differing in subtle articulatory properties (e.g., finger selection, bending, thumb position). The broad categories were generally well-separated, but fine-grained distinctions remain challenging.
• External Datasets: On the 24-class ASL Fingerspelling Dataset A, all models achieved near-ceiling performance (>97%), highlighting the difficulty of the proposed 160-class fine-grained task compared to restricted alphabets.

Significance and Claims
The paper positions this work as a reproducible resource for fine-grained isolated-handshape research. The authors claim that the dataset and baselines support the development of more accessible sign-language technologies, particularly for transcription, recognition, and translation across sign languages. The work addresses a specific resource gap by combining a broad, phonetically defined inventory with rigorous participant-independent evaluation.

The authors modestly acknowledge limitations: the data was collected in a controlled indoor environment with a single camera and a specific demographic (university students); the inventory is restricted to static, single-hand forms from the HamNoSys chart (excluding dynamic transitions, two-handed signs, and non-manual components); and only four baseline model families were evaluated. They suggest that future work should investigate more diverse participants, less-controlled acquisition settings, and advanced fine-grained models.