The Influence of Iconicity in Transfer Learning for Sign Language Recognition

This study demonstrates that leveraging the iconicity of signs in transfer learning from Chinese to Arabic and Greek to Flemish significantly improves sign language recognition performance, particularly yielding a 7.02% gain for Arabic, by utilizing MediaPipe-extracted spatial and temporal features processed through MLP and GRU architectures.

The Gist

Imagine you are trying to learn a new language, but you only have a tiny dictionary and a few hours to study. That is the daily reality for researchers trying to teach computers to understand Sign Language. Unlike spoken languages, which have massive libraries of text and audio, sign language datasets are often small, making it hard for computers to learn without getting confused (a problem called "overfitting").

To solve this, researchers usually use a trick called Transfer Learning. Think of this like a student who has already mastered French trying to learn Spanish. Because the languages share similar roots and words, the student doesn't start from zero; they just need to learn the differences.

This paper asks a fascinating question: Does it matter how the signs look?

The Big Idea: "Iconicity"

In sign language, some signs are iconic. This means the hand movement looks exactly like the thing it represents.

• Example: The sign for "Think" usually involves tapping your forehead. The sign for "Drink" mimics holding a cup. These are "pantomimes."
• The Theory: If the sign for "Love" looks the same in Chinese Sign Language and Arabic Sign Language (because everyone draws a heart shape with their hands), maybe a computer that learned "Love" in Chinese can instantly understand "Love" in Arabic, even if the two languages are otherwise totally different.

The Experiment: A Cross-Language Swap

The researchers set up a "language swap" test using two pairs of sign languages:

1. Chinese to Arabic: They took a computer trained on Chinese signs and tried to teach it Arabic.
   • The Match: They focused on signs that were iconic (like "Head," "Hair," "Love").
   • The Result: The computer got 7% better at recognizing Arabic signs. It was like giving the student a cheat sheet that matched perfectly.
2. Greek to Flemish: They did the same with Greek and Flemish signs.
   • The Match: Again, they used iconic signs (like "Anatomy" and "Food").
   • The Result: The computer got 1% better. It was a small win, but still a win.

The "Magic" Ingredients

To make this work, the researchers didn't feed the computer raw video (which is messy and heavy). Instead, they used Google's MediaPipe.

• The Analogy: Imagine watching a dance. Instead of recording the whole stage with lights and costumes, you just track the skeleton of the dancer (the joints and bones).
• The computer only looked at the "skeleton" coordinates of the hands and face. This made the learning much faster and less confused by things like the signer's shirt color or body size.

The Twist: What if the signs don't match?

The researchers also tested what happens if you try to transfer knowledge between languages that don't share many iconic signs.

• They tried to teach a computer using Iranian signs to recognize French-Belgian signs.
• These two languages only shared two similar concepts (Anatomy and Sound).
• The Disaster: The computer actually got worse at its job. This is called "Negative Transfer."
• The Lesson: It's like trying to teach someone to drive a car by first teaching them how to ride a horse. If the skills are too different, you confuse the learner. You need enough shared "iconic" ground to make the transfer work.

Why Does This Matter?

1. For Low-Resource Languages: Many sign languages don't have enough data to train computers. This study proves that if you can find "iconic" signs that look similar across languages, you can "borrow" knowledge from a language with lots of data to help the one with little data.
2. Speed: Even when the accuracy didn't skyrocket, the computer learned faster. It reached the same level of skill in fewer "training sessions" (epochs).
3. Better than Standard Methods: In one case, using these "iconic" signs worked better than the standard method of using generic image training (like teaching a computer to recognize cats and dogs first).

The Bottom Line

This paper is a proof that visual similarity matters. If two sign languages use the same hand gestures to represent the same ideas (like "eating" or "thinking"), a computer can easily jump from one language to the other. It's a powerful tool for making sign language technology accessible to more people, especially those speaking rare or under-represented sign languages.

In short: If you want to teach a computer a new sign language, start with the signs that look like the things they represent. It's the universal shortcut.

Technical Summary

Here is a detailed technical summary of the paper "The Influence of Iconicity in Transfer Learning for Sign Language Recognition."

1. Problem Statement

Sign Language Recognition (SLR) faces a critical challenge: data scarcity. Unlike spoken language datasets which often contain hundreds of thousands of samples, sign language datasets are typically small (often <30k samples), making models prone to overfitting.

• Current Solutions: Most research relies on Transfer Learning (TL) from large vision-based datasets like ImageNet. Some recent studies attempt TL between different sign languages (e.g., ASL to BSL), attributing success to shared linguistic similarities.
• The Gap: There is a lack of systematic investigation into whether iconicity (the semiotic property where a sign visually resembles its meaning, e.g., a "thinking" gesture mimicking the head) is a necessary condition for effective cross-linguistic knowledge transfer. It is unclear if movement similarities driven by iconicity significantly improve TL performance compared to non-iconic signs or standard ImageNet pre-training.

2. Methodology

2.1 Datasets and Experimental Design

The study conducted a comparative analysis of Transfer Learning across two distinct source/target language pairs, focusing on iconic subsets of signs:

1. Pair A (High Iconic Overlap):
   • Source: CSL SLR500 (Chinese Sign Language, 500 classes).
   • Target: KArSL (Arabic Sign Language, 502 classes).
   • Shared Concepts: 5 iconic concepts (Anatomy, Hair, Eyesight, Love, Sound).
2. Pair B (Moderate Iconic Overlap):
   • Source: GSL Isolated (Greek Sign Language).
   • Target: Woordenboek VGT (Flemish Sign Language).
   • Shared Concepts: 3 iconic concepts (Anatomy, Food, Sound).

Note: The study also included an ablation study using a pair with low overlap (Iranian MedSLset to French-Belgian LSFB, sharing only 2 concepts) to test for negative transfer.

2.2 Feature Extraction

• Tool: Google MediaPipe (Holistic Landmarker).
• Input: Extracted spatial coordinates ($x, y, z$) for hands, wrists, and shoulders.
• Preprocessing:
  • Facial keypoints were excluded as the study focused on word-level SLR where facial expressions were less relevant to the specific cross-linguistic iconicity patterns analyzed.
  • Frames without hand activity were filtered out to remove noise.
  • Coordinates were normalized to frame dimensions (0.0–1.0).

2.3 Model Architecture

The researchers utilized a lightweight MLP-GRU architecture:

• MLP (Multilayer Perceptron): Processes spatial information from the keypoints. It has a single hidden layer with ReLU activation.
• GRU (Gated Recurrent Unit): Processes temporal information (sequence of frames) generated by the MLP. It consists of a single recurrent layer.
• Output: Softmax layer for classification.
• Training: Adam optimizer, categorical cross-entropy loss, early stopping based on loss convergence.

2.4 Transfer Learning Strategy

• Weight Initialization: Weights learned from the Source task (e.g., CSL) were saved and used to initialize the Target model (e.g., KArSL).
• Fine-tuning: Only the MLP layer weights were initialized from the source; the GRU layer was kept fixed (single layer) and trained from scratch or fine-tuned depending on the specific experiment configuration.
• Ablation Studies: The authors compared:
  1. Iconic-only transfer.
  2. Non-iconic transfer.
  3. Combined Iconic + Non-iconic transfer.
  4. Comparison against standard ImageNet (ResNet50) pre-training.

3. Key Contributions

1. First Investigation of Iconicity in TL: The study is the first to explicitly isolate and test the impact of iconic concepts on cross-linguistic transfer learning performance in SLR.
2. Quantitative Validation: It provides empirical evidence that transferring knowledge between languages with shared iconic concepts yields measurable performance gains.
3. Ablation on Similarity Thresholds: The study demonstrates that transferability is sensitive to the number of shared iconic concepts. It identified a threshold where insufficient similarity leads to negative transfer.
4. Efficiency Analysis: The research highlights that while accuracy gains may vary, iconic transfer learning significantly reduces the number of epochs required to converge to optimal performance.

4. Key Results

4.1 Performance Improvements (Iconic TL)

• CSL $\to$ KArSL (5 shared concepts): Recognition accuracy improved by 7.02% (from 80.15% to 85.78%).
• GSL $\to$ VGT (3 shared concepts): Recognition accuracy improved by 1.07% (from 90.28% to 91.25%).
• Convergence Speed: For the VGT task, the iconic TL model achieved the same Macro F1 score (87.88%) in 1,867 epochs, compared to 2,000 epochs for the baseline (a ~6.6% reduction in training time).

4.2 Ablation Study Findings

• Non-Iconic Transfer: Still yielded positive results for CSL $\to$ KArSL (+6.84%), suggesting general feature transfer is possible, but slightly less effective than iconic transfer.
• Negative Transfer: When transferring between Iranian (MedSLset) and French-Belgian (LSFB) languages, which shared only 2 iconic concepts, the model suffered negative transfer. Accuracy dropped from 58.66% (baseline) to 50.36%. This confirms that a minimum threshold of shared iconic concepts is required for successful TL.
• ImageNet Comparison:
  • For KArSL, Iconic TL (85.78%) outperformed ImageNet pre-training (77.34%).
  • For VGT, ImageNet pre-training (98.95%) outperformed Iconic TL (91.25%). Note: The authors acknowledge this is not a direct comparison due to different input modalities (RGB frames vs. Skeleton Keypoints).

5. Significance and Conclusion

• Low-Resource SLR: The study proves that leveraging iconicity is a viable strategy for improving SLR in low-resource languages. By identifying signs that share visual/semantic roots (iconic concepts) across languages, researchers can transfer knowledge more effectively than random or purely visual transfer.
• Linguistic Insight: The results validate that the semiotic concept of iconicity is not just a linguistic curiosity but a functional feature for machine learning, enabling models to generalize across language barriers.
• Future Applications: The pre-trained models developed in this study can serve as robust backbones for Sign Language Translation (SLT) and sign spotting systems, particularly for under-resourced sign languages where collecting large datasets is difficult.

Conclusion: The paper establishes that while Transfer Learning is generally beneficial for SLR, the degree of iconic similarity between source and target datasets is a critical factor. High overlap in iconic concepts leads to significant accuracy gains and faster convergence, whereas low overlap can result in performance degradation.