💬 NLP

SyriSign: A Parallel Corpus for Arabic Text to Syrian Arabic Sign Language Translation

This paper introduces SyriSign, the first publicly available parallel corpus of 1,500 video samples for Syrian Arabic Sign Language, aimed at bridging communication barriers for the Deaf community in Syria and serving as a foundational benchmark for text-to-sign translation research.

Original authors: Mohammad Amer Khalil, Raghad Nahas, Ahmad Nassar, Khloud Al Jallad

Published 2026-04-01

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Mohammad Amer Khalil, Raghad Nahas, Ahmad Nassar, Khloud Al Jallad

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where a vital news broadcast is happening, but for a large group of people, it's like watching a movie with the sound turned off and no subtitles. This is the daily reality for many Deaf and Hard-of-Hearing (DHH) people in Syria. While news is broadcast in spoken Arabic, there are very few professional sign language interpreters available, and the specific sign language used in Syria (Syrian Arabic Sign Language, or SyArSL) has been largely ignored by technology.

This paper introduces SyriSign, a project designed to build a bridge between written Arabic news and Syrian Sign Language using artificial intelligence.

Here is a breakdown of what they did, using simple analogies:

1. The Problem: A Missing Dictionary

Think of sign language as a unique language with its own grammar, hand shapes, and facial expressions. For years, AI researchers have built massive "libraries" (datasets) for popular sign languages like American Sign Language (ASL). However, for Syrian Sign Language, the library was empty.

Without data, computers can't learn to translate. It's like trying to teach a robot to speak a language when you haven't given it a single book or a single recording to study.

2. The Solution: Building the Library (SyriSign)

The team created SyriSign, which is essentially a digital library of 1,500 video clips.

The Content: They recorded 150 common words often found in Syrian news (like "Ministry," "Health," "War," "Peace").
The Actors: Two people performed each sign five times to give the computer enough practice examples.
The Goal: To create a "Rosetta Stone" that allows computers to understand the connection between an Arabic word and the specific hand movement that represents it.

3. The Experiment: Three Different Translators

To see if they could turn text into sign language, the team tried three different AI "architects," each with a different style of learning:

The "Matchmaker" (SignCLIP):
- How it works: Imagine a librarian who has a huge stack of index cards. When you give it a word, it doesn't create a new sign; it just finds the best matching card from the library it already has.
- Result: It was good at finding the right sign for a word because it relies on matching patterns, but it can't invent new movements.
The "Painter" (MotionCLIP):
- How it works: This AI tries to understand the meaning of the word and then "paints" a movement from scratch, similar to how a human might gesture when they don't know the exact sign. It tries to learn the "vibe" of the motion.
- Result: It showed promise in understanding the concept, but because the library was small, the "painting" sometimes looked a bit shaky or inconsistent.
The "Storyteller" (T2M-GPT):
- How it works: This is like a robot that breaks a sentence down into tiny Lego bricks (tokens). It learns the rules of how to snap these bricks together to build a fluid motion sequence.
- Result: It struggled the most. Because it needs a massive amount of data to learn the complex rules of "grammar" for movement, the small library (1,500 clips) wasn't enough for it to become fluent. It was like asking a baby to write a novel after only reading a few picture books.

4. The Reality Check: Growing Pains

The researchers found that while the technology is exciting, it's currently limited by the size of their library.

The "Small Library" Effect: Generative AI (the kind that creates new things) usually needs millions of examples to work well. With only 1,500 examples, the AI can't generalize well. It's like trying to learn to cook a whole cuisine by tasting only three dishes.
Missing Details: The current system focuses mostly on hand movements. It misses the "spice" of sign language: facial expressions and body language, which are crucial for meaning in sign language (like raising an eyebrow to show a question).

5. The Future: Opening the Doors

The team has made their library (SyriSign) public. They hope this will be the "seed" for future growth.

Next Steps: They want to record more people, add more words, and eventually move from translating single words to translating full sentences with all the facial expressions included.

In Summary:
This paper is about taking the first brave step to give Syrian Deaf people access to their local news through technology. They built a small but crucial dictionary (SyriSign) and tested three different AI methods to see how well they could translate text into sign. While the AI isn't perfect yet, this project lays the foundation for a future where a computer can instantly translate a news broadcast into a sign language video, ensuring no one is left out of the conversation.

1. Problem Statement

The Deaf and Hard-of-Hearing (DHH) community in Syria faces significant barriers in accessing local news and public announcements, which are predominantly delivered in spoken or written Arabic. Approximately 8% of Syrians over age 17 suffer from hearing loss. The challenges are compounded by:

Resource Scarcity: There is no publicly available dataset for Syrian Arabic Sign Language (SyArSL), a distinct visual-spatial language with unique grammar and syntax.
Interpreter Shortage: Only 10 officially registered sworn interpreters exist in Syria, making real-time interpretation for broadcasts impossible.
Linguistic Proficiency: Many DHH individuals lack full proficiency in written Arabic, rendering standard captions ineffective.
Low-Resource Context: Existing Sign Language Production (SLP) models rely on large-scale datasets (e.g., for American Sign Language), which do not exist for Arabic dialects, leading to poor generalization for low-resource languages.

2. Key Contributions

The authors introduce SyriSign, a novel dataset and a comparative study of state-of-the-art (SoTA) architectures adapted for low-resource SyArSL.

Dataset Creation (SyriSign):
- Scale: 1,500 video samples covering 150 unique lexical signs.
- Domain: Focused on local news and public announcements (vocabulary selected based on frequency in regional newspapers).
- Collection: Recorded by two signers (5 repetitions per sign) at 60 FPS in RGB format.
- Availability: Publicly released via Hugging Face and GitHub to serve as an initial benchmark.
Benchmark Evaluation: A comparative analysis of three distinct deep learning architectures under fine-tuned, low-resource settings:
1. MotionCLIP (Hybrid/Alignment)
2. T2M-GPT (Generative)
3. SignCLIP (Retrieval/Embedding)
Methodological Analysis: An investigation into the trade-offs between retrieval-based and generative methods for Arabic Sign Language.

3. Methodology

A. Data Preprocessing

The authors utilized MediaPipe Holistic to extract 543 landmarks (pose, face, hands) in real-time.

MotionCLIP: Used truncated motion sequences for skeletal compatibility.
T2M-GPT: Extracted full-body key points but excluded facial landmarks to reduce dimensionality and improve stability in the low-resource setting (acknowledging this as a limitation for non-manual features).
SignCLIP: Implemented a landmark reduction strategy, filtering 543 points down to 193 (23 pose, 128 face, 42 hands) to focus on linguistically relevant features and improve computational efficiency.

B. Model Architectures & Adaptations

The study adapted three models for the Arabic context:

MotionCLIP (Hybrid Approach):
- Original: Aligns motion latent spaces with CLIP text/image embeddings.
- Adaptation: Added an Arabic-to-English translation layer. Modified the loss function to prioritize Reconstruction, Velocity Smoothness, and Latent Regularization to prevent embedding drift.
- Goal: Semantic motion generation.
T2M-GPT (Generative Approach):
- Original: Uses a VQ-VAE to discretize motion into tokens, then a GPT to generate token sequences conditioned on text.
- Adaptation (T2M-GPT-ARABIC):
  - Replaced the standard text encoder with AraT5 (pretrained on Arabic corpora) for better semantic understanding of the Syrian dialect.
  - Used fixed-length temporal windows (84 and 168 frames).
  - Implemented an Enhanced Reconstruction Loss combining spatial distance ( $L_{pose}$ ) and velocity smoothness ( $L_{delta}$ ).
  - Trained VQ-VAE independently with Exponential Moving Average (EMA) to prevent codebook collapse.
SignCLIP (Retrieval Approach):
- Original: Contrastive learning model aligning text and video in a shared embedding space.
- Adaptation (AraSignCLIP):
  - Used OPUS-MT to translate Arabic text to English.
  - Constructed prompts using ISO 639-3 codes (e.g., <eng> <jos> Ministry of Health), using Jordanian Sign Language (jos) as the closest proxy for Syrian Sign Language.
  - Tokenized with BERT base uncased.
- Goal: Text-to-sign retrieval via similarity matching.

4. Experimental Results

The dataset was split 80/10/10 (Train/Val/Test).

Model	Best Metric	Result	Observation
MotionCLIP	Validation Loss (Decoder)	0.5285	Performed well in retrieval/embedding alignment but struggled with stable motion generation.
T2M-GPT	Cross-Entropy Test Loss	0.0242	Showed promising qualitative behavior but was highly sensitive to dataset size. Low perplexity attributed to limited vocabulary diversity.
SignCLIP	Recall@5 (R@5)	0.5526	Produced the most stable results for retrieval-based tasks due to reliance on similarity matching rather than full synthesis.

Key Findings:

Retrieval vs. Generation: Retrieval-based models (SignCLIP) and alignment models (MotionCLIP) demonstrated more stability than generative models (T2M-GPT) in this low-resource setting.
Data Dependency: Generative models require large-scale, diverse datasets to generalize effectively; the current 1,500-sample limit constrained T2M-GPT's performance.
Pose Estimation: MediaPipe proved superior to SMPL-X for this application, particularly in handling hand overlaps which caused SMPL-X to fail.

5. Significance and Future Work

Significance: This work bridges a critical gap in accessibility for the Syrian DHH community by providing the first dedicated dataset for SyArSL. It establishes a baseline for automated news translation, potentially reducing reliance on scarce human interpreters.
Limitations:
- Word-Level Scope: The dataset is limited to single words, preventing sentence-level translation.
- Non-Manual Features: Facial expressions (crucial for grammar in sign language) were partially excluded or simplified to ensure model stability.
- Evaluation: Lacks standardized production metrics and human evaluation due to the ambiguity of sign language.
Future Directions:
- Scaling the dataset to include more signers and moving from word-level to sentence-level recording.
- Re-integrating full facial features to capture non-manual linguistic markers.
- Implementing standardized evaluation metrics, including back-translation and human assessment.

In conclusion, while current generative models struggle with the limited data size, the introduction of SyriSign and the comparative analysis of SoTA architectures provide a foundational step toward automated, accessible news delivery for the Syrian deaf community.