🤖 AI

Deep Learning-Based Sign Language Recognition from Videos and Cross-Lingual Translation to Indian Vernaculars

This paper presents a two-stage deep learning pipeline that utilizes a fine-tuned VideoMAE transformer to recognize isolated Indian Sign Language gestures from video and translates the resulting English labels into Hindi, Telugu, and Bengali using the NLLB-200 model, while also providing a Streamlit demo and analyzing performance limitations on a small-scale dataset.

Original authors: Chandranath Adak, Ramesh Nandipalli

Published 2026-06-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Chandranath Adak, Ramesh Nandipalli

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a computer to understand a secret language spoken with hands and faces instead of words. That is essentially what this paper does, but with a specific twist: it helps translate Indian Sign Language (ISL) into three major Indian spoken languages (Hindi, Telugu, and Bengali) by using English as a middleman.

Here is a simple breakdown of how they built this system, using everyday analogies.

The Big Picture: A Two-Step Relay Race

The researchers didn't try to teach the computer to jump straight from "Hand Signs" to "Hindi Words." That would be like trying to teach a dog to speak French directly. Instead, they set up a two-stage relay race:

Stage 1 (The Translator): A video camera watches a person signing. A smart AI (called VideoMAE) watches the video and says, "Ah, that sign means the English word 'Happy'."
Stage 2 (The Interpreter): Once the AI knows the word is "Happy," it passes that word to a second AI (called NLLB). This second AI acts like a polyglot translator, instantly turning the English word "Happy" into "Khush" (Hindi), "Santosham" (Telugu), and "Sukhi" (Bengali).

The Ingredients: What They Used

The Teacher (VideoMAE): Think of this as a student who has already watched millions of hours of general videos (like movies and sports) and learned how to recognize actions. The researchers took this "super-student" and gave them a crash course specifically on sign language videos.
The Textbook (The Dataset): They didn't have a whole library of sign language videos to work with due to computer storage limits. Instead, they used a small, curated "study guide" from IIT Madras. It contained videos of just 13 specific signs (like "loud," "quiet," "hat," "dress," "deaf," "blind").
The Test: They split their small study guide into a "practice test" (training) and a "final exam" (validation).

How It Performed: The Report Card

The results were a mix of "A-plus" and "needs improvement," which is common when learning a new skill with limited practice.

The Practice Run (Training): The AI studied the 13 signs and got 99% accuracy. It was like a student who memorized the flashcards perfectly.
The Final Exam (Validation): When tested on new, unseen videos, the accuracy dropped to 78%. This is still a solid score, but it shows the AI was a bit too focused on memorizing the practice cards rather than truly understanding the concept.

Where the AI Got Stuck (The Glitches):
The paper highlights two specific types of confusion, like a student mixing up similar-looking words:

The "Clothing" Mix-up: The AI sometimes confused signs for clothing items. For example, it might mistake a sign for a "Hat" for a "Dress" or a "Shirt" for a "Suit." This makes sense because these signs often involve similar hand movements near the head or torso.
The "Feeling" Mix-up: The AI struggled with abstract concepts like "Beautiful," "Ugly," "Deaf," and "Blind." It often confused these with each other or with the word "Sad." The researchers suspect this is because these signs rely heavily on facial expressions, and the AI didn't have enough examples to learn the subtle differences.

The Demo: A User-Friendly Tool

The team didn't just stop at the math; they built a Streamlit app (a simple web interface).

How it works: A user uploads a short video of someone signing.
What happens: The app extracts 16 frames (like taking 16 quick snapshots), runs them through the AI, and displays the result.
The Output: It shows the English word (e.g., "Happy") and immediately translates it into Hindi, Telugu, and Bengali scripts.

The Limitations: What This System Can't Do Yet

The authors are very honest about the boundaries of their work. This is not a magic wand that can translate a whole conversation yet.

One Word at a Time: The system only understands isolated words (like "Hat"). It cannot understand a full sentence like "I am wearing a hat." It's like a dictionary that knows words but not grammar.
Small Vocabulary: It only knows 13 specific words. If you sign something else, it won't know what to do.
No Real-Time Speed: It's designed to process uploaded videos, not to watch a live signer and translate instantly in real-time.
Context Issues: Because it translates single words, it can get confused. For example, the word "Suit" could mean a piece of clothing or the phrase "to suit someone." Without a full sentence, the computer has to guess.

The Bottom Line

This paper is a proof-of-concept. It proves that you can take a powerful video AI, teach it a few sign language words, and connect it to a translation AI to bridge the gap between sign language and regional Indian languages. While it's not ready to replace a human interpreter in a courtroom or hospital yet, it's a working prototype that shows the path forward for making technology more accessible to the deaf and hard-of-hearing community in India.

Technical Summary: Deep Learning-Based Sign Language Recognition from Videos and Cross-Lingual Translation to Indian Vernaculars

Problem Definition
Sign language serves as the primary communication mode for the global deaf and hard-of-hearing community, yet automated systems capable of recognizing sign gestures from video and translating them into natural language remain limited, particularly for low-resource Indian languages. Existing solutions often rely on static image classification or handle only narrow vocabularies, failing to capture the temporal, whole-body nature of signing. Furthermore, a scarcity of parallel corpora linking Indian Sign Language (ISL) directly to regional languages (such as Hindi, Telugu, and Bengali) hinders the development of end-to-end translation systems. This work addresses two coupled sub-problems: (1) predicting the English word label from a short ISL video clip, and (2) translating that English label into Hindi, Telugu, and Bengali.

Methodology
The authors propose a two-stage deep learning pipeline designed to bypass the lack of direct sign-to-regional-language data by using English as a high-resource pivot language.

Video-to-English Classification:
- Architecture: The system utilizes a fine-tuned VideoMAE (Masked Autoencoder for Video) transformer, specifically the MCG-NJU/videomae-base checkpoint from Hugging Face.
- Input Processing: Input videos are decoded, and 16 frames are sampled uniformly. These frames are resized to 224×224 and normalized using ImageNet statistics.
- Training: The model is fine-tuned on a 13-class subset of the AI4Bharat Indian Sign Language corpus (released by IIT Madras). The dataset consists of approximately 197 clips covering adjectives (e.g., loud, quiet, happy, sad, beautiful, ugly, deaf, blind) and clothing nouns (e.g., hat, dress, suit, skirt, shirt).
- Configuration: The classification head is replaced with a 13-unit linear layer. Training employs the AdamW optimizer (learning rate $1 \times 10^{-5}$ ), cross-entropy loss, mixed-precision (FP16), and a batch size of 2 for 15 epochs. The training set is split 80/20 (stratified) from the available clips.
English-to-Regional Translation:
- Architecture: The predicted English label is passed to Meta AI's NLLB-200 (No Language Left Behind) multilingual translation model (facebook/nllb-200-distilled-600M).
- Process: The model translates the isolated English word independently into three target languages: Hindi (hin_Deva), Telugu (tel_Telu), and Bengali (ben_Beng). The translation is performed by forcing the decoder's beginning-of-sequence token to the specific target language code.
Inference System:
- The pipeline is packaged into an interactive Streamlit application. Users upload video files, and the system outputs the predicted English label alongside its translations in the three vernaculars.

Key Results
Evaluated on a 40-clip validation set (20% of the 197 total clips), the system achieved the following:

Accuracy: The model reached 99.4% training accuracy and 77.5% validation accuracy after 15 epochs.
Per-Class Performance:
- High Performance: Classes such as loud, quiet, happy, Suit, and Skirt achieved perfect recall (100%).
- Moderate Performance: Hat, Dress, and Shirt achieved 75% recall.
- Failure Modes: The model failed completely (0% recall) on Beautiful, Deaf, and Blind, and performed at chance levels on Ugly.
Error Analysis: Confusion primarily occurred in two clusters:
1. Visual/Postural: Clothing items (Hat, Dress, Shirt, Suit) were confused due to overlapping hand placements near the torso and head.
2. Semantic/Affective: Beautiful, Ugly, Deaf, and Blind were mutually confused, likely due to overlapping facial expressions and hand cues, exacerbated by extremely small sample sizes (1–2 validation clips per class).
Overfitting: The significant gap between training (>99%) and validation (~78%) accuracy indicates overfitting, attributed to the small dataset size and limited intra-class diversity.

Significance and Scope
The paper positions this work as an initial, small-scale feasibility study rather than a production-ready system. Its primary contributions are:

Demonstrating a Two-Stage Pipeline: Validating that a pretrained VideoMAE encoder, when fine-tuned on a small ISL subset, can effectively classify isolated gestures, and that an English-pivot approach using NLLB-200 can bridge the gap to low-resource Indian languages without requiring direct sign-to-regional corpora.
Identifying Limitations: The study explicitly highlights the challenges of isolated-word classification, the sensitivity to signer style and recording conditions, and the ambiguity inherent in translating single words without sentence-level context (e.g., distinguishing "suit" as clothing vs. a verb).
Reproducibility: The authors have released the code for data loading, model fine-tuning, and the Streamlit inference demo to support community extension.

Future Directions
The authors outline specific paths for future work, including scaling to the full AI4Bharat vocabulary, transitioning from isolated-word classification to continuous sign language translation (generating full sentences), expanding to additional NLLB-supported Indian languages, and optimizing model latency for edge/mobile deployment.