Low Resource Multimodal Translation of Nepali Spoken Words into Emotion-Conditioned Sign Language Avatars

Technical Summary: Low Resource Multimodal Translation of Nepali Spoken Words into Emotion-Conditioned Sign Language Avatars

Problem Statement
Current spoken-to-sign language systems often fail to capture the emotional context of speech, resulting in robotic, lexically accurate but emotionally flat interactions. This limitation is particularly acute for low-resource languages like Nepali and its corresponding sign language (Nepali Sign Language, NSL), where research and datasets are scarce. Existing solutions frequently neglect the integration of emotional expression, which is essential for natural, human-centered communication. Furthermore, many current systems lack the computational efficiency required for deployment on edge devices, limiting their accessibility in resource-constrained environments.

Methodology
The authors propose NEST-V1 (Nepali Emotion and Speech Transformer – Version 1), a lightweight, multimodal framework designed to translate spoken Nepali into emotion-conditioned NSL avatar animations. The system operates through a unified, shared-acoustic encoder architecture that simultaneously performs Automatic Speech Recognition (ASR) and emotion classification.

• Dataset Creation: The study utilizes a pilot dataset comprising 600 labeled audio samples from 50 speakers (aged 15–45). The dataset covers four common Nepali words ("thank you," "hello," "house," "me") across three emotional states (happy, neutral, sad). The words were selected to include both dynamic gestures (involving continuous motion) and static gestures (fixed poses). To address data scarcity, the authors employed augmentation techniques including semitone shifting, Vocal Tract Length Perturbation (VTLP), and speed variations, expanding the dataset to over 4,400 samples.
• Audio Preprocessing: Raw audio files were standardized to 16kHz, clipped or padded to 2-second durations, and converted into Mel spectrograms (128 frequency bins × 200 time frames). These were resized to 128×128 grayscale images to serve as input for the transformer model.
• Model Architecture: NEST-V1 adapts a Vision Transformer (ViT) for audio spectrograms. The architecture treats spectrograms as 2D images, dividing them into 16×16 patches. It employs a shared encoder consisting of three Transformer layers with 12 attention heads and an embedding dimension of 768. The model utilizes a single [CLS] token for classification, feeding into two parallel heads: one for keyword recognition (4 classes) and one for emotion classification (3 classes). This shared encoder design reduces parameter redundancy.
• Avatar Generation: Upon classification, the system retrieves pre-rendered 2D avatar frames. A frame-blending pipeline using linear interpolation (alpha blending) generates smooth animations by morphing between a base neutral pose and an emotionally expressive variant. The output is a looping animated GIF, ensuring the system remains lightweight and deployable on web browsers and mobile devices without high-end graphics rendering.

Key Contributions

1. First NSL Emotional Dataset: The paper presents the first NSL-based speech dataset annotated with emotional context, addressing a critical gap in low-resource sign language research.
2. Modular, Lightweight Pipeline: The system features a modular architecture with independent components for ASR and emotion recognition, facilitating scalability. It is specifically designed for low-resource deployment, supporting edge applications.
3. Parameter Efficiency: By sharing the acoustic encoder between tasks, the model achieves a 37% reduction in parameters compared to separate model architectures, maintaining a footprint of only 22.1 million parameters.

Experimental Results
The model was evaluated on a split of the augmented dataset (training, validation, and testing sets).

• Performance: The system achieved an ASR accuracy of 81.1% and an emotion recognition accuracy of 79.21% on the test set. Validation accuracies reached 79.6% for ASR and 76.54% for emotion recognition.
• Efficiency: The NEST-V1 model requires approximately 22.1M parameters and 2.189M FLOPs (for the patch embedding) to 473.4M FLOPs (total), with a memory footprint of roughly 89 MB during inference. This is significantly more efficient than standard ViT models (86.6M parameters) and comparable or superior to CNN-LSTM hybrids in terms of parameter count while maintaining competitive accuracy.
• Classification Metrics: F1-scores for the four gesture classes ranged from 0.69 to 0.78, while emotion classes achieved F1-scores between 0.70 and 0.83.

Significance and Claims
The authors position this work as a proof-of-concept pilot study demonstrating the technical feasibility of generating emotion-conditioned sign language avatars from spoken input in low-resource settings. The significance lies in establishing a scalable foundation for real-time, emotionally expressive communication systems for the hearing-impaired community.

The paper claims that NEST-V1 successfully bridges the gap between lexical translation and emotional context, moving beyond robotic gesture mimicking toward more authentic human interaction. While the current scope is limited to a small vocabulary and three emotions, the authors assert that the framework provides a clear pathway for future expansion to larger vocabularies, diverse emotional expressions, and real-world assistive applications. The work emphasizes that such systems can be deployed on edge devices, making advanced assistive technology more accessible.