Technical Summary: Emotion Recognition in Sign Language Conversation

Problem Statement

Current research in Emotion Recognition in Conversation (ERC) and sign language analysis faces a critical structural limitation: existing resources primarily consist of isolated sentences or unidirectional expressions (e.g., eJSL Solo, EmoSign). These datasets lack conversational context, preventing models from utilizing historical dialogue flow to understand emotional dynamics. Consequently, models trained on isolated utterances demonstrate degraded performance in real-world scenarios where emotional meaning is contingent upon continuous interaction and interlocutor responses. Furthermore, sign language presents a unique challenge where facial expressions and upper body movements serve dual purposes: encoding grammatical structures (e.g., questions) and affective states simultaneously. This overlap creates significant ambiguity for automatic recognition systems, particularly when applying generic multimodal ERC models designed for spoken languages to sign language data.

Methodology

Dataset Construction: eJSL Dialog

To address the lack of conversational data, the authors constructed the eJSL Dialog dataset.

• Source: Scripts were derived from the STUDIES Japanese Empathetic Dialogue Speech Corpus, specifically a subset of 480 unique dialogues involving teacher-student interactions.
• Scale: The dataset comprises 1,920 video samples (4 utterances per dialogue).
• Participants: Two native Japanese Sign Language (JSL) signers (one male, one female) acted as both teachers and students.
• Recording: Videos were recorded in a controlled indoor environment with a white background using a single RGB camera (1440 × 1080 resolution, 30 fps). Actors were provided with text and emotion labels on-screen to ensure consistent performance.
• Annotations: Each utterance is annotated with one of four emotion categories: Neutral, Happy, Sad, and Angry. The dataset is balanced across these categories and includes explicit dialogue history.
• Purpose: The dataset is designed strictly as a benchmark for evaluation and testing, not for training, to serve as a gold standard for assessing model generalization.

Baseline Evaluation Framework

The authors established an objective benchmark by evaluating five distinct models on the eJSL Dialog dataset to isolate the contributions of modality and context:

1. Visual Models:
   • EmoAffectNet: Processes frame-level facial features.
   • EANwH: Extends EmoAffectNet by concatenating facial and hand skeletal features, utilizing an LSTM to capture temporal dynamics. Both were pre-trained on the BOBSL (British Sign Language) dataset to learn generic visual features before fine-tuning.
2. Text-Based Models:
   • EmoTrans: An ERC model operating exclusively on textual input (Japanese transcripts translated to English), concatenating recent utterances to capture state changes.
3. Multimodal Conversational Models:
   • TelME: Uses cross-modal knowledge distillation and shifting fusion to combine textual and non-verbal features.
   • MMGCN: A multimodal fused graph convolutional network modeling inter- and intra-speaker dependencies.
   • Note: Audio modalities were excluded from these multimodal models to align with the visual/text nature of the dataset.

Key Results

The evaluation revealed significant performance disparities and a distinct domain gap:

• Visual Models: EANwH (33.31 weighted F1) outperformed EmoAffectNet (29.06 weighted F1), confirming that hand skeletal features and temporal modeling provide explicit affective information. However, both struggled with the "Sad" category (F1 scores of 13.24 and 17.03, respectively).
• Text-Only Model: EmoTrans achieved the highest overall performance (55.40 weighted F1), demonstrating that textual semantics provide direct and generalizable emotional cues, especially given the high correlation between the script text and assigned emotion labels in the source corpus.
• Multimodal Conversational Models: Generic multimodal models (TelME and MMGCN) exhibited severe performance degradation, achieving weighted F1 scores of 10.46 and 8.50, respectively.
  • Analysis: The authors attribute this failure to a domain gap. Generic models trained on non-signing populations struggle to distinguish between linguistic markers (grammatical facial expressions) and emotional transitions in sign language. The visual features in sign language interfere with the text modality during fusion, degrading overall performance.

Case Studies

Two cases highlighted the complementary nature of modalities:

1. Context Dependency: In a scenario where a student's visual expression was restrained but the context was positive, text-based and multimodal models correctly identified "Joy," while the visual-only model failed, predicting "Neutral."
2. Visual Necessity: In a scenario where the text described a factual event without explicit emotional modifiers, text-only models failed, while visual-aware models correctly identified "Joy" based on facial cues.

Contributions

The paper makes three primary contributions:

1. Task Definition: It formally defines the Emotion Recognition in Conversation (ERC) task specifically for sign language video analysis, establishing a benchmark for bidirectional interaction scenarios.
2. Dataset Release: It introduces the eJSL Dialog dataset, the first resource providing sign language video samples with explicit multi-turn dialogue context and emotion annotations, addressing the structural limitations of isolated utterance datasets.
3. Empirical Findings: Through systematic benchmarking, the study demonstrates the limitations of isolated visual models and generic multimodal conversational architectures. It confirms the explicit need for context-aware visual extractors specific to sign language rather than relying on generic architectures.

Significance and Future Directions

The paper concludes that expanding the scale of conversational sign language datasets is a necessary next step to support large-scale pre-training. The findings indicate that current generic multimodal models cannot simply be transferred to sign language tasks due to the unique dual-role nature of visual features in sign language. Future work aims to address current limitations (such as the controlled environment and limited number of signers) by collecting spontaneous conversations, expanding the dataset scale, and integrating large language models to better process multi-turn dialogue history. This research lays the groundwork for developing empathetic dialogue systems capable of tracking and responding to the emotional history of deaf users in natural human-computer interactions.