💻 computer science

JSL-DC: A Word-Level Japanese Sign Language Dataset with Linguist-Derived Descriptions for Distinguishing Confusable Signs

This paper introduces JSL-DC, the largest Deaf-centric Japanese Sign Language dataset featuring 36.7K videos and linguist-derived descriptions for confusable signs, which enables a new model to outperform state-of-the-art recognition methods by 9.8% on challenging subsets.

Original authors: Ken Takaki, Asuka Ando, Misa Suzuki, Uiko Yano, Masaya Tsujimoto, Bill Neubauer, Ananay Vikram Gupta, Rose Shao, Matthias Hoppe, Sahir Shahryar, Celeste Mason, Kai Kunze, Yohei Oseki, Yoshihiro Kawaha

Published 2026-08-20

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ken Takaki, Asuka Ando, Misa Suzuki, Uiko Yano, Masaya Tsujimoto, Bill Neubauer, Ananay Vikram Gupta, Rose Shao, Matthias Hoppe, Sahir Shahryar, Celeste Mason, Kai Kunze, Yohei Oseki, Yoshihiro Kawahara, Thad Starner

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

For many children who are deaf, the most critical years for language development happen before they ever enter a classroom. During this time, they rely on their parents to teach them how to communicate. Yet, statistics show that ninety-five percent of deaf children are born to hearing parents who do not know sign language. This creates a silent gap in the home, where parents struggle to connect with their children because they lack the vocabulary to express basic needs, emotions, or concepts. While online videos exist to help parents learn, true fluency often requires hands-on practice, a method that has proven difficult to scale without the right tools. To build effective learning aids, computer scientists need vast libraries of video data showing how different people perform sign language words. However, for Japanese Sign Language, such data has been scarce, often limited to a single person or a small set of words, making it impossible to train computers to recognize the language from a new user.

A team of researchers has now filled this void with the creation of JSL-DC, the largest collection of Japanese Sign Language videos ever assembled. The project was driven by a simple but profound realization: to teach a computer to understand sign language, the data must come from the people who use it every day. Instead of hiring actors or students to mimic signs, the researchers recruited nineteen deaf adults who use Japanese Sign Language as their primary mode of communication. These participants recorded themselves performing 270 specific words, chosen specifically to help hearing parents communicate with their young children. The resulting dataset contains over 36,000 videos, a volume three times larger than any previous Japanese sign language collection.

The true innovation of this work lies not just in the volume of data, but in how it was curated. The researchers worked closely with deaf linguists to select words that are most useful for early childhood communication, prioritizing verbs and adjectives over nouns, as these are the building blocks of early sentences. More importantly, the team identified pairs of signs that look nearly identical to the untrained eye but carry different meanings. In many cases, these signs differ only in subtle facial movements or mouth shapes, cues that standard computer vision models often miss. To solve this, the linguists provided written descriptions explaining exactly how to tell these confusing pairs apart. They noted, for instance, that the sign for "bitter" and the sign for "spicy" use the same hand movements but require different mouth shapes to distinguish them.

The researchers then tested whether these human descriptions could actually improve a computer's ability to recognize signs. They built a system that first identified the sign using standard video analysis, and then, if the sign was one of the confusing pairs, it examined the mouth movements specifically to make the final decision. This approach, inspired directly by the linguists' notes, significantly improved the computer's accuracy on the difficult signs. The system correctly identified the confusing words nearly ten percent more often than the best existing methods. This proves that incorporating human linguistic knowledge into machine learning models can bridge the gap where pure visual data fails.

The team also demonstrated how this technology can be applied in the real world by adapting a popular arcade game into a learning tool. In this version, players sign a word to release a colored ball into a field of other balls. If the computer correctly recognizes the sign, the ball matches the color and clears the screen; if the sign is wrong, the ball is the wrong color. This interactive game allows hearing parents to practice signing in a low-pressure environment, receiving immediate feedback on their performance. The entire dataset, along with the linguistic descriptions and the game software, is being released to the public to accelerate further research and development.

By centering the project on the deaf community, from the selection of the words to the final review of every video, the researchers ensured the data is authentic and reliable. Every video was checked twice by deaf experts to ensure the signs were performed correctly and that no personal information was accidentally captured in the background. This rigorous process filtered out nearly ten percent of the initial recordings, leaving a clean, high-quality resource. The work suggests that the future of sign language technology lies in collaboration between linguists and engineers, using human insight to guide machines. As these tools improve, they offer a tangible path toward closing the communication gap in homes across Japan, allowing parents to finally speak the language of their children.

Technical Summary: JSL-DC

Problem Statement

Effective sign language (SL) acquisition is critical for the language development of deaf children, yet 95% are born to hearing parents who often lack proficiency in SL. While isolated sign language recognition (ISLR) models can power learning tools to bridge this communication gap, their development for Japanese Sign Language (JSL) is severely hindered by a lack of large-scale, multi-signer datasets. Existing JSL datasets are either limited to a single signer (preventing generalization to new users) or contain a very limited vocabulary. Furthermore, existing datasets rarely identify "confusable" sign pairs—those that share handshapes but differ in non-manual cues like mouth movements or context—and provide no linguistic guidance on how to distinguish them. This leaves computer vision researchers without the necessary linguistic data to improve recognition accuracy on these difficult cases.

Methodology

The authors introduce JSL-DC, a dataset collection process and resulting dataset designed with a "Deaf-centric" philosophy ("by the community, for the community").

Data Collection Process

Participants: 19 Deaf adults (11 male, 8 female, ages 18–70s) who use JSL daily. All participants were recruited via snowball sampling.
Lexicon Selection: A lexicon of 270 JSL words was curated by Deaf and Coda (Child of Deaf Adult) linguists. The selection prioritized verbs, adjectives, and greetings essential for early parent-child communication, rather than nouns which can often be substituted with pointing.
Recording: Participants used a custom open-source app on Google Pixel Tablets to record 20 repetitions of each of the 270 signs in their natural home environments. Reference videos were shot in a studio to guide participants.
Review Pipeline: A rigorous two-stage review process was conducted by Deaf and Coda linguists.
1. Initial Quality Check: Filtering for unnatural signing and privacy issues.
2. Linguistic Verification: Deaf linguists reviewed clips to ensure the sign matched the reference meaning. This step was crucial for "confusable" signs; 9.8% of clips for these specific glosses were removed as variants because the meaning differed from the reference.

Dataset Composition

Scale: 36,705 videos covering 270 signs (average 136 repetitions per sign).
Split: Data is divided into train (15 signers), validation (2 signers), and test (2 signers) sets, ensuring a disjoint signer setup to evaluate generalization to unseen users.
Unique Feature: The dataset includes 22 glosses that form a set of confusable signs which cannot be distinguished by handshape alone. For these glosses, the authors provide linguist-derived descriptions explaining the distinguishing features (e.g., specific mouth movements, eye gaze).

Experimental Approach

To validate the utility of the linguistic descriptions, the authors proposed a mouthing-based reranker.

Base Models: They evaluated existing state-of-the-art ISLR models (I3D, NLA-SLR, SPOTER, MASA) on the dataset.
Proposed Model: Among the 22 confusable glosses, 16 were identified as distinguishable by mouth movement according to the linguist-derived descriptions. For these 16 glosses, they integrated an AV-HuBERT model (typically used for lip-reading) to extract mouth-region features.
Reranking Logic: If a base ISLR model predicted one of the 16 confusable glosses, an SVM classifier trained on the AV-HuBERT features was used to re-rank the prediction based on the mouth movement description.

Key Contributions

JSL-DC Dataset: The largest JSL dataset by video count (36.7K videos), exceeding previous multi-signer datasets by 3x. It is the first JSL dataset collected using mobile devices in natural settings with a diverse group of native signers.
Deaf-Centric Curation: The entire process, from lexicon selection to data review, was led by Deaf and Coda experts, ensuring the data reflects authentic daily usage and high-quality standards.
Linguist-Derived Descriptions: The dataset includes explicit annotations for 22 confusable glosses, detailing how to distinguish them (e.g., via mouthing). This provides a novel resource for guiding model attention.
Demonstrated Performance Gain: The authors demonstrated that incorporating these linguistic descriptions into a model architecture significantly improves recognition accuracy on confusable signs.

Results

Baseline Performance: State-of-the-art models achieved high accuracy (>90%) on the general 270-gloss set but struggled significantly on the confusable subset (e.g., NLA-SLR achieved 58.1% F1 on confusable glosses vs. 94.9% on others).
Impact of Linguistic Descriptions: By adding the mouthing-based reranker inspired by the linguist descriptions:
- The F1 score on the confusable subset improved by 9.8% (from 53.5% to 63.3% for the RGB-based NLA-SLR).
- The overall Top-1 accuracy improved from 91.9% to 92.5%.
- Statistical significance was confirmed via a Wilcoxon signed-rank test ( $p < 0.01$ ).
Application Prototype: The authors developed a prototype "PopSign JSL" game (a bubble-bursting game) using a 3-layer Bi-LSTM recognizer trained on the dataset, demonstrating real-time performance on smartphones for both hearing parents learning JSL and Deaf students learning English.

Significance and Claims

The paper claims that JSL-DC addresses a critical gap in sign language research by providing a large-scale, multi-signer dataset specifically curated for sign language learning tools rather than translation. The authors emphasize that the inclusion of linguist-derived descriptions is a key innovation; their experiments prove that these descriptions can directly guide model architecture to resolve ambiguities that pure visual features miss.

The dataset is released under a CC-BY 4.0 license to accelerate research. The authors position this work as a foundational step toward supporting language development in deaf children by enabling hearing parents to acquire basic JSL vocabulary through interactive tools. They explicitly note that the dataset is not intended for establishing a standard for JSL or for JSL-to-Japanese translation, but rather for recognition tasks in educational contexts.