💻 computer science

Toward Phonology-Guided Sign Language Motion Generation: A Diffusion Baseline and Conditioning Analysis

This paper proposes a diffusion-based baseline for generating 3D sign language motions that outperforms state-of-the-art methods by systematically analyzing how text encoders and phonological attribute conditioning strategies, particularly the translation of symbolic annotations to natural language, influence generation quality.

Original authors: Rui Hong, Jana Kosecka

Published 2026-03-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Rui Hong, Jana Kosecka

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to speak American Sign Language (ASL). The robot has a body made of digital joints (like a 3D avatar), and you want it to move its hands and face naturally to convey a specific word, like "BOOK" or "COOL."

The problem is that ASL isn't just about the word; it's about the physics of the sign. A sign is made up of specific ingredients: What shape are the fingers? Where is the hand? How does it move?

This paper is like a cooking experiment where the researchers are trying to figure out the best way to give the robot a recipe so it can cook up the perfect sign.

The Ingredients: Phonology

In spoken language, words are made of sounds (phonemes). In sign language, words are made of phonological attributes. Think of these as the "DNA" of a sign:

Handshape: Is it a fist? A flat hand?
Location: Is the hand near the chin or the chest?
Movement: Does it wiggle, slide, or punch?

The researchers used a massive dictionary called ASL-LEX 2.0 that breaks down thousands of signs into these specific ingredients.

The Kitchen: The Diffusion Model

The "chef" in this experiment is a type of AI called a Diffusion Model.

The Analogy: Imagine a blurry, noisy photo of a person signing. The AI's job is to slowly "denoise" this picture, step-by-step, until a clear, sharp video of the sign emerges.
The Baseline: They first built a very strong chef (a baseline model) that could already cook decent signs just by being told the word (the "gloss"). This new chef was already better than the previous best chef (called SignAvatar) at making signs that looked like the real thing.

The Experiment: How to Give the Recipe

The researchers wanted to know: Does giving the chef the specific "ingredients" (phonological attributes) make the sign even better?

They tested three main variables:

The Translator (Text Encoder): How does the computer read the recipe?
- CLIP: A translator that is great at understanding common English words and pictures, but gets confused by weird symbols.
- T5: A translator that is a master of grammar and structure, regardless of whether the words are normal or weird symbols.
The Recipe Format (Notation):
- Symbolic: The raw code from the dictionary (e.g., "open b" for a hand shape). This is like giving the chef a recipe written in a secret code only linguists understand.
- Mapped (Natural Language): Translating that code into plain English (e.g., "all four fingers extended together"). This is like rewriting the recipe in a language the chef actually speaks.
The Condition:
- Gloss Only: Just telling the chef "Make 'BOOK'."
- Gloss + Attributes: Telling the chef "Make 'BOOK', and here are the exact finger shapes and locations to use."

The Big Discovery: The "Translation" Problem

Here is the most interesting part of the story:

The CLIP Chef: When they gave CLIP the raw symbols (the secret code), the robot went crazy and made terrible signs. It was like giving a chef a recipe written in alien hieroglyphs; it had no idea what to do.
- However, once they translated those symbols into plain English, CLIP became a superstar. It used the extra details to make signs that were even more accurate than before.
- Lesson: CLIP needs the recipe to be in "human language" to work with extra details.
The T5 Chef: This chef didn't care. Whether the recipe was in secret code or plain English, T5 performed about the same. It's like a chef who can read any language, so translating the recipe didn't change much.
- But, T5 wasn't as good at understanding the simple word "BOOK" to begin with compared to CLIP.

The Result: A Better Robot

The winning combination was using the CLIP chef but giving it the translated, plain-English recipe with all the extra phonological details.

Before: The robot made signs that were okay, but sometimes the hands looked stiff or the movements were wrong.
After: The robot made signs that were smoother, more diverse, and much more recognizable as the correct sign. It beat the previous best robot (SignAvatar) in almost every category.

Why Does This Matter?

Control: If we can tell the robot exactly how to shape its hand and where to move it, we can fix mistakes. If the robot makes a sign that looks weird, we can tweak the "ingredients" without retraining the whole brain.
New Signs: Because the robot understands the parts of a sign (hand shape + location + movement), it might eventually be able to invent new, valid signs by mixing and matching these parts, just like we mix and match sounds to make new words.
Accessibility: This technology helps create better avatars for deaf and hard-of-hearing people to communicate with the hearing world, making virtual interactions feel more natural and human.

The Catch (Limitations)

The researchers admit there is still some "noise" in the kitchen. The dictionary (ASL-LEX) describes the "perfect" version of a sign, but real people signing on video (the training data) might do it slightly differently. Sometimes the robot gets confused because the "recipe" doesn't perfectly match the "video" it's trying to copy.

In summary: To teach a robot to sign well, you can't just give it the word. You have to give it the "ingredients" of the sign, but you must write those ingredients in a language the robot's brain can actually understand (plain English, not secret codes). When you do that, the robot becomes a much better signer.

1. Problem Statement

The generation of natural, linguistically correct, and visually smooth 3D avatar sign language motion from text inputs remains a significant challenge. Existing approaches often rely on gloss labels (written representations of signs) but ignore the internal phonological structure of sign language.

The Gap: Unlike spoken language, which decomposes words into phonemes, sign language decomposes signs into articulatory parameters (hand shape, location, movement, palm orientation, etc.). Current generative models often treat signs as atomic units, missing the compositional nature of the language.
The Goal: To investigate whether incorporating phonological attributes (specifically from the ASL-LEX 2.0 database) as conditioning signals can improve the quality, discriminability, and controllability of 3D sign language motion generation within a diffusion framework.

2. Methodology

A. Motion Representation

Model: The system uses the SMPL-X body model, focusing on the upper body (43 joints: spine, torso, arms, hands).
Parameterization: Joint rotations are represented in 6D continuous rotation representation.
Sequence: Fixed length of $T=60$ frames. The root joint is normalized to a canonical orientation to focus on relative articulation.

B. Diffusion Framework

The authors adopt a Human Motion Diffusion Model (MDM) architecture:

Denoiser: A Transformer Encoder (4 layers, 8 heads, 512 dim) that predicts the clean motion $x_0$ from noisy input $x_t$ , timestep $t$ , and condition $c$ .
Training Objective: A combination of MSE reconstruction loss and velocity loss (to ensure temporal smoothness).
- Weighting: Losses are weighted per body part ( $w_g$ ), heavily prioritizing arms and hands (weight 5.0) over the torso (weight 0.5) to reflect the linguistic importance of hand dynamics in sign language.
Inference: Uses DDIM sampling with 50 steps.

C. Conditioning Mechanisms

The core innovation lies in how conditioning inputs are processed. The authors compare:

Text Encoders:
- CLIP (ViT-B/32): Trained on image-text pairs; produces 512-d embeddings.
- T5 (Base): A seq2seq language model; produces 768-d mean-pooled embeddings.
Conditioning Modes:
- Gloss-only: Input is just the gloss label (e.g., "BOOK").
- Gloss + Phonological Attributes: The gloss is concatenated with 8 specific attributes from ASL-LEX 2.0 (Handshape, Sign Type, Path Movement, Locations, etc.).
Attribute Notation Formats:
- Symbolic: Uses raw ASL-LEX symbols (e.g., "open b", "imrp").
- Mapped (Natural Language): Translates symbols into descriptive English sentences (e.g., "all four fingers extended together").

3. Key Contributions

Strong Diffusion Baseline: Established a new state-of-the-art baseline using an MDM-style diffusion model on SMPL-X motion. This baseline outperforms SignAvatar (a leading CVAE method) in gloss discriminability while maintaining comparable distributional quality.
Systematic Conditioning Analysis: Conducted a comprehensive ablation study over text encoders, conditioning modes, and notation formats.
Discovery of Input Representation Sensitivity: Revealed that the effectiveness of attribute conditioning is heavily dependent on the text encoder's ability to handle the input format.
- CLIP requires attributes to be translated into natural language to function effectively; symbolic notation causes performance collapse.
- T5 is robust to both symbolic and natural language formats but generally underperforms CLIP on gloss-only tasks.
Proposed Future Direction: Motivated the shift from text-string encoding to Structured Tensor Conditioning, where gloss and attributes are encoded via independent pathways to preserve compositional independence.

4. Experimental Results

Dataset: ASL3DWord (derived from WLASL), containing 103 gloss classes with SMPL-X motion data. ASL-LEX 2.0 provided the phonological annotations.

Key Findings:

Baseline Performance: The diffusion baseline (Gloss-only, CLIP) achieved 0.838 Accuracy (vs. 0.758 for SignAvatar) and 0.378 KNN Accuracy (vs. 0.271), demonstrating superior sign recognition capabilities.
Impact of Notation Format (CLIP):
- Symbolic: Performance collapsed (Accuracy dropped to 0.201, FID increased to 127.95).
- Mapped (Natural Language): Performance recovered and improved (Accuracy 0.794, FID 55.11).
- Reasoning: CLIP is trained on natural language; symbolic tokens are out-of-distribution. Furthermore, different signs often share identical symbolic attributes, causing the attribute string to overwhelm the gloss token in the pooled embedding.
Impact of Notation Format (T5): T5 showed negligible difference between symbolic and mapped formats (Accuracy ~0.52–0.54), as its tokenizer handles symbolic strings naturally. However, T5's overall performance was lower than CLIP's best variant.
Best Overall Model: CLIP + Gloss + Mapped Attributes achieved the best FID (55.11) and outperformed SignAvatar across all metrics.
Variance Analysis:
- SignAvatar consistently under-articulated (Variance Ratio < 1.0, especially for hands).
- Diffusion models showed mild over-articulation (Variance Ratio > 1.2 for hands/arms), suggesting the diffusion process captures dynamics well but may lack fine-grained precision.

5. Significance and Conclusion

Linguistic Grounding: The paper demonstrates that incorporating phonological structure (hand shape, location, movement) significantly enhances the controllability and quality of sign language generation, moving beyond simple gloss-to-motion mapping.
Encoder Selection: The study highlights that input representation is the critical bottleneck. For CLIP-based systems, translating structured linguistic data into natural language is a necessary preprocessing step.
Future Directions: The authors argue that while natural language mapping works, the ideal solution is Structured Tensor Conditioning. This would encode gloss and attributes through independent pathways (e.g., using FiLM modulation), avoiding the semantic dilution caused by concatenating text strings and better preserving the compositional nature of sign language phonology.

Limitations: The study is limited to a 103-word vocabulary. Additionally, there is a "noisy alignment" issue where the canonical phonological attributes in ASL-LEX do not always perfectly match the specific signing variants in the WLASL video dataset, introducing label noise.