How Pragmatics Shape Articulation: A Computational Case Study in STEM ASL Discourse

Technical Summary: How Pragmatics Shape Articulation in STEM ASL Discourse

Problem Statement
Current state-of-the-art sign language models are predominantly trained on interpreter-produced data or isolated vocabulary videos. These datasets capture clear, standardized productions but fail to account for the variability inherent in natural dialogue. Human communication is inherently adaptive, characterized by entrainment (adjusting linguistic behaviors to an interlocutor) and effort reduction (simplifying articulation through repetition). In sign languages, these adaptive processes manifest as spatiotemporal changes such as sign lowering, weak drop, and location undershooting. However, existing computational models often overlook these context-dependent articulatory shifts, leading to a gap in understanding how pragmatics shape sign articulation and how current technologies represent these variations. Specifically, there is a lack of quantitative data comparing dyadic interactive signing against isolated productions and monologic contexts within specialized domains like STEM (Science, Technology, Engineering, and Mathematics), where vocabulary fluidity is high.

Methodology
To address this gap, the authors constructed a multimodal dataset and a computational framework to analyze American Sign Language (ASL) STEM discourse.

• Data Collection: The study utilized motion capture (Vicon system, 120 fps, 73 markers) to record two fluent deaf signers (a biology instructor and a student) in two primary conditions:
  1. Isolated Vocabulary: The student produced 77 STEM signs individually to establish a baseline for spatial and temporal properties.
  2. Dyadic Dialogue: An 8.52-minute spontaneous conversation between the instructor and student covering biology topics (e.g., cell structure, genetics).
  • Supplementary Data: To disentangle individual effort reduction from dialogue-specific entrainment, the authors included:
    • Monologic Lecture: A solo lecture by the same instructor on the same topics.
    • Interpreter Data: Publicly available signed Wikipedia articles (ASL STEM Wiki) and interpreter-signed content, representing typical training data distributions.
• Annotation: Video data was annotated using ELAN following SLAASH ID glossing principles, focusing on sentence-level translation and specific STEM sign identification.
• Kinematic Metrics: The authors adapted gesture studies metrics to quantify articulatory variability:
  • Spatial Reduction: Measured via Spatial Extent (3D bounding box diagonal) and Path Length (cumulative distance traveled).
  • Temporal Reduction: Measured via Average Velocity and Articulation Duration.
  • Sign Lowering: Measured as the mean vertical displacement of the dominant hand.
• Computational Analysis:
  • Entrainment vs. Effort Reduction: The authors compared reductions in dialogue against monologue. Reductions present in both were attributed to individual effort; reductions unique to dialogue were attributed to entrainment.
  • Model Evaluation: Two pre-trained encoders, SignCLIP (skeletal keypoints, contrastive learning) and I3D (RGB video, action recognition), were evaluated.
  • Tasks:
    1. Embedding-based Entrainment: Analyzed changes in cosine similarity between signer embeddings over repeated mentions.
    2. Sign Spotting: A retrieval task where isolated signs served as queries to search continuous dialogue/monologue sequences, measuring Mean Reciprocal Rank (MRR) and Recall@k.
  • Avatar Rendering: To mitigate visual noise from motion capture suits, 3D avatars were rendered from motion data to test model robustness to out-of-distribution inputs.

Key Contributions

1. Framework for Disentanglement: Introduced a method to isolate dialogue-specific entrainment from individual effort reduction in ASL using continuous spatial, temporal, and vertical motion metrics.
2. Quantitative Evidence of Adaptation: Demonstrated that dialogue signing exhibits significant spatial and temporal reduction compared to isolated vocabulary, reflecting adaptive changes characteristic of interactive communication.
3. Context-Specific Reduction: Showed statistically significant spatial and temporal reduction in STEM signs across repeated mentions in dialogue, a pattern absent in monologue contexts, indicating that such adaptation reflects entrainment rather than general efficiency.
4. Model Limitations: Evaluated existing sign embedding models and demonstrated their failure to generalize to these pragmatic articulatory differences, particularly in interactive settings.

Results

• Articulatory Reduction: On average, signs in dialogue were 24.6% shorter (instructor) and 44.6% shorter (student) in duration compared to isolated vocabulary signs. Path lengths also showed a consistent downward trend in dialogue.
• Entrainment vs. Effort:
  • Dialogue: Showed systematic spatial and temporal reduction across repeated mentions, particularly in the left hand and arm (suggesting "weak drop").
  • Monologue: The same instructor showed little to no spatial reduction and inconsistent temporal trends in solo lectures, suggesting that the compression observed in dialogue is driven by interactional adaptation rather than individual efficiency.
  • Sign Lowering: No significant differences were found in vertical hand position, indicating that sign lowering is highly context-dependent and not a primary feature of this specific dialogue reduction.
• Model Performance:
  • Sign Spotting: Performance was highest in the instructor-monologue condition (closest to pretraining data) and lowest in dialogue conditions. SignCLIP failed to retrieve any correct items within the top-50 ranks in dialogue contexts.
  • Entrainment Metrics: While embedding similarity increased slightly between signers over time (suggesting some entrainment), the models struggled to maintain stable representations of the same sign across different pragmatic contexts.
  • Avatar Rendering: Rendering motion capture data as avatars did not consistently improve performance and, in some cases, reduced it, highlighting that the issue is not merely visual noise but a fundamental mismatch in articulatory representation.

Significance and Claims
The paper claims to provide the first quantitative evidence that pragmatic adaptation in sign language follows measurable articulatory principles comparable to spoken dialogue. The authors argue that current sign language technologies conceptualize variation incorrectly; they are trained to recognize what sign is produced, but not how or why it varies in interactive contexts.

The study emphasizes that signing is an adaptive, interactional system rather than a sequence of static lexical items. The findings suggest that models trained predominantly on interpreter or isolated data fail to capture the gradient articulatory variability of real-world signing, particularly in specialized domains like STEM where vocabulary is fluid. The authors conclude that developing robust educational technologies and sign language models requires frameworks that account for continuous, discourse-dependent variability and the specific pressures of pragmatic adaptation.

Note: The authors modestly frame these findings as a case study due to the small sample size (one instructor-student pair) and the specific nature of the STEM domain, cautioning against broad generalization without further research across diverse signers and discourse contexts.