Toward Signing Activity Projection in Sign Language Interaction

Technical Summary: Toward Signing Activity Projection in Sign Language Interaction

Problem Statement
Social robots currently lack robust interaction capabilities with users who rely on communication modalities other than speech, such as sign language. A critical gap exists in predictive turn-taking for sign language interactions. While Voice Activity Projection (VAP) has successfully modeled future voice activity to support turn-taking tasks (e.g., determining whether a speaker will hold the floor or shift it to an interlocutor) in spoken dialogue, it remains unclear whether this framework transfers to sign language. Sign language interactions rely on visual, modality-specific cues (hand movements, gaze, facial expressions) rather than audio, and the standard speech-derived definitions of "activity" and turn boundaries may not be directly applicable.

Methodology
This study presents an initial transfer experiment adapting the VAP architecture to dyadic sign language interaction using the Public DGS Corpus.

• Data and Proxy Labels: The authors utilized the dialogue-tagged subset of the Public DGS Corpus, which includes OpenPose-based 2D keypoints and manual annotations of lexical signing segments. Instead of using complex utterance-unit annotations, the study constructed binary "signing activity streams" by taking the union of all annotated lexical sign segments. A frame was labeled active if it fell within any annotated segment.
• Input Features: The model utilized pose-derived visual features extracted for each signer, specifically:
  • Hand keypoints.
  • Eye-region keypoints.
  • Mouth-region keypoints.
    Missing values were replaced with zeros, and data was resampled to 30 fps.
• Model Architecture: The authors adopted the original VAP formulation, replacing the audio encoder with a pose encoder. The architecture consists of:
  • Keypoint Encoders: Modality-specific linear projections mapping raw features to 256-dimensional embeddings.
  • Self-Attention Transformer: A layer modeling each signer's pose stream separately.
  • Cross-Attention Transformer: Three layers performing cross-attention over both signers' encoded streams.
  • Prediction Heads: One head for the 256-class future activity state (predicting the joint state of two signers over four future time bins) and one for per-signer activity detection.
• Tasks: The study formulated two proxy tasks based on the binary activity streams:
  1. SHIFT/HOLD: Predicting whether, after mutual inactivity, the same signer resumes (HOLD) or the turn shifts to the other (SHIFT).
  2. SHIFT-prediction: Predicting whether an ongoing activity is likely to be followed by a turn shift within the next 600 ms.
• Training and Evaluation: Models were trained on 20-second windows using AdamW. Performance was evaluated using balanced accuracy and macro-F1 scores, with results averaged over 10 random seeds.

Key Results
The experiments yielded the following findings regarding modality contributions and task difficulty:

• SHIFT/HOLD Prediction: Models incorporating hand cues consistently outperformed the constant baseline (which always predicts HOLD). Hand-only configurations achieved the highest mean balanced accuracy (0.646) and macro-F1 (0.501). Adding eye or mouth cues to hand cues did not consistently improve performance over hand-only inputs.
• SHIFT-prediction: All learned models outperformed the constant baseline. Interestingly, the eye-only configuration achieved the highest mean scores (balanced accuracy 0.558), though the margin over other configurations was small.
• Modality Fusion: Combining multiple modalities (hands, eyes, mouth) did not consistently yield better results than single-modality inputs, suggesting potential issues with noise in non-manual keypoints or a mismatch between the proxy labels and the granular interactional events.

Significance and Claims
The paper claims to provide initial evidence regarding both the promise and the current limitations of transferring predictive turn-taking models from spoken to sign language interaction.

• Promise: The study demonstrates that predictive turn-taking modeling is feasible for sign language when using pose-derived visual cues, particularly hand movements, for local turn-state estimation (SHIFT/HOLD).
• Limitations: The authors emphasize that the current success is constrained by the use of "proxy" labels derived from lexical sign annotations. These labels align naturally with manual activity but may not capture non-manual behaviors (e.g., gaze, head nods) that are crucial for turn-taking in sign language. Consequently, the difficulty in SHIFT-prediction suggests that the current speech-derived event definitions and binary activity streams are insufficient for capturing the full complexity of sign language turn-taking.
• Conclusion: The authors conclude that while the VAP framework can be partially transferred, effective predictive modeling for sign language interaction requires sign-language-specific event definitions that go beyond speech-derived categories and incorporate richer multimodal representations of interactional boundaries.