🤖 AI

SignVLA: Real-Time Sign Language-Guided Robotic Manipulation via Attention LSTM and Vision-Language-Action Models

SignVLA is a real-time framework that enhances accessibility in human-robot interaction by converting sign language gestures into semantic instructions via an attention-enhanced LSTM and temporal stabilization module, enabling vision-language-action models to execute robotic manipulation tasks for deaf and speech-impaired users.

Original authors: Ningwei Bai, Xinyu Tan, Harry Gardner, Zhengyang Zhong, Liuhaichen Yang, Luoyu Zhang, Zhekai Duan, Monkgogi Galeitsiwe, Zezhi Tang

Published 2026-06-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ningwei Bai, Xinyu Tan, Harry Gardner, Zhengyang Zhong, Liuhaichen Yang, Luoyu Zhang, Zhekai Duan, Monkgogi Galeitsiwe, Zezhi Tang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a robot that usually only understands you if you speak to it or type a command on a keyboard. Now, imagine that same robot could understand you if you "spoke" with your hands using sign language. That is exactly what the SignVLA system does.

Here is a simple breakdown of how it works, using everyday analogies:

The Problem: A Robot with "Hearing" Only

Most advanced robots today use Vision-Language-Action (VLA) models. Think of these models as a robot's brain that looks at a scene, reads a text instruction (like "pick up the cup"), and then moves its arm to do it.

However, these robots usually only "listen" to spoken words or typed text. This creates a barrier for people who are deaf, hard of hearing, or cannot speak. It's like having a very smart librarian who can only take requests if you whisper them or write them on a slip of paper, but refuses to understand if you wave your hands to ask for a book.

The Solution: SignVLA (The "Hand-to-Brain" Translator)

The researchers built a system called SignVLA that acts as a universal translator between hand gestures and robot commands. It doesn't change the robot's brain; it just adds a new way to talk to it.

The system works in three main steps, like a relay race:

1. The Eyes (Seeing the Hands)
First, the system watches a video of a person signing. Instead of trying to understand the whole picture, it uses a tool called MediaPipe to focus strictly on the "skeleton" of the hands. It tracks exactly where the fingers, knuckles, and wrists are moving, frame by frame.

Analogy: Imagine a security camera that ignores the background and only draws a glowing stick-figure outline of your hands to track their movement.

2. The Brain (Understanding the Gesture)
Next, the system feeds those hand movements into a special computer model called an Attention LSTM.

LSTM: Think of this as a memory bank that remembers what your hands did a second ago, so it understands that a movement is a sequence (like a sentence), not just a single frozen pose.
Attention: This is like a spotlight. When the model looks at the sequence of hand movements, it knows to focus on the most important parts of the gesture (the "key words") and ignore the wiggly, less important parts.
The Output: This part turns the hand movements into a list of "glosses" (simple sign language words like "APPLE," "PICK," or "BASKET").

3. The Translator (Making it Natural)
The list of sign words is then passed to a Large Language Model (LLM), which acts like a human translator. It takes the list (e.g., "PICK BUTTER BASKET") and turns it into a full, natural sentence the robot already knows how to follow: "Pick up the butter and place it in the basket."

The Safety Net: The "Stability Buffer"

One big problem with sign language is that hands can jitter or move quickly, causing the computer to get confused for a split second. If the robot reacted to every tiny glitch, it would start shaking or changing its mind constantly.

To fix this, SignVLA uses a Temporal Stability Buffer.

Analogy: Imagine a bouncer at a club. If someone shouts a command once, the bouncer might ignore it to make sure they aren't just coughing. But if they shout the same command three times in a row, the bouncer lets them in.
The system waits to see if the same sign command appears consistently for a few frames before it sends the order to the robot. This stops the robot from getting "jittery."

The Results: Does it Work?

The researchers tested this system in a computer simulation using a robotic arm (a Franka Emika Panda).

Recognition: The system was very good at recognizing 33 specific sign words (like "Apple," "Bottle," "Pick," "Place"). It got about 89% of them right on the first try, which is a huge improvement over older methods that didn't use the "attention" spotlight.
Robot Action: When the robot received these translated instructions, it successfully completed tasks like picking up objects and putting them in baskets about 95% to 98% of the time.

The Bottom Line

The paper shows that you don't need to rebuild a robot's entire brain to let it understand sign language. You just need a lightweight, real-time "translator" that watches the hands, remembers the sequence, stabilizes the signal, and hands a clear English sentence to the robot.

The team has also built the physical hardware (a real robot arm) and is preparing to test this in the real world, moving from computer simulations to actual physical robots.

Technical Summary: SignVLA

Problem Statement

Vision-Language-Action (VLA) models have emerged as a powerful paradigm for generalist robotic manipulation, enabling robots to ground semantic goals in physical scenes using natural language instructions. However, current VLA interfaces predominantly rely on speech or text inputs. This reliance creates a significant accessibility barrier for deaf, hard-of-hearing, and speech-impaired users. Furthermore, in environments where speech is unreliable, noisy, or socially inappropriate, these systems face deployment limitations.

Integrating sign language as an input modality presents unique technical challenges. Unlike discrete text, sign language consists of continuous spatio-temporal visual signals involving hand shape, articulation, position, and motion trajectories. This creates a modality mismatch between visual sign streams and the discrete token-based interfaces expected by VLA policies. Additionally, robotic execution is highly sensitive to recognition instability; transient frame-level errors in sign recognition can lead to command flickering, incorrect task selection, or unsafe downstream actions. Existing sign language recognition (SLR) systems often rely on large-scale annotated datasets and computationally intensive backbones (e.g., Transformers), making real-time deployment on resource-constrained robotic platforms difficult.

Methodology

The paper proposes SignVLA, a real-time framework that bridges visual sign gestures and robotic manipulation through a modular, lightweight pipeline. The system architecture consists of three primary stages:

1. Sign Perception and Recognition

The front-end module processes live video streams to extract semantic instructions:

Feature Extraction: Instead of processing raw RGB frames, the system uses MediaPipe to extract 544-dimensional hand landmark features in real-time. These features include absolute and wrist-relative 3D coordinates, finger direction vectors, and frame-to-frame velocity differences.
Temporal Modeling: The landmark sequences are fed into an Attention-Enhanced Long Short-Term Memory (LSTM) network.
- The LSTM models temporal dependencies across the signing sequence.
- An additive attention mechanism is incorporated to assign higher weights to discriminative temporal segments, improving classification accuracy by focusing on key motion patterns.
Training Strategy: The model is trained on a filtered subset of the Microsoft ASL Citizen dataset, comprising 33 sign vocabularies relevant to robotic manipulation (e.g., object names like "apple," "butter," and actions like "pick," "place"). The training employs a weighted cross-entropy loss with label smoothing to address class imbalance and improve generalization.

2. Temporal Stabilization

To mitigate the instability inherent in real-time video processing, a Temporal Stability Buffer is implemented:

Predicted sign labels are accumulated over a temporal window.
A command is only committed if the same class is the argmax for 3 consecutive frames and an 8-frame cooldown has elapsed.
This mechanism prevents repetition artifacts and ensures that the downstream VLA policy receives stable, consistent instructions rather than noisy, flickering predictions.

3. Language Conversion and Robotic Execution

Sign-to-Text Interface: The confirmed gloss sequence is mapped to natural-language instructions using predefined templates (e.g., converting "PICK" + "BUTTER" + "PLACE" + "BASKET" into "pick up the butter and place it in the basket").
VLA Policy: The generated natural language instruction serves as the language condition for the downstream GR00T N1 policy (a VLA model utilizing a Diffusion Transformer).
Execution: The policy predicts robot actions ( $a_t$ ) conditioned on the current visual observation ( $o_t$ ) and the confirmed sign-derived instruction ( $l$ ): $a_t = \pi_{GR00T}(o_t, l)$ . The instruction remains constant until a new sign command is explicitly confirmed.

Key Contributions

SignVLA Framework: The first system to directly connect sign language recognition to a VLA backend for physical robot manipulation, enabling accessible human-robot interaction without modifying the VLA backbone.
Lightweight Real-Time Recognition: A novel Attention-LSTM architecture that processes MediaPipe landmark features, achieving high accuracy with low latency suitable for resource-constrained platforms.
Temporal Stabilization Module: A practical filtering mechanism that converts noisy visual observations into stable semantic commands, addressing the critical issue of command flickering in real-time robotic control.
Integrated Evaluation: Experimental validation demonstrating that sign-derived instructions can effectively drive robotic manipulation tasks in simulation.

Experimental Results

The system was evaluated on the LIBERO benchmark suite, which covers object-to-basket, object-to-container, and scene interaction tasks.

Sign Recognition Performance:
- On a 33-sign vocabulary aligned with manipulation tasks, the proposed Attention-LSTM achieved 88.89% Top-1 accuracy and 96.53% Top-5 accuracy, significantly outperforming a vanilla LSTM baseline (34.72% Top-1).
- The model achieved perfect recognition (100%) for 13 specific signs, including critical objects (e.g., "ball," "bowl," "butter," "cheese") and actions (e.g., "pick," "turn," "up").
- Errors were sparse and concentrated among visually or temporally similar signs (e.g., "bottle" vs. "box" vs. "bowl"), indicating the model's robustness to fine-grained gesture ambiguity.
Robotic Manipulation Success:
- When conditioned on sign-derived instructions, the pre-trained GR00T policy achieved high success rates across LIBERO task suites:
  - LIBERO-Spatial: 97.50% (195/200)
  - LIBERO-Goal: 97.50% (195/200)
  - LIBERO-Object: 98.50% (197/200)
  - LIBERO-10 (Long-horizon): 94.50% (189/200)
- The results suggest that object-centric manipulation tasks are highly compatible with the task-constrained sign vocabulary used.

Significance and Claims

The paper posits that SignVLA demonstrates that lightweight temporal sign recognition can serve as an effective and practical accessibility layer for multimodal embodied intelligence. By decoupling the sign recognition front-end from the VLA policy, the system enables deaf and hard-of-hearing users to control robots via sign language without requiring the VLA model to directly interpret complex video streams.

The authors emphasize that the system successfully bridges the gap between continuous visual gestures and discrete robotic instructions. While the current work focuses on a constrained vocabulary and simulation-based evaluation, the successful integration of a physical Franka Emika Panda robot setup indicates a clear path toward real-world deployment. The paper concludes that task-constrained sign vocabularies provide a viable interface for embodied AI, with future work directed toward expanding to large-vocabulary continuous sign understanding and full real-world physical deployment.