💻 computer science

SocioGesture: Real-Time and Adaptive Social Gesture Perception for Human-Robot Interaction

The paper presents SocioGesture, a real-time, adaptive social gesture perception system for human-robot interaction that utilizes a lightweight dual-stream model trained with occlusion-aware corruption to achieve robust, low-latency recognition on edge devices while continuously expanding its vocabulary through offline adaptation.

Original authors: Wenjin Fu, Li-Fan Wu, Jerin Peter, Chip Huyen, Boyuan Chen, Jan Liphardt

Published 2026-09-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Wenjin Fu, Li-Fan Wu, Jerin Peter, Chip Huyen, Boyuan Chen, Jan Liphardt

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Robots are learning to move through our world, but for them to interact with us naturally, they must do more than follow spoken commands. They need to understand the silent language of human movement: a wave to say hello, a raised palm to say stop, or a phone held to an ear to signal "I am busy." This is the challenge of human-robot interaction, a field dedicated to teaching machines how to read these social cues. The difficulty lies in the messy reality of the real world. Unlike a video watched in a quiet office, a robot sees people from moving angles, often with hands hidden behind bodies or lost in shadows. Furthermore, a robot cannot afford to pause and think for a long time; if it hesitates too long to interpret a gesture, the interaction feels broken and unnatural. The goal is to build a system that is fast enough to keep up with a conversation, smart enough to handle missing pieces of information, and capable of learning from its mistakes without needing a human to reprogram it every time.

Researchers at OpenMind have developed a new system called SocioGesture to solve this problem. It is a real-time perception tool designed specifically for robots that need to recognize social gestures while they are moving and working. The system works by watching a person's skeleton—the map of their joints and bones—rather than trying to analyze their clothing or the background. This approach is chosen because a skeleton remains recognizable even when the lighting changes or the person is wearing different clothes. However, the researchers knew that standard skeleton tracking often fails when a hand is occluded or a camera angle shifts. To fix this, they built a model that pays attention to the confidence of every single joint. If the robot's camera is unsure where a hand is, the system notes that uncertainty rather than guessing blindly. It combines the broad movement of the body with the fine details of the hand, fusing them together in a single, lightning-fast calculation that runs directly on the robot's onboard computer.

The core innovation of SocioGesture is how it handles the inevitable gaps in data. In many previous systems, if a key joint like a wrist was missing, the whole recognition attempt would fail. This new system is trained to expect those gaps. The researchers deliberately "corrupted" their training data by hiding hands and arms in the computer simulations, forcing the model to learn how to recognize a gesture even when parts of the skeleton are invisible. This training happens before the robot ever leaves the lab, so the robot does not need extra computing power to be robust. When the robot is in the field, it makes decisions based on what it sees. If it is very confident about a gesture, it acts immediately. If it is unsure, it does not guess; instead, it saves that moment of interaction for later review. This creates a safety loop where the robot avoids making dangerous mistakes, like approaching someone who is trying to stop it, while still gathering data to get smarter.

The system was tested in a variety of indoor and outdoor settings with real people. The researchers collected a dataset of seven common social gestures, including waving to invite a robot closer, holding up a hand to stop it, or pretending to be on a phone to signal unavailability. They also included a "distractor" gesture, like scratching one's head, to ensure the robot didn't confuse it with a phone call. When tested on people the robot had never seen before, the system correctly identified gestures in nearly all cases, even when the hands were partially hidden. In a specific test where the hands were completely masked out of the data, the system's accuracy jumped from a poor 31 percent to a strong 85 percent, proving that the training method worked. The system also ran fast enough to keep up with a standard video camera, processing a full frame in just 25 milliseconds on a robot-mounted computer, which is fast enough to feel instant to a human observer.

Perhaps the most significant finding is the system's ability to learn after deployment. The researchers set up a process where the robot would flag moments it was unsure about and send those video clips to a more powerful computer in the cloud. That powerful computer would label the gesture, and the robot would use that new information to update its own brain. In a test where the robot was taught three new gestures—giving a thumbs-up, a handshake, and a salute—it successfully learned them without forgetting the original seven. The system maintained its high accuracy on the old gestures while correctly identifying the new ones about two-thirds of the time. This demonstrates a path forward where robots can expand their vocabulary of social cues over time, adapting to new situations without needing to be shut down and retrained from scratch.

The researchers also tested the system on a live robot, a humanoid machine named Unitree G1, interacting with five new people who had never been part of the training data. In 150 trials, the robot correctly recognized the gesture 97 percent of the time. When the robot decided to act, it chose the correct behavior—like approaching or stopping—98 percent of the time. The few times it did not act were not because it made a mistake, but because it was being cautious; it chose to wait rather than risk a wrong move. This conservative approach is exactly what the designers intended for a robot in a public space. The study suggests that by combining a lightweight, fast model with a safety-first strategy and a loop for offline learning, we can build robots that are not only efficient but also socially aware and adaptable to the unpredictable nature of human interaction.

Technical Summary: SocioGesture

Problem Statement

Human-Robot Interaction (HRI) requires robots to interpret not only explicit verbal commands but also implicit social cues, such as invitations, refusals, and signals of unavailability. Misinterpreting these cues can lead to socially inappropriate or unsafe behaviors (e.g., approaching when a "stop" signal is given). However, recognizing these gestures in real-world deployments presents a unique set of challenges distinct from offline video classification:

Noisy Perception: Robots operate under partial occlusion, changing viewpoints, varying lighting, and intermittent keypoint failures, particularly affecting small but critical body parts like hands and arms.
Latency Constraints: Interaction flow is disrupted by delayed responses; the system must operate in real-time on onboard edge devices.
Deployment Open-Endedness: Practical systems must adapt to uncertain or previously unseen gesture styles encountered in the field without compromising safety.
Conflicting Requirements: Models capable of rich video reasoning are often too slow or cloud-dependent for safe, closed-loop control, while lightweight onboard models are often brittle when pose estimators fail.

Existing skeleton-based action recognition methods are typically evaluated on curated, offline benchmarks decoupled from robot control, failing to address the robustness and adaptability needs of a deployable perception-control loop.

Methodology

SocioGesture is a real-time, adaptive social gesture perception system designed to resolve the tension between low-latency onboard inference and robust, adaptive learning. The system operates via two distinct paths: an online path for immediate behavior selection and an offline path for model adaptation.

1. Confidence-Aware Skeleton Encoding

Instead of discarding frames with missing joints, SocioGesture retains a compact 9-channel joint feature for each frame $t$ and joint $j$ :
$x_{t,j} = [p_{t,j}, c_{t,j}, p_{t,j} - p_{t-1,j}, b_{t,j}, \Delta b_{t,j}]$
Where $p$ represents coordinates, $c$ is the confidence score from the pose estimator, $b$ is the bone vector relative to the parent joint, and $\Delta b$ is the bone motion. This representation allows the model to distinguish between a stationary visible hand and an unreliable/missing hand, keeping partially observed windows usable.

2. Lightweight Dual-Stream Body-Hand Model

The architecture employs a dual-stream design to separate coarse body posture ( $X_B$ ) from fine hand articulation ( $X_H$ ).

Encoding: Two lightweight graph-temporal backbones ( $f_B, f_H$ ) encode the streams.
Fusion: The streams exchange information via temporal cross-attention mechanisms.
Efficiency: Unlike multi-stream ensembles that require multiple inference passes, SocioGesture fuses features and classifies in a single forward pass, keeping inference costs low for edge deployment.
Uncertainty Estimation: An uncertainty score $u$ is calculated using entropy, disagreement between body and hand auxiliary heads, and average skeleton confidence. This score does not drive behavior directly but flags segments for offline labeling.

3. Occlusion-Aware Training

To improve robustness without inference-time overhead, the model is trained with structured skeleton corruption. This includes:

Weak Augmentation: Flipping, scaling, temporal shifting, and confidence-proportional noise.
Structured Corruption: Hand masking, arm-hand masking, temporal hand disappearance, and random joint masking.
Loss Function: The training objective combines supervised cross-entropy on clean and corrupted views with a consistency term (ramped over early epochs) to align predictions between clean and corrupted inputs.

4. Uncertainty-Driven Offline Adaptation

The system implements a closed-loop adaptation strategy:

Selection: Interaction segments with high uncertainty scores are saved with paired RGB and skeleton data.
Labeling: These segments are labeled offline by a stronger cloud-based video understanding model (e.g., Gemini 2.5 Flash).
Expansion: The classifier is expanded to include new gesture classes by copying original weights and initializing new ones.
Fine-Tuning: The model is fine-tuned on a mixture of original human-labeled data (rehearsal) and pseudo-labeled deployment data, then redeployed after validation.

Key Contributions

Deployable System: A closed-loop system connecting onboard body-hand pose estimation to conservative robot behaviors, separating immediate action from slower adaptation.
Compact Representation & Model: A confidence-aware skeleton representation and a lightweight dual-stream body-hand fusion model evaluated in a single forward pass for low-latency edge inference.
Robustness via Training: Occlusion-aware training that significantly improves robustness to missing hands and arms without adding inference-time computation.
Adaptive Vocabulary: An uncertainty-driven offline adaptation loop that expands the gesture vocabulary while preserving performance on original classes.

Experimental Results

The system was evaluated on a 7-class social gesture dataset collected in mixed indoor-outdoor HRI scenarios, involving 10 subjects and 7,055 skeleton windows.

Recognition Accuracy: The system achieved 96.6% clean accuracy (LOSO cross-validation). Under structured occlusion, it demonstrated substantial robustness: hand-occlusion accuracy improved from 31.8% (standard model) to 84.9%, and arm-occlusion from 28.3% to 79.5%.
Latency: On a robot-mounted NVIDIA Jetson Thor, the full-stack pipeline (detection, tracking, pose, gesture) runs at 25 ms per frame for a single person (real-time at 30 FPS) and 42 ms for three persons.
Offline Adaptation: When expanding from 7 to 10 classes using pseudo-labeled deployment clips, the model achieved 87.9% overall accuracy on the new 10-class test set. Crucially, accuracy on the original 7 classes remained unchanged at 98.5%, demonstrating that rehearsal preserved the original vocabulary.
Live Deployment: In a live closed-loop test with 5 unseen participants (150 trials), the system correctly recognized gestures in 97.3% of trials. Behavior success (correct action execution) was 79.3%, with the majority of failures being "conservative non-action" (robot waiting) rather than incorrect actions, which is a safe trade-off for greeter robots.

Significance and Claims

The paper claims that SocioGesture demonstrates a practical path toward robust, efficient, and adaptive social perception for interactive robots. Its significance lies in:

Decoupling Safety and Learning: By separating immediate, conservative behavior selection from slower, data-hungry adaptation, the system ensures safety while allowing the robot to learn from real-world interactions.
Robustness to Real-World Noise: The occlusion-aware training proves that models can be made robust to the specific failure modes of robot-mounted pose estimators (missing hands/arms) without sacrificing speed or requiring complex inference-time ensembles.
Scalability: The uncertainty-driven adaptation loop shows that robots can expand their interaction vocabulary post-deployment without retraining from scratch or requiring extensive new human-labeled datasets.

The authors position SocioGesture not as a state-of-the-art benchmark setter for offline action recognition, but as a solution to the specific engineering constraints of deployable HRI: low latency, robustness to sensor failure, and the ability to adapt to open-ended social environments.