💻 computer science

UGotMe: An Embodied System for Affective Human-Robot Interaction

This paper introduces UGotMe, an embodied affective human-robot interaction system designed for multiparty conversations that overcomes environmental noise and real-time latency challenges through active face extraction and efficient data transmission, validated on the humanoid robot Ameca.

Original authors: Peizhen Li, Longbing Cao, Xiao-Ming Wu, Xiaohan Yu, Runze Yang

Published 2026-03-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Peizhen Li, Longbing Cao, Xiao-Ming Wu, Xiaohan Yu, Runze Yang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are at a busy, noisy party. You are trying to have a deep conversation with a friend, but there are other people shouting, dancing, and making funny faces right next to you. Now, imagine a robot is standing in the middle of this party, trying to understand only your friend's feelings so it can react appropriately.

This is the exact problem the paper "UGotMe" tries to solve.

Here is a simple breakdown of what the researchers did, using everyday analogies:

1. The Problem: The Robot's "Confused Brain"

Current robots are like people who are bad at filtering out noise. If a robot looks at a group of people, it sees everyone's face.

The Scenario: You (the active speaker) are telling a sad story. But standing right next to you is a stranger laughing, and behind you is a shelf with a weird vase.
The Robot's Mistake: Without help, the robot looks at the whole picture. It sees the laughing stranger and the weird vase. It gets confused and might think, "Oh, everyone is happy!" or "What is that vase?" It fails to focus on your sad face.
The Speed Issue: Also, robots are often slow. By the time the robot figures out what you said, the conversation has already moved on. It's like trying to have a conversation with someone who takes 10 seconds to reply to every word.

2. The Solution: UGotMe (The "Smart Party Host")

The researchers built a system called UGotMe (short for "You Got Me," implying the robot finally understands you). It acts like a super-smart party host who knows exactly how to handle the chaos.

It solves the two main problems with two clever tricks:

Trick A: The "Spotlight" Strategy (Denoising)

Instead of looking at the whole messy room, the robot puts a spotlight on the person talking to it.

How it works: The robot listens to where the voice is coming from. It physically turns its head (like a human turning to face a speaker) so that the person talking is dead center in its camera view.
The Filter: It then digitally "cuts out" just that person's face and ignores the laughing stranger and the vase in the background. It's like using a photo editor to crop out the background so you only see the person you care about.
The "Neutral" Reset: Sometimes, a person's face looks different depending on their mood or lighting. The system takes a "neutral" snapshot of that person's face first (like a baseline photo) and then measures how much their expression changes from that baseline. This helps the robot understand the emotion rather than just the person's unique face shape.

Trick B: The "Express Lane" (Real-Time Speed)

To make the robot fast, they didn't send the video data in a slow, clunky way.

The Analogy: Imagine sending a letter vs. sending a live video stream. Most systems send the whole letter and wait for a reply. UGotMe sends a continuous, high-speed stream of data (like a live video feed) to a powerful computer nearby.
The Result: The robot processes the data instantly, so it can react in real-time, just like a human would in a conversation.

3. The Brain: VL2E (The "Emotion Detective")

The system uses a special AI model called VL2E (Vision-Language to Emotion).

What it does: It combines what the robot sees (the cropped face) with what the robot hears (the words being spoken).
The Magic: It doesn't just look at the face; it remembers the last few sentences of the conversation. If you say, "I'm fine," but your face looks sad and you were just crying, the robot understands you are actually sad, not fine. It acts like a good friend who pays attention to context.

4. The Test: The Robot "Ameca"

The researchers tested this on a real, physical robot named Ameca (a very expressive humanoid robot).

The Experiment: They had real humans talk to Ameca in a group setting.
The Result:
- Without UGotMe: The robot got confused by the background people and gave the wrong emotional reaction (e.g., smiling when the human was sad).
- With UGotMe: The robot correctly identified the speaker's emotion 77% of the time (a huge jump from previous methods) and made people feel like they were having a natural, empathetic conversation.

The Bottom Line

Think of UGotMe as giving a robot "social skills." It teaches the robot how to:

Tune out the noise (ignore the background party).
Focus on the speaker (turn its head and zoom in).
Listen to the context (remember what was just said).
React instantly (don't be slow).

The goal is to make robots that don't just act like humans, but actually feel like they understand us, making our interactions with them much more natural and less robotic.

1. Problem Statement

The paper addresses the challenges of deploying vision-aware multimodal emotion recognition models for affective human-robot interaction (HRI) in real-world, multiparty conversation scenarios. Current models face two primary "embodiment challenges":

Environmental Noise in Visual Inputs: In multiparty settings, a robot's camera captures not only the active speaker but also distracting objects (e.g., furniture) and inactive speakers or irrelevant people. These individuals often display facial expressions different from the active speaker, confusing the model and leading to incorrect emotion recognition.
Real-Time Response Requirements: Interactive systems require low latency. However, large deep learning models, data transmission bottlenecks between the robot and server, and hardware limitations often prevent real-time inference, making fluid conversation difficult.

2. Methodology: The UGotMe System

The authors propose UGotMe, an embodied system designed specifically for multiparty HRI. The system pipeline consists of three phases: On-robot multimodal perception, On-edge vision-language to emotion modeling, and Robotic facial expression execution.

A. Denoising Strategies (Addressing Environmental Noise)

To filter out visual noise, UGotMe employs two specific strategies before feeding data to the emotion model:

Face Extraction: Instead of processing raw RGB frames, the system extracts face images of speakers from the robot's onboard camera. This removes distracting background objects.
Customized Active Face Extraction: To handle inactive speakers, the system utilizes audio-visual alignment:
- The robot adjusts its head pose and left-eye camera direction based on the direction of sound arrival to center the active speaker.
- It extracts the face centered along the x-axis of the image as the active speaker.
- Person-Specific Neutral Normalization: To account for individual differences in facial structure, the system subtracts a "neutral" face image (either manually selected or the first frame of the sequence) from the current frame, feeding the resulting "delta image" to the vision encoder.

B. Efficient Data Transmission (Addressing Real-Time Constraints)

Streaming: Captured RGB images are streamed in bytes from the robot to a local server continuously via a separate thread using ZMQ over TCP.
Buffering: The server buffers the latest $T$ frames (set to 640 frames, approx. 25 FPS) to ensure models can consume data immediately without waiting for full video uploads.
Text Transmission: Dialogue text is transcribed via Google Cloud Speech-to-Text and transmitted only when a conversational turn is generated.

C. The VL2E Model (Vision-Language to Emotion)

The core emotion recognition model, VL2E, is designed to be compatible with the denoising strategies:

Visual Encoder: Uses InceptionResnetv1 (pretrained on CASIA-WebFace) to process face sequences of active speakers. It employs a Self-Attention Transformer to model intra-modal interactions within visual features.
Text Encoder: Uses SimCSE to process dialogue context. It incorporates Context Modeling by concatenating the current utterance with the previous $k$ turns.
Prompt Learning: A prompt ( $P_t$ ) is appended to the context to help the model distinguish between the target turn and the context (e.g., "for [utterance], [speaker] feels ").
Multimodal Fusion: A Cross-modal Transformer fuses the visual and textual features to predict the emotion.
Expression Execution: Recognized emotions are mapped directly to the robot's (Ameca) predefined set of 7 basic facial expressions (Neutral, Surprise, Fear, Sadness, Joy, Disgust, Anger) following parallel empathy (matching the user's emotion).

3. Key Contributions

UGotMe System: An integrated embodied system that solves environmental noise and real-time latency issues for multiparty HRI.
VL2E Model: A novel Vision-Language to Emotion model that outperforms existing baselines on the MELD dataset and is specifically optimized for the denoising strategies (face extraction and neutral normalization).
Real-World Deployment: Successful deployment on the Ameca humanoid robot, demonstrating practical, real-time emotional exchange capabilities in complex, noisy environments.

4. Experimental Results

A. Dataset Performance (MELD)

Metric: Weighted Average F1 Score on the MELD dataset (multiparty conversation).
Result: VL2E achieved an F1 score of 67.29%, outperforming all baselines (including DialogueRNN, ConGCN, MMGCN, and FacialMMT).
Comparison: It surpassed the previous best (TelME*) by +0.71% and DialogueRNN by +10.26%, despite using only Text and Vision modalities (excluding audio).

B. Ablation Studies

Modality: Removing either text or vision significantly dropped performance.
Denoising: Removing face extraction (using full frames) dropped F1 by 1.34%, proving the necessity of focusing on active faces.
Normalization: Removing person-specific neutral normalization dropped F1 by 0.69%, highlighting the importance of individual differences.

C. Real-World Deployment (Ameca Robot)

Setup: Compared UGotMe-VL2E against UGotMe-TelME (without denoising) and UGotMe-VL2E1 (without active face extraction).
Metrics: Emotion Response Accuracy and User Experience (0-10 scale).
Results:
- Accuracy: UGotMe-VL2E achieved 77.29%, a 26.66% increase over UGotMe-TelME and 18.21% over UGotMe-VL2E1.
- User Experience: Scored 7.89, significantly higher than baselines.
Qualitative: Visual examples showed that without denoising, the robot often misinterpreted the emotion of the active speaker due to the distracting expressions of inactive bystanders.

5. Significance and Future Work

Significance: This work bridges the gap between theoretical multimodal emotion recognition and practical embodied robotics. It demonstrates that specific architectural changes (denoising, streaming) are critical for real-world HRI where environmental noise is unavoidable.
Limitations & Future Work:
- Currently uses parallel empathy (mimicking the user). Future work aims to implement reactive empathy (generating a different, supportive emotion).
- Audio signals are currently not fed into the emotion model (only used for speaker localization). Future iterations plan to integrate acoustic signals for better emotion understanding.
- Neutral face estimation in the wild (without a pre-recorded neutral frame) needs further refinement.