Vision-Language System using Open-Source LLMs for Gestures in Medical Interpreter Robots

This paper presents a privacy-preserving vision-language framework for medical interpreter robots that leverages locally deployed open-source LLMs and a novel annotated dataset to accurately detect clinical speech acts and generate human-like, appropriate robotic gestures.

The Gist

Imagine a robot doctor's assistant standing in a busy hospital room. Its job is to help a doctor talk to a patient who speaks a different language. Usually, these robots just translate words, which can feel a bit robotic and cold. But what if the robot could also nod, point, or hold up a hand at the exact right moment, just like a human would? That's the goal of this paper.

The researchers built a "smart body" for a medical robot that understands not just what is being said, but how it should be said physically. Here is how they did it, explained simply:

1. The Problem: The "Silent" Translator

In a hospital, words aren't enough. If a doctor says, "I need your permission to proceed," they might also hold out their hand or lean forward to show they are asking for consent. If they say, "Take a deep breath," they might demonstrate the motion.
Current translation tools are like a radio: they only broadcast the voice. They miss the body language. The researchers wanted a robot that could "speak" with its hands and arms, too, to make the patient feel safer and understood.

2. The Solution: A "Privacy-First" Brain

To make the robot smart enough to know when to gesture, they needed a brain (an AI) that could listen to the conversation and decide: "Is the doctor asking for permission? Is the doctor giving an instruction? Or is this just small talk?"

• The Local Brain: Most AI models live in the "cloud" (huge servers far away). But in a hospital, patient privacy is everything. You don't want sensitive medical data flying over the internet. So, the team built a system that runs entirely on the robot's own computer. It's like having a personal librarian in the room who never leaves the building and never tells anyone what you read.
• The "Few-Shot" Trick: They taught this local AI using a clever method called "few-shot prompting." Imagine you are teaching a child to recognize a cat. Instead of showing them a million pictures, you show them three examples: "This is a cat. This is a cat. This is a cat. Now, is this a cat?" The AI learned to spot "Consent" and "Instruction" sentences just by seeing a handful of examples.

3. The Dataset: The "Gestural Dictionary"

To train the robot, the team needed a dictionary of medical conversations paired with gestures.

• They took 58 real videos of doctors talking to patients (from a public YouTube channel).
• They broke these videos into thousands of tiny clips.
• They labeled each clip: "This sentence is a request for consent," or "This is an instruction to move an arm."
• They even recorded the doctor's hand movements in these clips so the robot could learn to copy them.

4. The Two-Mode System

The robot has two ways of moving, depending on what the AI hears:

• Mode A: The Mirror (Human-Mimic)
  If the AI hears a Consent or Instruction sentence, the robot switches to "Mirror Mode." It looks at the video of the human speaker, grabs their hand movements, and copies them exactly.

  • Analogy: It's like a dance partner who perfectly mirrors your steps. If the doctor raises a hand to say "Stop," the robot raises its hand too. This feels very natural and human.

• Mode B: The Improviser (Speech-Gesture Generation)
  If the AI hears normal conversation (not a specific instruction or consent), the robot uses a different AI to invent a gesture that fits the mood.

  • Analogy: This is like a jazz musician. If the conversation is casual, the robot adds a little nod or a wave to keep the rhythm going, even if no one told it exactly what to do.

5. The Results: Does It Work?

They tested this on a Pepper robot (a friendly, humanoid robot often used in research) with 26 human volunteers.

• The "Human" Test: People watched videos of the robot moving. When the robot was copying real human gestures (Mode A), people rated it as more human-like than when it was just generating random gestures. It felt less like a machine and more like a person.
• The "Appropriateness" Test: People also checked if the gestures matched the words. The robot did just as well as the best existing systems at making sure the gestures made sense.
• The Privacy Win: Because everything runs locally on the robot, it uses very little memory and keeps all patient data safe inside the machine.

The Big Picture

Think of this system as giving the robot a soulful body. Before, medical robots were like telephones—good for hearing, bad for feeling. This new system allows the robot to use its "body language" to build trust, reduce anxiety, and make sure patients truly understand their care, all while keeping their private medical secrets safe in a locked box on the robot itself.

It's a step toward robots that don't just translate words, but translate human connection.

Technical Summary

Here is a detailed technical summary of the paper "Vision-Language System using Open-Source LLMs for Consent and Instruction Gestures in Medical Interpreter Robots."

1. Problem Statement

Effective communication in healthcare is often hindered by language barriers, necessitating the use of interpreters. While non-verbal cues (gestures) are critical for conveying consent and medical instructions, current technological solutions (e.g., video services, translation software) fail to replicate these non-verbal elements. Furthermore, existing robotic gesture generation systems face three major hurdles:

• Data Scarcity: A lack of domain-specific datasets containing clinical conversations paired with gesture annotations.
• Precision & Context: General gesture generation models fail to capture the specific precision required for medical instructions or account for cultural/individual variability.
• Computational Constraints: High-stakes clinical environments require strict data privacy (no cloud processing) and real-time performance, which limits the deployment of large, complex models on resource-constrained robotic platforms.

2. Methodology

The authors propose a privacy-preserving, on-device vision-language framework designed for the Pepper robot. The system operates entirely locally using open-source models to ensure data security.

A. System Architecture

The framework consists of three primary modules:

1. Gesture Sentence Detection (GSD): An on-device Large Language Model (LLM) analyzes transcribed speech to classify utterances into three categories: Consent, Instruction, or Neither.
2. Human-Mimic Module: If the GSD classifies the input as "Consent" or "Instruction," this module captures the user's video, estimates their pose, and maps the human kinematics directly to the robot's joint angles to reproduce the specific gesture.
3. Speech-Gesture Generation (Baseline): If the input is "Neither," the system defaults to a semantic-aware gesture generator (Semantic Gesticulator) to produce generic, context-appropriate gestures.

B. Dataset Creation

To address the data scarcity issue, the authors created a novel clinical conversation dataset:

• Source: 58 clinical training videos from the "Dr. James Gill" YouTube channel.
• Processing: Videos were transcribed using OpenAI Whisper. A custom script reconstructed split utterances and aligned them with video timestamps, resulting in 3,736 complete sentences.
• Annotation: Sentences were classified using a consensus of three LLMs (gpt-oss:20b, qwen3:30b, deepseek-r1:8b) and validated by human annotators. The final dataset contains 117 Consent, 912 Instruction, and 2,707 Neither samples, annotated with sentence-level gesture labels.

C. Technical Implementation

• LLM Selection: The GSD module utilizes a lightweight, edge-optimized LLM with few-shot prompting (11-shot strategy). The system prioritizes models that balance accuracy with low memory footprint (e.g., qwen3:8b).
• Pose Estimation: The Human-Mimic module uses MediaPipe Pose Landmarker for CPU-optimized, low-jitter skeletal tracking.
• Kinematic Mapping: Human joint coordinates are mapped to the Pepper robot's 12 joint angles. A scaling factor of 12 is applied to compensate for dataset characteristics and robot speed limits.
• Safety & Optimization: For the baseline generator, motion data (BVH format) is down-sampled by a factor of 12 to ensure joint velocities remain within the robot's safety thresholds.

3. Key Contributions

1. Novel Dataset: Introduction of a sentence-level annotated dataset of clinical conversations paired with gesture clips, specifically for medical interpreter scenarios.
2. Privacy-Preserving Framework: A fully local, lightweight LLM-based pipeline that detects specific speech acts (Consent/Instruction) without sending data to the cloud.
3. Human-Mimic Pipeline: A method to translate real-time human pose kinematics directly into robot motor commands, enabling the robot to mimic specific user gestures for consent and instruction.
4. Comprehensive Evaluation: A user study benchmarking the system against a state-of-the-art speech-gesture baseline (Semantic Gesticulator) on the Pepper robot.

4. Results

A. Gesture Sentence Detection (GSD)

The authors evaluated nine lightweight LLMs. Qwen3-8B emerged as the optimal model:

• Accuracy: 0.90
• Weighted Precision: 0.93
• Weighted F1-Score: 0.91
• Memory Usage: 7.2 GB
• Note: Smaller models (e.g., Gemma 270m) suffered from hallucinations and low accuracy, while larger models offered diminishing returns relative to their resource cost.

B. User Study (Human-Likeness & Appropriateness)

A within-subject study with 26 participants compared the proposed "Human-Mimic" approach against the "Semantic Gesticulator" (SG) baseline.

• Human-Likeness: The proposed approach scored significantly higher (5.78/10) compared to SG (5.24/10) with a p-value of 0.019. Participants perceived the mimicked gestures as more natural.
• Appropriateness: Both approaches scored similarly (5.20 vs. 4.76), with no statistically significant difference (p=0.277), indicating the mimicked gestures were just as contextually appropriate as the generated ones.
• Efficiency: The proposed approach required only 3 MB of GPU RAM for the gesture generation phase, whereas the SG baseline required 2260 MB, highlighting a massive reduction in computational overhead.

5. Significance

This work bridges the gap between advanced AI capabilities and the strict reliability/privacy requirements of clinical environments. By demonstrating that open-source, locally deployed LLMs can effectively detect critical speech acts and drive human-like robotic gestures, the paper offers a viable path for:

• Enhanced Patient Autonomy: Reinforcing consent through synchronized verbal and non-verbal communication.
• Reduced Cognitive Load: Using instructional gestures to clarify complex medical concepts.
• Secure Deployment: Providing a framework that operates entirely on-device, ensuring patient data never leaves the medical facility.

The system successfully proves that high-fidelity, context-aware human-robot interaction in healthcare is achievable without compromising data privacy or relying on massive cloud infrastructure.