🤖 AI

A Framework for Low-Latency, LLM-driven Multimodal Interaction on the Pepper Robot

This paper introduces an open-source Android framework for the Pepper robot that utilizes end-to-end Speech-to-Speech models and agentic Function Calling to overcome the latency and multimodal limitations of traditional cascaded pipelines, enabling low-latency, expressive, and fully integrated embodied interactions.

Original authors: Erich Studerus, Vivienne Jia Zhong, Stephan Vonschallen

Published 2026-03-24

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Erich Studerus, Vivienne Jia Zhong, Stephan Vonschallen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a robot friend named Pepper. Right now, talking to Pepper is a bit like trying to have a conversation with someone through a very slow, broken walkie-talkie. You speak, the robot has to write down what you said, send it to a super-smart computer brain (an AI) to think of an answer, write that answer down, and then read it back to you out loud. By the time the robot replies, you've already forgotten what you were talking about, and the robot sounds like a robot reading a script, not a friend having a chat.

This paper introduces a brand new "brain upgrade" for Pepper that fixes these problems. Think of it as swapping that broken walkie-talkie for a direct, high-speed brain-to-brain connection.

Here is how the new system works, explained with some everyday analogies:

1. The "Instant Translator" (Low Latency & Speech-to-Speech)

The Old Way: Imagine a relay race where the baton is your voice. You run to the first person (who turns your voice into text), they run to the second person (the AI who thinks of a reply), and they run to the third person (who turns the text back into voice). By the time the baton gets back to you, the race is over.

The New Way: The researchers installed a Speech-to-Speech (S2S) engine. Now, the robot hears your voice and immediately hears its own voice back, skipping the "writing it down" step entirely.

The Magic: It's like talking to a human who understands not just what you say, but how you say it. If you sound excited, the robot sounds excited. If you sound sad, the robot sounds sympathetic. It preserves the "music" of your voice (your tone and emotion) instead of just the lyrics.
The Result: The conversation flows instantly, just like a real chat between friends, with no awkward 5-second pauses.

2. The "Action-Oriented Manager" (Function Calling)

The Old Way: Previously, the robot was mostly a "talking head." It could chat, but if you asked it to "look at the ceiling," it might just say, "Okay, I see the ceiling," without actually moving its eyes. It was like a tour guide who describes the museum but never lets you look at the exhibits.

The New Way: The AI is now an Agentic Planner. Think of it as a personal assistant who doesn't just talk to you but does things for you.

The Magic: When you ask, "What's on the ceiling?", the robot doesn't just guess. It has a "toolbelt" of actions. It automatically:
1. Moves its eyes to look up.
2. Takes a picture with its camera.
3. Shows the picture to the AI brain.
4. Tells you exactly what it sees.
The Result: The robot can navigate rooms, play games, tell jokes, and react to you touching its hand, all while you are talking to it.

3. The "Universal Adapter" (Runs on Anything)

The Problem: Pepper robots are old, and the company that made them is shutting down support. It's hard to make new apps for them because they are picky and expensive to test on.

The Solution: The researchers built this system like a universal app (like a video game that runs on both a PlayStation and a PC).

The Magic: You can write the code on your regular Android phone or tablet. If you plug it into a Pepper robot, it uses the robot's cameras and motors. If you run it on your phone, it uses your phone's camera and just "pretends" to move.
The Result: Researchers don't need to buy a $20,000 robot to test their ideas. They can build the robot's "brain" on their lunch break using a cheap tablet, and only plug it into the real robot when they are ready to show it off.

Why Does This Matter?

This framework is like giving the robot a superpower.

Before: The robot was a slow, scripted puppet that could only say what it was told.
After: The robot is a fluid, responsive partner that can hear your emotions, look at what you're looking at, and take action in the real world, all in real-time.

The authors made this "brain" open-source, meaning they are handing the blueprints to the whole world for free. This allows other scientists and developers to skip the hard work of building the foundation and start creating amazing new ways for humans and robots to connect.

In short: They took a robot that was like a dial-up modem and upgraded it to a 5G connection, giving it the ability to not just talk, but to act and feel in the moment.

1. Problem Statement

The paper identifies two critical limitations in current Large Language Model (LLM) implementations for social robotics, specifically on the Pepper robot:

High Latency and Loss of Paralinguistics: Existing systems typically use a cascaded pipeline (Speech-to-Text $\rightarrow$ LLM $\rightarrow$ Text-to-Speech). This sequential processing introduces significant delays (reported as 3.84–8.96 seconds), exceeding the 1–2 second threshold required for natural conversation. Furthermore, converting audio to text and back discards crucial paralinguistic cues (prosody, intonation, emotional tone), resulting in robotic and less empathetic interactions.
Limited Agentic Control and Multimodal Perception: Most implementations fail to fully utilize LLMs as "agentic planners." They lack mechanisms to autonomously orchestrate complex robot actions (navigation, gaze, touch) or integrate real-time multimodal feedback (vision, haptics, system state) to ground the LLM in the physical environment.

2. Methodology

The authors present an open-source, native Android framework designed to overcome these barriers through a modular architecture that decouples development from specific hardware.

A. Architectural Design

Native Android Execution: The framework runs natively on the Pepper robot's onboard tablet (using the QiSDK) or on standard Android smartphones/tablets. This eliminates the need for external bridging computers, simplifying deployment.
Dual Build-Flavor System: The codebase uses a build flavor system to separate core logic from hardware-specific implementations.
- Pepper Flavor: Uses QiSDK for physical movement and robot-specific sensors.
- Standalone Flavor: Simulates robot actions or uses standard device sensors (e.g., front camera) for testing on non-robotic Android devices.

B. Core Interaction Engine: End-to-End Speech-to-Speech (S2S)

Direct Audio Processing: Instead of STT/LLM/TTS cascades, the system integrates end-to-end S2S models (supporting providers like OpenAI, Azure, x.ai, and Google).
Paralinguistic Preservation: By processing audio directly, the model preserves user intonation and generates responses with adaptive tone, pacing, and prosody without complex prompt engineering.
Duplex Communication & Interruption: While the Pepper robot lacks echo cancellation (requiring microphone muting during speech), the system implements a "tap-to-interrupt" feature via the UI status capsule. It also leverages duplex capabilities to process non-auditory inputs (touch, tool results) while the robot is speaking.

C. Agentic Planning via Function Calling

The LLM acts as an autonomous planner using a rich library of function calls (Tools):

Tool Orchestration: The LLM can sequence multiple tools to fulfill complex requests. For example, the query "What do you see at the ceiling?" triggers a sequence: look_at_position (gaze control) $\rightarrow$ analyze_vision (image capture) $\rightarrow$ S2S backend (visual analysis).
Multimodal Integration:
- Vision: Event-driven image capture (on-demand) sent directly to the multimodal S2S model.
- Haptics: Touch sensors on the head, hands, and bumpers send contextual messages to the LLM for socially appropriate responses.
- Human Perception: A local, on-device system (YuNet/SFace) tracks people and recognizes faces via WebSocket, replacing unreliable cloud-based metadata.
- System State: The LLM receives feedback on navigation failures, charging status, and error messages to reason about physical constraints.

D. Configurability

Runtime Switching: Users can switch between S2S providers and audio input modes (Direct Audio vs. STT-transcribed) via an in-app settings panel.
Language Flexibility: The system supports on-the-fly language switching without reconfiguration.

3. Key Contributions

Low-Latency S2S Framework: A novel implementation for the Pepper robot that achieves fluid, real-time dialogue by bypassing text intermediaries, preserving emotional nuance and reducing response times.
Agentic Multimodal Control: A comprehensive Function Calling architecture that elevates the LLM from a chatbot to an embodied agent capable of controlling navigation, gaze, and perception while integrating diverse sensor data.
Hardware-Agnostic Development: A dual-build system allowing researchers to develop and prototype on standard Android devices, decoupling the development cycle from expensive robotic hardware.
Open-Source Artifact: The release of a fully documented, MIT-licensed framework (Kotlin/Android) that serves as a reference implementation for the HRI community.

4. Results and Implementation Details

Performance: The system successfully demonstrates sub-second latency interactions (implied by the shift from 3–8s cascaded systems to streaming S2S), enabling natural conversation flow.
Functionality: The framework supports a wide range of tools including navigation, gaze control, vision analysis, web search, weather, and interactive games (e.g., Tic-Tac-Toe, memory games).
Robustness: It handles asynchronous inputs (touch, vision) during speech output and includes a local face-tracking system to ensure privacy and stability independent of cloud services.
Adaptability: The code is built on modern Android toolchains (API 35, Java 17, Kotlin) and is compatible with the latest Android Studio versions, addressing the obsolescence of the original Pepper SDK.

5. Significance and Impact

Advancing Embodied AI: This work bridges the gap between abstract LLM reasoning and physical robot execution, providing a blueprint for "grounded" interaction where the robot understands and reacts to its physical environment in real-time.
Community Enablement: By open-sourcing the framework and enabling development on standard Android devices, the authors lower the barrier to entry for HRI research, particularly given the uncertainty surrounding official Pepper support following Aldebaran's liquidation.
Future-Proofing: The modular design and abstraction layers allow for easy adaptation to other Android-based humanoid robots and future LLM capabilities, fostering a more extensible ecosystem for social robotics.

Limitations Noted: The framework does not support lip-syncing (due to Pepper's hardware), relies on cloud APIs (introducing potential network latency), and currently lacks lip-sync capabilities relevant to other robots like Furhat. Additionally, the current API key storage method is suitable for development but requires secure handling for production.