💻 computer science

StretchBot: A Neuro-Symbolic Framework for Adaptive Guidance with Assistive Robots

This paper introduces StretchBot, a neuro-symbolic robotic framework that combines multimodal perception with knowledge-graph-grounded LLM reasoning to provide adaptive guidance for stretching routines, demonstrating through a pilot study that such adaptive systems improve perceived contextual relevance while maintaining the predictability of scripted approaches.

Original authors: Luca Vogelgesang, Ahmed Mehdi Soltani, Mohammadhossein Khojasteh, Xinrui Zu, Stefano De Giorgis, Madalina Croitoru, Filip Ilievski

Published 2026-04-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Luca Vogelgesang, Ahmed Mehdi Soltani, Mohammadhossein Khojasteh, Xinrui Zu, Stefano De Giorgis, Madalina Croitoru, Filip Ilievski

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a personal trainer who is incredibly smart but also a bit rigid. This trainer knows the perfect workout routine, but if you start to look tired or if there's a glass of water on the table, they might keep pushing you to do more reps because their "script" says so. They can't really see or feel your situation.

Now, imagine a different trainer. This one is a hybrid genius: part human-like conversationalist who understands your feelings, and part strict librarian who knows exactly what exercises are safe and how they connect. This is StretchBot.

Here is a simple breakdown of how this robot works, using everyday analogies:

1. The Core Idea: The "Brain" vs. The "Rulebook"

Most robots today are like GPS navigation systems that only know the pre-planned route. If you take a wrong turn, they just say, "Recalculating," but they don't really understand why you took the turn or if you're lost because of traffic.

StretchBot is different. It uses a Neuro-Symbolic approach. Think of this as a team of two:

The "Big Brain" (The LLM): This is like a very chatty, empathetic friend. It can understand if you sound tired, if you're in pain, or if you're joking. It's flexible and creative.
The "Rulebook" (The Knowledge Graph): This is like a strict coach's playbook or a library of facts. It knows that "Bananas = Energy" and "Pain = Stop." It keeps the Big Brain from making up crazy things (like telling you to jump off a chair).

The Magic: The robot asks the "Big Brain" for ideas, but the "Rulebook" checks them first. If the Big Brain says, "Hey, you look thirsty, let's go get a drink!" the Rulebook confirms, "Yes, there is a water bottle nearby, and that is a safe suggestion."

2. How It Sees You (The Senses)

StretchBot doesn't just watch you; it perceives you through three senses, like a detective gathering clues:

Eyes (Pose Monitoring): It uses a camera to track your body, like a dance instructor watching your form. It knows if your arms are high enough or if you're touching your toes correctly.
Ears & Face (Emotion Recognition): It listens to your voice (is it shaky?) and looks at your face (are you frowning?). It combines these clues to guess if you are happy, tired, or frustrated.
Eyes (Object Detection): It scans the room. Is there a chair? A banana? A towel? It knows what these things are for.

3. The Pilot Test: Scripted vs. Adaptive

The researchers tested StretchBot with three people. They compared two modes:

Mode A (The Scripted Robot): Like a music player on shuffle. It plays the songs in a fixed order. It's smooth, predictable, and easy to follow.
Mode B (The Adaptive Robot): Like a DJ who reads the crowd. If you look tired, it slows down. If you look bored, it changes the vibe.

What happened?

The Good News: The Adaptive Robot felt much more relevant. When the robot noticed a water bottle and the user looked tired, it pointed at the bottle and said, "Want a sip?" The users loved this "thoughtfulness."
The Trade-off: The Scripted Robot felt smoother. Because the Adaptive Robot was thinking and reacting, it sometimes paused a bit longer or changed the plan unexpectedly. Some users found this a little distracting, like a conversation that jumps topics too quickly.

4. The "Safety Net" (The Verifier)

Before the robot speaks or moves, it runs its ideas through a Safety Net (a verifier module).

Analogy: Imagine you are writing a text message to a boss. You type a draft, but before you hit send, you have a "grammar check" that also checks, "Is this tone appropriate?"
This ensures the robot doesn't accidentally say something rude or weird, even if its "Big Brain" got a little carried away.

5. The Big Takeaway

The study found that predictability and adaptability are a bit of a tug-of-war.

If you want a robot that is a reliable, no-surprise coach, a scripted approach is great.
If you want a robot that feels like it truly cares about your specific needs and the environment around you, an adaptive approach is better, even if it's a little less smooth.

In short: StretchBot is a step toward robots that aren't just following a script, but are actually paying attention. They are learning to be the kind of coach who notices you're sweating and suggests a break, rather than just yelling "One more rep!" until the timer runs out.

The researchers admit this was just a small test (like a pilot flight), but it proves that mixing "human-like chat" with "strict rulebooks" is a promising way to build robots that help us stay healthy without losing our minds.

1. Problem Statement

Assistive robots in healthcare and home settings often rely on rigid, scripted routines. While these systems offer predictability, they lack the ability to adapt to a user's real-time physical state (e.g., fatigue, pain), emotional condition, or environmental context. Conversely, purely reactive systems driven by Large Language Models (LLMs) can adapt but often suffer from incoherence, hallucinations, and a lack of safety constraints. The core challenge is balancing structured predictability (ensuring a complete, safe exercise routine) with contextual responsiveness (adapting to user needs in real-time).

2. Methodology: The StretchBot Framework

The authors propose StretchBot, a hybrid neuro-symbolic robotic coach designed to guide users through stretching routines. The system integrates multimodal perception with a reasoning engine that couples a Knowledge Graph (KG) with a Large Language Model (LLM).

System Architecture

The framework operates in a continuous loop consisting of four stages:

Perception Module:
- Object Detection: Uses YOLOv8n to identify relevant environmental objects (e.g., water bottles, chairs, bananas) to ground suggestions in reality.
- Pose Monitoring: Utilizes MediaPipe Pose to track skeletal landmarks. It employs rule-based geometric criteria to verify exercise completion (e.g., checking wrist-to-nose distance for arm raises or calculating trunk inclination angles for lateral leans).
- Emotion Recognition: Fuses three modalities—voice (acoustic features), facial expressions, and text (sentiment analysis)—using a late-fusion weighted average to estimate the user's affective state.
- Speech-to-Text: Converts user dialogue into text for intent interpretation.
Reasoning Module (Neuro-Symbolic Core):
- Context Integration: Aggregates perception data (objects, pose status, emotion) into a context package.
- Knowledge Graph (KG) Retrieval: A manually constructed, task-specific KG encodes entities (e.g., Fatigue, Banana, ToeTouch) and relations (e.g., Fatigue $\to$ suggests $\to$ DrinkWater). This provides explicit, interpretable grounding.
- LLM Reasoning: The LLM receives the context enriched with KG relations. It performs high-level reasoning to decide on the next action.
- Constraint & Verification: The LLM is constrained to output specific action prefixes (NEXT EXERCISE, POINT <OBJECT>, STOP ROUTINE) followed by natural language. A secondary Verifier Module checks the output for formatting, tone, and coherence before execution.
Action Execution Module:
- Translates symbolic commands into robot actions.
- Motion Control: Converts POINT <OBJECT> into Cartesian target poses for the robotic arm, generating collision-free trajectories.
- Speech Synthesis: Converts the LLM's natural language output into spoken instructions via Text-to-Speech (TTS), synchronized with motion.
Planning & Adaptation Module:
- Maintains the state of the routine. It ensures the session follows a global script (baseline plan) while allowing local branching (adaptations) based on the reasoning module's decisions (e.g., inserting a break if fatigue is detected).

Implementation Details

Software: Built on ROS2. Uses OpenRouter for LLM access (specifically deepseek-r1-0528-qwen3-8b) and a lightweight JSON-based internal KG.
Hardware: A low-cost robotic arm setup with a standard camera and PC, demonstrating feasibility without specialized high-end compute.

3. Key Contributions

Hybrid Neuro-Symbolic Architecture: A novel framework combining multimodal perception with a KG-grounded LLM. This approach leverages the explainability of symbolic knowledge (KG) for safety and grounding, while utilizing the flexibility of LLMs for natural interaction and context adaptation.
Action-Oriented Knowledge Representation: A methodology for integrating KG reasoning directly into LLM prompts. The system retrieves specific relations (e.g., Sweating $\to$ DrySweat) to ground the robot's decisions in both perceived user states and environmental affordances.
Collaborative Interaction Protocol: A design that balances proactive adaptation with human oversight by requiring explicit user confirmation for major routine changes, preventing the robot from becoming too intrusive.
Exploratory Pilot Study: A comparative evaluation between scripted and adaptive guidance, providing early empirical data on user perceptions of adaptability, trust, and naturalness.

4. Results (Pilot Evaluation)

The authors conducted a pilot study with three participants, comparing a Scripted Condition (fixed routine) against the Adaptive Condition (dynamic adjustments).

Quantitative Findings:
- Adaptability: The adaptive condition scored significantly higher ( $M=3.67$ vs. $1.67$) in perceived adaptability and object-pointing relevance ($4.67$ vs. $1.33$).
- Smoothness & Trust: The scripted condition scored slightly higher in comfort, trust, and naturalness. Users found the scripted routine easier to follow and less cognitively demanding.
- Clarity: Both conditions were rated equally high for clarity.
Qualitative Findings:
- Participants preferred the scripted version for its predictability and "relaxing" flow.
- The adaptive version was praised for being "thoughtful" and "attentive" but was sometimes perceived as introducing extra cognitive load or interrupting the flow.
- Latency: The adaptive system occasionally suffered from 3–10 second delays due to LLM inference and verification steps, which impacted the perceived naturalness.
Technical Observations: The internal KG successfully covered common scenarios (fatigue, hydration), while ConceptNet served as a fallback for novel inputs. The verifier module improved output consistency but added latency.

5. Significance and Future Directions

Bridging the Gap: StretchBot demonstrates that structured, actionable knowledge (KGs) can effectively ground LLM-based adaptation in embodied robotics, mitigating the risk of incoherent or unsafe suggestions.
Design Trade-offs: The study highlights a critical trade-off: Adaptability vs. Predictability. While adaptive systems offer better contextual relevance, they may reduce user comfort and flow if not carefully managed. The optimal solution may be user-dependent, suggesting a need for systems that allow users to toggle between modes.
Safety and Grounding: The neuro-symbolic approach provides a pathway for safer AI in assistive robotics by constraining LLM outputs with explicit domain knowledge and verification layers.
Future Work: The authors call for larger, longitudinal studies to validate robustness, improved latency management (via lighter models or local deployment), and the integration of physiological signals for more accurate state estimation.

In conclusion, StretchBot offers a practical blueprint for next-generation assistive robots that are not just reactive but context-aware, balancing the need for structured safety with the flexibility of personalized human-robot interaction.