💻 computer science

InteLiPlan: An Interactive Lightweight LLM-Based Planner for Domestic Robot Autonomy

InteLiPlan is an interactive, lightweight, and robot-agnostic LLM-based framework that enhances domestic robot autonomy through real-time onboard computing and human-in-the-loop intervention, achieving a 95% success rate in complex tasks with failure recovery across both simulation and real-world platforms.

Original authors: Kim Tien Ly, Kai Lu, Ioannis Havoutis

Published 2026-01-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Kim Tien Ly, Kai Lu, Ioannis Havoutis

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, well-read robot butler named InteLiPlan. Its job is to help you around the house, like fetching a drink or tidying up. However, there's a catch: this robot doesn't have a supercomputer in its head, and it can't see everything perfectly all the time. Sometimes it gets confused, or it tries to grab something that is actually too far away.

This paper introduces InteLiPlan as a solution to make domestic robots smarter, safer, and more independent without needing massive amounts of data or expensive hardware. Here is how it works, broken down into simple concepts:

1. The "Lightweight" Brain

Most advanced robot brains today are like giant libraries that need to be stored in a cloud server far away. They are heavy, slow to load, and require huge datasets to learn.

InteLiPlan is different. Think of it as a pocket-sized encyclopedia that fits right inside the robot's own computer. It uses a "lightweight" version of a Large Language Model (LLM)—the same kind of AI that powers chatbots—but it's been specially trained to understand robot commands. It doesn't need to call the internet for help; it can think and decide right there on the robot.

2. The "Three-Check" System

Before the robot moves, InteLiPlan doesn't just guess. It acts like a careful project manager who runs three quick checks before giving an order:

The Eyes (Vision): "Can I actually see the object you asked for?"
The Body (Feasibility): "Even if I see it, can my arm physically reach it without bumping into the table?"
The Memory (History): "Did we try this before and fail? What happened last time?"

If the answer to any of these is "No," the robot doesn't just crash or give up. It pauses and says, "I can't do that yet."

3. The "Human-in-the-Loop" Safety Net

This is the most unique part. Imagine you ask the robot to "bring me a drink."

Scenario A: The robot sees a Coke and a Pepsi. It doesn't know which one you want.
Scenario B: The robot sees the Coke, but it's behind a heavy chair it can't move.

Instead of guessing and potentially failing, InteLiPlan has a "Human-in-the-Loop" feature. It acts like a student raising their hand to ask the teacher for help. It tells you, "I see two drinks, which one?" or "I see the Coke, but I can't reach it."

You then give a quick correction (e.g., "Grab the Pepsi" or "Move the chair first"). The robot instantly updates its plan and tries again. This makes the system robust because it admits when it's stuck and asks for guidance rather than making a mistake.

4. Learning Without a Massive Library

Usually, to teach a robot a new trick, you need thousands of hours of video data. InteLiPlan is like a chameleon. It was trained on a small, specific set of examples (using a technique called "fine-tuning") that teaches it how to talk to the robot's sensors and motors.

Because it learns the language of the robot rather than memorizing specific movements for one specific robot, it can jump from a Toyota Human Support Robot (which looks like a small cart with arms) to an ANYmal D (a four-legged dog robot with an arm) and work perfectly without needing to be retrained. It's like a universal translator that works for any robot body.

5. Real-World Results

The researchers tested this on real robots in real rooms.

Success Rate: In a "fetch me" task, the robot succeeded 95% of the time, even when it had to recover from mistakes or ask for help.
Speed: It can plan a whole sequence of actions in less than 7 seconds using only the computer built into the robot.
Comparison: It performed just as well as much larger, more expensive systems (like the famous SayCan system) but did it with a tiny fraction of the computing power.

Summary

InteLiPlan is a smart, lightweight planner that lets domestic robots think for themselves but know when to ask for help. It checks if it can see and reach things, admits when it's confused, and uses human guidance to fix problems on the fly. It proves that you don't need a supercomputer to make a helpful, safe, and adaptable robot for your home.

Technical Summary of "InteLiPlan: An Interactive Lightweight LLM-Based Planner for Domestic Robot Autonomy"

Problem Statement
The integration of Large Language Models (LLMs) and Vision-Language Models (VLMs) into robotics has advanced embodied AI, yet significant challenges remain in deploying these models for domestic autonomy. Current approaches often rely on massive datasets, offboard computational resources, or handcrafted domain specifications (e.g., PDDL), which limit scalability and real-time deployment. Furthermore, existing planners frequently fail to account for robot kinematics and geometric constraints, leading to plans that are semantically correct but physically infeasible. Additionally, purely autonomous systems struggle with partial observability, ambiguous user instructions, and execution failures, lacking robust mechanisms for recovery without extensive retraining or human oversight.

Methodology
The authors propose InteLiPlan, a lightweight, robot-agnostic framework designed to run entirely onboard using real-time computing resources. The system functions as a centralized, multimodal planner that bridges natural language commands, visual perception, and motion feasibility.

Multimodal Input Formulation: The planner accepts three primary inputs:
1. User Command ( $T_{user}$ ): Natural language requests or recovery guidance.
2. Visual Observation ( $f_{seen}$ ): Textual feedback derived from an onboard RGB-D camera (using YOLO for object detection) indicating the presence and location of target objects.
3. Feasibility Score ( $f_{score}$ ): A binary signal generated by a motion verification module (utilizing a reachability graph from R-LGP) that determines if a whole-body collision-free trajectory exists to the target.
4. History ( $T_{hist}$ ): A record of the conversation and previous failure states to maintain context.
Architecture and Fine-Tuning: The core is a fine-tuned lightweight LLM (specifically Mistral 7B and Deepseek 8B). Instead of relying on zero-shot prompting or massive end-to-end training, the authors employ Low-Rank Adaptation (LoRA) for parameter-efficient fine-tuning. The training dataset consists of pure-text examples formatted as multimodal conversations (e.g., <history>, <user>, <vision>, <feasibility> $\rightarrow$ <robot>), where the output is a direct sequence of action primitives (e.g., go, pick, place). This approach allows the model to learn the mapping from multimodal constraints to executable API calls without requiring robot-specific data.
Human-in-the-Loop (HITL) Mechanism: A critical component of the framework is the ability to handle failures through human intervention. The system categorizes failures into planning (vision/feasibility), task confusion (ambiguity), and execution failures. Upon detecting a failure, the robot reports the issue contextually to the user. The human can then provide corrective guidance, which the LLM uses to replan. This mechanism ensures safety and robustness in complex, partially observable domestic environments.

Key Contributions

Lightweight Onboard Planning: Demonstrates that a 7B parameter LLM, when fine-tuned with a specific multimodal structure, can achieve state-of-the-art planning performance while running entirely on onboard hardware (NVIDIA RTX 2070), eliminating the need for cloud dependency.
Robot-Agnostic Pipeline: The framework does not require retraining for different robot embodiments. By abstracting robot capabilities into a shared skill set and using reachability graphs for feasibility checks, the same model was successfully deployed on two distinct platforms: the Toyota Human Support Robot (HSR) and the ANYmal D with a Unitree Z1 arm.
Interactive Failure Recovery: Unlike prior works that focus solely on self-recovery or high-level task planning, InteLiPlan explicitly integrates human guidance for corner cases (e.g., ambiguous references, lighting-induced vision failures), significantly improving task completion rates in failure scenarios.
Textualized Multimodal Data: The authors introduce a method to convert visual and geometric constraints into textual signals for LLM training, enabling the model to reason about physical feasibility without processing raw sensor data directly.

Experimental Results
The framework was evaluated in simulation and on real-world robots across three main metrics: task planning success, failure explanation, and failure recovery.

Robustness to Failure: In the "fetch me" task, InteLiPlan achieved a 95% success rate in scenarios requiring failure recovery. In contrast, prompt-based baselines (Mistral-Prompt, Deepseek-Prompt) saw success rates drop significantly (e.g., from 100% to ~50% or lower) when facing unseen objects or commands requiring recovery.
Generalization: The method achieved 100% success in "no-failure" scenarios across both seen and unseen commands/objects. In "failure" scenarios, the fine-tuned models (Mistral-Ours) consistently outperformed their prompt-based counterparts, particularly in handling task confusion and execution failures.
Scalability (SayCan Benchmark): When evaluated on 101 tasks from the SayCan dataset, InteLiPlan (Mistral-7B) achieved an 83% planning success rate and 82% execution success rate. This performance is comparable to the 504B parameter PaLM-SayCan system (84% planning, 74% execution), despite using a model with significantly fewer parameters and no robot-specific training data.
Cross-Embodiment: The system successfully generalized across the HSR (11-DoF) and Anytree (12-DoF) platforms without additional fine-tuning, validating its robot-agnostic nature.
Latency: The system generates executable action sequences in less than 7 seconds on onboard resources, with the feasibility check being the most time-consuming component (~5s).

Significance and Claims
The paper claims that InteLiPlan addresses the critical gap between the reasoning capabilities of LLMs and the physical constraints of robotic systems. By combining a lightweight, fine-tuned LLM with a structured multimodal input pipeline and a human-in-the-loop recovery mechanism, the authors demonstrate a path toward robust, interpretable, and deployable domestic robot autonomy. The work highlights that high-level planning does not necessarily require massive models or extensive datasets; rather, effective integration of perception, feasibility checks, and human guidance allows lightweight models to achieve reliability comparable to much larger state-of-the-art systems. The authors conclude that their approach reduces the effort of defining traditional task domains and offers a practical solution for resource-constrained, real-world robotic applications.