💻 computer science

SHRIMP: Iterative Refinement of Robot Task Plans

SHRIMP is a system that enables non-expert users to generate and iteratively refine hierarchical robot task plans using natural language, incorporating simulation-based validation to improve perceived control and transparency before executing tasks on physical robots.

Original authors: Mya Schroder, Yuna Hwang, Callie Y. Kim, Leqian Cheng, Jeffrey Li-cheng Liu, Chenchen Zheng, Xinning He, Bilge Mutlu

Published 2026-08-11

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Mya Schroder, Yuna Hwang, Callie Y. Kim, Leqian Cheng, Jeffrey Li-cheng Liu, Chenchen Zheng, Xinning He, Bilge Mutlu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where you could ask a robot to "clean up the messy kitchen" and it would just know exactly what to do, moving every cup and bowl with perfect precision. That's the dream of collaborative robots, or "cobots," machines designed to work right alongside us in factories, farms, and hospitals. But here's the catch: telling a robot what to do is surprisingly hard. If you just say "move the cup," the robot might not know which cup, how far to move it, or whether it should pick it up gently or shove it. This is where Large Language Models (LLMs) come in. Think of these as super-smart AI brains that are great at understanding human words. Scientists have been trying to teach these AIs to translate our messy, vague sentences into precise robot instructions. However, there's a big problem: these AI brains are often "black boxes." You type in a request, and out pops a plan, but you have no idea how the AI decided to do it, or if the plan will actually work without the robot dropping a bowl or crashing into a wall. If the plan is wrong, you might not find out until the robot breaks something. This paper tackles that scary uncertainty by asking: How can we let regular people see, understand, and fix robot plans before the robot even moves?

Enter SHRIMP, a new system created by researchers at the University of Wisconsin–Madison that acts like a "flight simulator" for robot tasks. The name stands for Simulation-driven Human-in-the-loop Refinement Interface for Manipulation Planning, but you can think of it as a "Robot Plan Playground." Instead of just trusting the AI's first guess, SHRIMP lets you see the robot's plan broken down into a step-by-step list of tiny actions (like "grab," "move," "tilt") before it ever touches the real world. If the plan looks weird—say, the robot tries to pick up a bowl from the wrong angle—you can fix it right there in the computer simulation. You can tweak the numbers, rearrange the steps, or just ask the AI to try again with better instructions. Once the plan looks perfect in the simulation, only then does it get sent to the real robot.

The researchers tested this idea with 35 people who had to plan tasks like setting a table or cleaning up a kitchen. They compared three different ways of using the system: one where people could see and edit every tiny step (the full SHRIMP experience), one where they could only see the big picture steps, and one where they had to just type instructions and hope for the best (the "black box" method). The results were clear: people felt much more in control and understood the robot's actions much better when they could see and edit the detailed steps. It was like having a map versus just being told "go north." However, there was a trade-off: having all those details to check and fix made the task feel a bit more mentally tiring, especially for simple jobs where the robot didn't really need help. But for complex tasks, that extra control was a lifesaver. The study suggests that while AI is great at starting the work, we need tools that let us peek under the hood and make sure the robot isn't about to trip over its own feet.

The Core Idea: From "Black Box" to "Glass Box"

The paper argues that while AI can write robot plans, we can't just trust it blindly. The authors propose SHRIMP as a solution that turns the "black box" of AI planning into a "glass box" where everything is visible. The system works in a loop:

You speak: You type a natural language prompt, like "Clean up the table."
The AI thinks: The system uses a Large Language Model to turn your words into a hierarchical plan. This plan isn't just a paragraph; it's a list of "primitives," which are like Lego blocks of robot actions. Some blocks are big (like "Pick up the bowl"), and some are small (like "Move arm to specific coordinates").
You check: Before the robot moves, you see this plan in a physics-based simulation. This is a digital twin of the real robot and the real table. You can watch the robot try to pick up the bowl. If the simulation shows the bowl falling off the table, you know something is wrong.
You fix: You can fix the plan in two ways. You can type a new instruction (re-prompting) or you can directly edit the numbers in the plan (like changing the height of the robot's arm).
You execute: Once the simulation looks perfect, you send the plan to the real robot.

What the Study Found

The researchers ran an experiment where 35 participants tried to complete three different tasks: setting a table, preparing a snack, and cleaning up a table. Each person tried the system in three different "modes":

Plan+Edit: The full SHRIMP experience where they could see and edit every step.
Plan-Only: They could see the high-level steps but couldn't edit the details.
No-Plan: The baseline where they just typed instructions and the robot tried to do it without any visible plan or editing (the "black box").

The results showed that the Plan+Edit mode was the winner for two main reasons:

Control: People felt much more in charge of the robot. They could see exactly what the robot was going to do and change it if they didn't like it. One participant said, "Being able to edit specific parts of the robot's movements... made me feel like I had more control."
Transparency: People understood the robot's plan much better. They could see why the robot was doing something. When they couldn't see the plan (in the "No-Plan" mode), they felt confused and didn't know how to fix things when the robot made a mistake.

However, the study also found a downside. The Plan+Edit mode made people feel more mentally tired (higher "task load"). It was like having a very detailed map: it's great for getting to a complex destination, but if you're just walking to the corner store, looking at a map with every single street name might feel like too much work. For simple tasks, the extra features felt unnecessary. But for harder tasks, that extra control was essential.

How People Actually Used It

The researchers noticed that people didn't all use the system in the same way. They found three main strategies:

Stepwise: People built the plan one small piece at a time. They'd say, "Move the cup," check it, then say, "Move the bowl," and check that.
Cumulative: They started with one part and kept adding more. "Move the cup," then "Move the cup and the bowl," then "Move the cup, bowl, and spoon."
Oneshot: They tried to write the whole plan in one big, detailed sentence right from the start.

Interestingly, even when people had the ability to click and drag to fix numbers (the "Edit" part), many still preferred to just type new words to fix things. This suggests that while the tools are there, people often find it easier to just talk to the robot again than to fiddle with complex numbers.

The Takeaway

The paper concludes that for robots to be truly helpful to regular people, we can't just rely on AI to do the thinking. We need interfaces that let us see the robot's "thought process" and fix it before it causes a mess. The SHRIMP system proves that giving people the ability to see and edit the robot's plan makes them feel more confident and in control. But it also warns us that we need to be careful not to overwhelm people with too much information, especially for simple tasks. The future of robot planning isn't just about smarter AI; it's about building better bridges between human intent and robot action, letting us peek under the hood and make sure the engine is running smoothly before we hit the road.

Technical Summary: SHRIMP – Iterative Refinement of Robot Task Plans

Problem Statement

As collaborative robots (cobots) expand into domains like manufacturing, agriculture, and healthcare, the barrier to entry remains high for non-expert end-users. Traditional programming tools, such as teach pendants, require low-level specification of joint positions and step-by-step construction, which is inaccessible to most users. While Natural Language (NL) interfaces powered by Large Language Models (LLMs) and Vision-Language-Action (VLA) models offer a promising alternative for translating high-level intent into robot actions, they suffer from two critical limitations:

Semantic Ambiguity: NL instructions are often imprecise (e.g., "move left" lacks distance or reference frame), leading to plans that do not align with user intent.
Lack of Transparency: Generative models often function as "black boxes." Even when plans are generated in text, users cannot easily validate the feasibility of complex robot trajectories or object manipulations before execution, particularly when the system relies on simplified perception or lacks physics-grounded simulation.

Existing systems often fail to provide a mechanism for users to review, edit, or verify plans in a safe environment prior to physical deployment.

Methodology: The SHRIMP System

To address these issues, the authors propose SHRIMP (Simulation-driven Human-in-the-loop Refinement Interface for Manipulation Planning). SHRIMP is a system that bridges natural language input with physics-grounded simulation to enable iterative plan refinement.

System Architecture

The system integrates hardware and software components to create a "digital twin" environment:

Hardware: Two 7-DOF Franka Emika Panda robot arms equipped with Tesollo 3-finger grippers.
Perception: A single RGB-Depth Realsense camera using YOLO11 for object localization.
Simulation: Isaac Sim serves as the physics-based simulation engine, running at ~200 Hz. It shares the same motion control layer as the physical robots (using RangedIK for inverse kinematics and PD torque controllers with Coriolis compensation) to ensure the simulation acts as a faithful digital twin.
Backend/Frontend: A FastAPI backend handles task planning and state management, while a web-based frontend (HTML/JS) streams the simulation via WebRTC.

Workflow

Task Input: Users enter a natural language task prompt (e.g., "Clean up the table").
Plan Generation: An LLM (GPT-5.1) receives the task prompt, robot configuration, available primitives, and object positions. It generates a hierarchical primitive plan.
- Primitives: The plan is decomposed into High-Level primitives (e.g., pick, pour, pick_and_place) and Low-Level primitives (e.g., move_to_pose, grasp, release).
Verification: Before simulation, the system performs three automated checks:
- Grasp Validation: Ensures the gripper is within range of the target object.
- Collision Avoidance: Inserts retract motions to prevent arm-to-arm collisions.
- Trajectory Safety: Adjusts trajectory height to prevent robot-object collisions.
Iterative Refinement: Users interact with the plan in the simulation:
- Visualization: Users can step through the plan at high or low levels of abstraction.
- Editing: Users can re-order primitives, add new ones, or explicitly edit parameters (e.g., pose coordinates) via a visual interface (indicated by red arrows).
- Re-prompting: Users can modify the natural language prompt to adjust the plan.
Execution: Once the user is satisfied with the simulation, the plan is executed on the physical robot.

Key Contributions

The paper outlines three primary contributions:

System Contribution: SHRIMP introduces a physics-grounded, iterative refinement loop that allows users to edit generated plans at multiple levels of abstraction (natural language and primitive parameters) within a simulation environment.
Empirical Contribution: The authors conducted a within-participants user study ( $N=35$ $N = 35$ ) comparing three conditions:
- Plan+Edit: Full access to hierarchical plan viewing and parameter editing.
- Plan-Only: Access to high-level plan viewing only (no editing or low-level details).
- No-Plan: A black-box baseline where users can only re-prompt via natural language.
Design & Research Implications: The study provides insights into how to balance transparency, control, and cognitive load in end-user robot programming.

Results

The study evaluated perceived autonomy (control), transparency (TOROS), task load (NASA-TLX), usability (SUS), and task success.

Control and Transparency: Participants in the Plan+Edit condition reported significantly higher perceived autonomy ( $p=0.007$ ) and transparency ( $p=0.026$ ) compared to the No-Plan baseline. However, there were no significant differences between Plan+Edit and Plan-Only, nor between Plan-Only and No-Plan. This suggests that simply viewing a high-level plan is insufficient; the ability to interact with and edit the full hierarchical plan is required to achieve higher levels of control and transparency.
Task Load: The Plan+Edit condition resulted in significantly higher perceived task load ( $p=0.002$ ) compared to Plan-Only. Qualitative analysis indicated this was due to the added complexity of managing detailed plan information. However, this load increase was task-dependent; it was most significant for the simpler "Clean the Table" task, suggesting that for easier tasks, detailed editing features may introduce unnecessary complexity.
Usability and Success: There were no significant differences in System Usability Scale (SUS) scores or task success rates across conditions. Notably, all conditions received below-average usability scores, which the authors attribute to the participants' lack of familiarity with the interface and simulation tools.
Iteration Strategies: Users employed three distinct prompting strategies: Stepwise (testing subtasks individually), Cumulative (adding subtasks iteratively), and Oneshot (specifying the full plan and refining details). The Plan+Edit condition saw the highest number of iterations and simulations.
Frustration with Inconsistency: Some participants reported frustration due to the non-deterministic nature of LLMs, where similar prompts produced different plans. The transparency of the primitive plan helped users identify and correct these inconsistencies.

Significance and Claims

The paper claims that SHRIMP successfully demonstrates that iterative refinement in a physics-grounded simulation is a viable method for improving end-user control and transparency in robot planning.

Transparency: By exposing the hierarchical decomposition of tasks into primitives, SHRIMP allows users to understand how a plan differs from their intent, enabling them to reduce language ambiguity through explicit parameter editing or targeted re-prompting.
Control: The system empowers users to take precise control over robot actions, particularly when natural language is insufficient to describe specific spatial constraints.
Design Trade-offs: The authors modestly conclude that while increased control and transparency are achieved, they come at the cost of higher cognitive load. Therefore, future systems should not default to exposing all editing features but should instead adapt the level of detail based on task complexity and user expertise.

The work does not claim to solve all issues of robot programming but highlights the necessity of integrating grounded perception, structured planning, and intuitive interfaces to make generative AI-based robot planning safe and effective for non-experts.