🤖 AI

STEP: State-Aware Task Estimation and Planning with Multi-Modal LLMs for Human-Robot Collaboration

The paper proposes STEP, a state-aware framework that leverages Multi-modal Large Language Models to explicitly estimate system states and predict state transitions, thereby overcoming hallucinations and ambiguity in human-robot collaboration to significantly improve action executability and reduce final-state errors in industrial assembly tasks.

Original authors: Maitrey Gramopadhye, Prakash Baskaran, Xiao Liu, Songpo Li, Soshi Iba

Published 2026-08-28

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Maitrey Gramopadhye, Prakash Baskaran, Xiao Liu, Songpo Li, Soshi Iba

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the bustling world of modern industry, the dream of seamless teamwork between humans and machines is becoming a reality, yet it remains fraught with a fundamental misunderstanding. For a robot to truly assist a human, it must do more than simply follow a list of commands; it must understand the human's intention and anticipate what comes next. This challenge lies at the heart of a field known as long-term action anticipation, where computers try to predict a sequence of future events based on what they have just seen. In recent years, powerful artificial intelligence systems capable of processing both images and text have emerged as promising tools for this task. These systems can look at a video of a person working and guess what they are trying to build. However, a significant gap remains: while these intelligent systems are excellent at language and pattern recognition, they often lack a true understanding of the physical world. They do not naturally keep track of how the state of a room changes when an object is moved, leading them to imagine actions that are impossible or to lose sight of the ultimate goal.

To bridge this gap, researchers at the Honda Research Institute and the University of North Carolina at Chapel Hill have developed a new method called STEP, which stands for State-Aware Task Estimation and Planning. The team focused on a common industrial scenario where a human operator is assembling a structure using wooden blocks on a workbench, while a robot watches and waits to take over. In their experiments, the human would begin building a specific shape, such as a bridge or a tower, and then stop, handing the planning responsibility to the robot. The researchers wanted to see if they could teach the robot to not only guess what the final structure should look like but also to constantly update its mental map of the workspace as it planned the next moves. Unlike previous approaches that simply asked the robot to predict the next step in a sentence, this new system forces the robot to explicitly describe the current arrangement of every block, predict how that arrangement will change after each action, and then use that updated description to plan the subsequent steps.

The core innovation of STEP is its insistence on maintaining a structured, digital record of the physical world. When the robot observes the workbench, it does not just generate a vague idea of "building a tower." Instead, it creates a detailed, organized list of facts about the scene: where each of the five wooden blocks is located, whether a block is standing upright or lying flat, and exactly how it is oriented relative to the camera and other objects. The system then uses this precise description to simulate the future. It asks itself, "If I move this block here, what will the scene look like next?" and then repeats that question for the following move. By constantly checking its own predictions against the goal, the system can measure how far it is from finishing the task. If a planned sequence of moves leads to a dead end or a state that is far from the goal, the system recognizes this error and shortens its plan, choosing a different path that brings it closer to the desired outcome. This process of planning, simulating the result, and correcting the course is repeated multiple times to ensure the robot stays on track.

The researchers tested this approach in a simulated environment using a dataset of human operators teleoperating two robotic arms to assemble wooden blocks. They compared their method against a leading existing technique that did not track the state of the environment in this structured way. The results showed a clear advantage for the new system. The plans generated by STEP were significantly more practical; the researchers found that the actions predicted by their method could be successfully executed by the robot 32.8 percent more often than those from the baseline method. Furthermore, when the robot finished its task, the final structure it built was 14.8 percent closer to the intended goal than the structures produced by the older method. The study also revealed that while the older method sometimes produced longer lists of actions that matched the human's original sequence, those extra steps were often unnecessary or incorrect, whereas the new method produced shorter, more efficient plans that reliably reached the goal.

The team discovered that the success of this approach relied heavily on the accuracy of the initial steps. If the system correctly identified what the human was trying to build and accurately described the current state of the blocks, the subsequent planning was highly effective. However, if the system made an error in guessing the goal or misjudged the position of a block, those mistakes would ripple through the rest of the plan. This finding highlights the importance of the system's ability to constantly re-evaluate the physical reality of the workspace. The researchers noted that while their method is currently designed for this specific block-building scenario, the underlying principle of explicitly tracking state changes could be adapted for other industrial tasks, such as assembling machinery or constructing modular fixtures. They also acknowledged that the current system requires significant computing time to run these multiple planning cycles, making it slower than simpler methods, but they believe that as technology advances, these delays can be reduced. Ultimately, this work demonstrates that by giving artificial intelligence a way to keep a running tally of the physical world, we can create robots that are not just reactive, but truly collaborative partners capable of understanding and completing complex tasks alongside humans.

Technical Summary: State-Aware Task Estimation and Planning with Multi-Modal LLMs (STEP)

Problem Statement

Effective human-robot collaboration (HRC) in industrial settings requires robots to infer human intentions and generate actionable long-horizon plans. While Multi-Modal Large Language Models (MM-LLMs) have shown promise in Long-Term Action Anticipation (LTA) through in-context learning, they face two critical limitations in robotics:

Lack of State Awareness: MM-LLMs are pre-trained on static text and image data without real-world interaction. Consequently, they struggle to track system state transitions, often leading to "hallucinated" actions that deviate from the intended goal. Existing approaches that attempt state tracking often rely on natural language descriptions (which lack precision) or require access to ground-truth initial states, which is unrealistic in dynamic industrial scenarios.
Action Ambiguity: Standard MM-LLM outputs are often limited to high-level "verb-noun" pairs (e.g., "place on table"). This abstraction fails to capture necessary execution parameters such as specific placement coordinates, object orientation, or spatial relationships, leading to ambiguity during physical execution.

Methodology: The STEP Framework

The authors propose STEP (State-Aware Task Estimation and Planning), a pipeline that prompts an MM-LLM to explicitly estimate system states and predict state transitions alongside action plans. The framework operates without requiring direct interaction with the environment or ground-truth state inputs.

The pipeline consists of five sequential stages:

1. Task Estimation

The system takes the sequence of actions performed by a human operator ( $A_1, ..., A_k$ ) and an image of the current workstation ( $I$ ). Using a small set of in-context examples, the MM-LLM estimates the overall task ( $T$ ) the operator intends to complete (e.g., "Build an Arch"). The output is constrained to a predefined list of possible tasks to guide the model.

2. State Representation Generation

Simultaneously, the MM-LLM analyzes the workstation image ( $I$ ) to generate a structured state representation ( $S$ ) in JSON format. Unlike natural language descriptions, this structured output captures:

Location: Position of each block (e.g., on the table, on top of another specific block).
Orientation: Standing, lying face down, or lying on its side.
Top Face Orientation: Parallel or perpendicular to the camera.
Relative Positioning: Specific location (left/right/center) if stacked.

This structured format allows for quantitative analysis of the workspace.

3. Action Prediction

Using the estimated task ( $T$ ) and the history of actions ( $A_1, ..., A_k$ ), the MM-LLM predicts the sequence of future actions ( $A_{k+1}, ..., A_n$ ) required to complete the task. Notably, the authors found that including image data in this specific prompt degraded performance; thus, this stage relies solely on the textual task and action history.

4. State Propagation

To address the lack of state tracking, STEP explicitly propagates the state. The MM-LLM is prompted with the current state ( $S$ ), the action history, and the next predicted action ( $A_{k+1}$ ) to output the next state ( $S_{k+1}$ ) as a JSON object. This is performed iteratively to generate a sequence of future states ( $S_{k+1}, ..., S_n$ ) corresponding to the predicted actions. A chain-of-thought approach is used, where the model first describes changes in natural language before formatting the next state as JSON.

5. Rollouts and Truncation

STEP employs an iterative "rollout" mechanism to refine the plan:

Distance Calculation: The system calculates a quantitative distance ( $d_T$ ) between the current/predicted states and the estimated goal state. This is the total number of differences across the 5 blocks and their 4 characteristics (max $d_T = 20$ ).
Truncation: The system evaluates multiple rollouts and truncates the action sequence at the state with the lowest $d_T$ .
Iteration: The truncated state becomes the new starting point for the next rollout. This process repeats until $d_T = 0$ or a maximum number of rollouts is reached, ensuring the plan converges toward the goal.

Experimental Setup

Dataset: A simulated block assembly scenario involving 495 instances from 19 users. Users teleoperated two Franka robot arms to assemble five identical wooden blocks into one of eight structures (e.g., Tuning Fork, Bridge, Horse).
Baseline: A state-of-the-art LTA approach inspired by AntGPT, which performs task estimation and a single action prediction rollout without state tracking or structured state representation.
Metrics:
- Executability: Percentage of actions passing logical and physical constraints (e.g., preconditions, object poses).
- Final-State Error: The difference between the achievable final state and the true goal state.
- Longest Common Subsequence (LCS): Measures the overlap in action order between predicted and recorded sequences.
- Task Correctness & Current State Error: Accuracy of task inference and state estimation.

Key Results

Evaluated on a 50% task completion scenario using GPT-4o, STEP demonstrated significant improvements over the baseline:

Executability: The paper reports an overall improvement of 32.8% in action executability compared to the state-of-the-art. In the specific GPT-4o ablation (Table III), STEP achieved 78.12% executability compared to the baseline's 71.68%.
Final-State Error: The paper reports an overall improvement of 14.8% in final-state error. In the specific GPT-4o ablation (Table III), STEP reduced the error to 4.38 compared to the baseline's 4.53.
LCS: The baseline achieved a higher LCS. The authors attribute this to the baseline generating longer, less efficient action sequences that happened to overlap more with the recorded data, whereas STEP's truncation strategy produced shorter, more direct plans.
Robustness: STEP outperformed the baseline across various MM-LLM sizes (GPT-4o, GPT-4.1, GPT-5.1, GPT-5.2), though performance degraded with smaller models (GPT-4o mini) due to lower task estimation accuracy.

Significance and Claims

The paper claims that STEP bridges the gap between the high-level reasoning of MM-LLMs and the precise requirements of robotic execution in industrial settings. By explicitly modeling state transitions and using structured JSON representations, the method:

Mitigates Hallucination: Prevents the generation of actions that are physically impossible or diverge from the goal by quantitatively tracking state distance.
Enables Parameterized Execution: Moves beyond "verb-noun" pairs to provide the specific assistance parameters (location, orientation) required for real-world execution.
Operates Without Ground Truth: Unlike prior works requiring initial ground-truth states, STEP infers the state from visual input, making it applicable to real-world scenarios where such data is unavailable.

The authors acknowledge limitations, specifically that the current implementation is tailored to block assembly and that the iterative rollout process introduces latency (approx. 148 seconds vs. 18 seconds for the baseline), which must be addressed for real-time deployment. They suggest future work could focus on automating prompt generation for new scenarios and optimizing inference speed through local hosting or model fine-tuning.