💻 computer science

IMR-LLM: Industrial Multi-Robot Task Planning and Program Generation using Large Language Models

The paper proposes IMR-LLM, a novel framework that combines large language models with deterministic solving methods and process trees to generate feasible high-level task plans and executable low-level programs for complex industrial multi-robot collaboration, demonstrating superior performance on the newly introduced IMR-Bench.

Original authors: Xiangyu Su, Juzhan Xu, Oliver van Kaick, Kai Xu, Ruizhen Hu

Published 2026-03-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xiangyu Su, Juzhan Xu, Oliver van Kaick, Kai Xu, Ruizhen Hu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a busy factory floor where a team of robotic arms needs to work together to build something complex, like polishing and moving metal parts. In the past, telling these robots what to do was like trying to direct a chaotic traffic jam with a walkie-talkie: you had to give very specific, step-by-step instructions, and if you missed one detail, the whole line would crash.

This paper introduces IMR-LLM, a new "super-brain" system that helps a team of robots collaborate efficiently. Think of it as a highly skilled project manager who doesn't just give orders but understands the big picture, the rules of the road, and the specific tools each worker has.

Here is how it works, broken down into three simple parts:

1. The Problem: Why Robots Get Confused

In a home, if you tell a robot to "make a sandwich," it can cut the bread first or the cheese first; the order doesn't matter much. But in a factory, the rules are strict.

The Traffic Jam: Imagine three robots trying to use the same polishing machine. Only one can use it at a time. If they all rush in, they crash.
The Dependency: You can't polish a part until it's been moved to the table. If the robot tries to polish it while it's still on the conveyor belt, nothing happens.

Old AI methods tried to guess the best order by just "chatting" with the robots. But as the factory gets bigger, the AI gets confused, makes up rules that don't exist, or forgets who is supposed to do what.

2. The Solution: The "Traffic Cop" and the "Recipe Book"

The authors created a two-step system to fix this:

Step A: The Traffic Cop (Task Planning)

Instead of asking the AI to guess the entire schedule, they use the AI to draw a map of the factory's rules (called a Disjunctive Graph).

The Analogy: Imagine the AI is a traffic cop drawing a map of a city. It marks where the roads (robots) are and where the intersections (machines) are. It knows that "Robot A can't be at Intersection 1 and Intersection 2 at the same time."
The Magic: Once the map is drawn, the system uses a mathematical calculator (a deterministic solver) to find the fastest, crash-free route. The AI doesn't have to do the hard math; it just builds the map, and the calculator solves the traffic jam perfectly. This ensures the robots never collide and always finish as fast as possible.

Step B: The Recipe Book (Program Generation)

Once the schedule is set, the robots need to know how to move their arms to do the job.

The Old Way: The AI would look at a few examples of code and try to copy them. This is like trying to write a cookbook by memorizing three specific recipes. If you ask for a new dish, the AI gets confused and writes nonsense.
The New Way (Process Tree): The authors built a modular "Recipe Tree."
- Imagine a tree where the trunk is "Transport a part."
- The branches are different ways to do it: "Use a camera on a stand" vs. "Use a camera in the robot's hand."
- The AI looks at the specific job, walks down the tree to find the right branches, and snaps together the correct code snippets.
- The Benefit: It's like having a Lego set. Instead of building a whole new castle from scratch every time, the AI just picks the right pre-made walls and windows and snaps them together. This makes the code much more reliable and less likely to break.

3. The Results: A Smooth Operation

The team tested this system in a virtual factory and a real one with actual robots.

The Benchmark: They created a new test called IMR-Bench, which is like a "driving test" for robots, ranging from easy (one robot) to hard (seven robots working together on complex tasks).
The Outcome: Their method beat all the other existing AI methods. It was better at figuring out who should do what (Task Planning) and better at writing code that actually worked (Program Generation).
Real World: They even ran it on real robots in a lab, and the robots successfully moved and polished parts without crashing, proving the system works outside of the computer.

The Big Picture

IMR-LLM is like upgrading a factory from a chaotic construction site to a well-oiled machine.

It uses AI to understand the goals and the rules.
It uses Math to solve the traffic jams.
It uses Modular Trees to write the instructions.

This means factories can finally use large teams of robots to do complex, coordinated work without needing a human to micromanage every single second. It's a huge step toward the "smart factories" of the future.

1. Problem Statement

The paper addresses the challenges of multi-robot task planning and program generation in industrial manufacturing environments. While Large Language Models (LLMs) have shown promise in simple household or manipulation tasks, they struggle in industrial settings due to:

Strict Sequential Constraints: Industrial operations (e.g., welding, polishing) have rigid execution orders and complex dependencies that differ from flexible household tasks.
Resource Conflicts: Multiple robots often compete for limited shared resources (machines, workpieces), requiring precise scheduling to avoid collisions or deadlocks.
Program Executability: Existing LLM-based code generation methods often rely on few-shot prompting with specific examples, leading to code that overfits the examples, fails to adapt to specific production line configurations, or lacks critical process steps.

The core goal is to automatically transform natural language instructions into a high-level task plan (decomposition, allocation, scheduling) and low-level executable programs for a fleet of collaborative robots.

2. Methodology: IMR-LLM Framework

The authors propose IMR-LLM, a hybrid framework that integrates LLM reasoning with deterministic solving algorithms and structured data representations. The framework consists of three main stages:

A. Task Planning (High-Level)

Instead of asking the LLM to directly output a complex schedule for all robots (which often leads to errors), the method decomposes the problem:

LLM Reasoning: The LLM analyzes the scene and instruction to generate:
- Operation Set ( $O$ ): Decomposing the task into atomic operations (e.g., transport, polish).
- Allocation Set ( $A$ ): Assigning operations to specific robots.
- Intra-workpiece Precedence ( $Q$ ): Determining the order of operations for a single workpiece.
- Note: The LLM uses rule-based prompting rather than few-shot examples to improve generalization.
Disjunctive Graph Construction: The LLM's output is converted into a Disjunctive Graph ( $G$ $G$ ).
- Vertices: Represent operations.
- Conjunctive Arcs: Represent fixed precedence constraints within a workpiece.
- Disjunctive Arcs: Represent resource conflicts (competing for the same machine or robot).
Deterministic Solving: A heuristic algorithm (First-In-First-Out, FIFO) solves the disjunctive graph to determine the optimal execution order for conflicting resources, producing a feasible Scheduling Graph ( $F$ ). This ensures the plan is mathematically valid and efficient, bypassing the LLM's weakness in complex combinatorial optimization.

B. Program Generation (Low-Level)

To generate executable code that adapts to specific environments without overfitting:

Operation Process Tree ( $T$ ): Instead of providing raw code examples, the LLM constructs a tree-structured representation of execution procedures based on provided program examples.
- The tree organizes operations into nodes (e.g., "Detect Object," "Compute Trajectory").
- It handles variations (e.g., camera mounted vs. handheld) via branching structures.
- It allows for the reuse of common functional steps across different operations.
Guided Code Generation: The LLM uses the Process Tree to generate Python code.
- It identifies the specific path (branch) in the tree required for each operation based on the scene.
- It combines code snippets from the tree nodes to create wrapper functions and execution functions.
- This transforms code generation into a path selection problem, significantly improving code completeness and executability.

C. Execution

The system drives the robots using the generated high-level plan (execution sequence) and low-level code (function calls). Operations only start when their dependencies are met.

3. Key Contributions

IMR-LLM Framework: A novel architecture that synergizes LLMs with disjunctive graphs for robust task planning and process trees for modular program generation.
IMR-Bench: A new benchmark dataset comprising 23 industrial scenes and 50 manufacturing tasks across three difficulty levels (Single Robot, Simple Multi-Robot, Complex Multi-Robot). It includes ground truth for operations, allocation, scheduling, and programs.
Comprehensive Evaluation: The method was tested in both simulation and a real-world environment with three robotic arms, demonstrating superior performance over existing baselines.

4. Experimental Results

The authors evaluated IMR-LLM against state-of-the-art baselines (SMART-LLM, LaMMA, LiP-LLM) using metrics like Operation Consistency (OC), Scheduling Efficiency (SE), Executability (Exe), and Success Rate (SR).

Performance: IMR-LLM significantly outperformed all baselines across all task complexities.
- Complex Multi-Robot Tasks: Achieved an SR of 0.68 (GPT-4o) and 0.60 (Qwen3-32B), whereas the best baseline (LiP-O) only reached 0.24.
- Scheduling Efficiency (SE): Achieved 1.00 for single and simple multi-robot tasks, and 0.75 for complex tasks, compared to near-zero for baselines in complex scenarios.
- Executability: The process tree approach resulted in high Executability scores (0.76–0.94), proving the generated code works in practice.
Ablation Studies:
- Removing the Disjunctive Graph (relying solely on LLM for scheduling) caused a drastic drop in Scheduling Efficiency, especially in multi-robot tasks.
- Removing the Process Tree (using direct examples) significantly reduced code Executability and Goal Condition Recall.
Real-World Validation: The system successfully deployed in a real-world setting with three robots, completing a transport task with correct planning and execution.

5. Significance

Bridging the Gap: IMR-LLM effectively bridges the gap between the reasoning capabilities of LLMs and the strict, deterministic requirements of industrial automation.
Reliability: By offloading complex scheduling to deterministic solvers and using structured trees for code generation, the system avoids the "hallucinations" and infeasible plans common in pure LLM approaches.
Scalability: The process tree approach allows for modular expansion; new operation types can be added by modifying the tree structure without retraining the model.
Industrial Applicability: The creation of IMR-Bench and the successful real-world deployment provide a foundational step toward autonomous, collaborative industrial robotics powered by AI.

In conclusion, IMR-LLM demonstrates that combining LLM reasoning with classical optimization techniques and structured data representations is the most viable path for deploying multi-robot systems in complex industrial environments.