💻 computer science

iPack: Intuitive Bin Packing with Large Language Models

This paper introduces LLM-Pack, a novel, modular approach that leverages language and vision foundation models to generate human-like grocery packing sequences without requiring dedicated training for new items, thereby addressing the critical challenge of preventing product damage in retail automation.

Original authors: Yannik Blei, Michael Krawez, Adrian Göß, Devadas Vijayan Sheela, Tobias Jülg, Pierre Krack, Florian Walter, Wolfram Burgard

Published 2026-06-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yannik Blei, Michael Krawez, Adrian Göß, Devadas Vijayan Sheela, Tobias Jülg, Pierre Krack, Florian Walter, Wolfram Burgard

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are helping a friend pack a moving truck. You have a pile of random items: a heavy cast-iron skillet, a box of delicate glassware, a bag of frozen peas, and a loaf of bread. If you just throw them in randomly, the glass will shatter, the bread will get squished, and the frozen peas will melt. You need a plan that respects the "personality" of each item.

This is exactly the problem the paper iPack tries to solve, but for robots.

The Problem: Robots Are Bad at "Common Sense"

Robots are great at moving things in factories where everything is a perfect box. But in the real world—like a grocery store or a kitchen—items are weird. Some are heavy, some are fragile, and some are sticky. Traditional robots don't have "common sense." They don't know that you shouldn't put a heavy watermelon on top of a carton of eggs. Usually, to teach a robot this, you have to spend months manually programming rules for every single type of object, which is impossible because there are too many things in the world.

The Solution: iPack (The Robot with a Brain)

The authors created iPack, a system that lets a robot pack items intelligently without needing to be retrained for every new object. Think of iPack as a robot that has two superpowers: Eyes and Common Sense.

Here is how it works, step-by-step:

1. The "Eyes" (Perception)

First, the robot looks at the table where the items are scattered. It uses advanced cameras and AI to find every object.

The Magic Trick: Instead of just seeing "a box," the robot asks a smart AI (a Vision Language Model) to identify the object. Is it a can of soup? A bag of chips?
The Guess: The robot also needs to know how heavy or big the items are. Since it can't weigh them perfectly with a camera, it uses the AI to "look up" the item online (like checking a recipe or product page) to get the standard weight and size. If it's a weird fruit, the AI guesses based on how it looks.

2. The "Common Sense" (Planning)

This is the most important part. The robot asks a Large Language Model (LLM)—basically a super-smart chatbot trained on all human knowledge—a simple question: "If I have these specific items, what is the safest way to stack them so nothing breaks?"

The AI answers with rules like:

"Don't put the frozen spinach next to the bread (it will make the bread soggy)."
"Put the heavy jar at the bottom."
"The banana needs a soft spot, not under the cereal box."

The robot then takes these "human-like" rules and feeds them into a mathematical puzzle solver (called Mixed-Integer Linear Programming). This solver acts like a 3D Tetris master, calculating the perfect arrangement that fits everything in the box while obeying all the safety rules.

3. The "Hands" (Execution)

Once the plan is ready, the robot picks up the items one by one, starting from the bottom of the stack and working its way up. It uses a special, soft gripper (like a gentle hand) that can feel if it's squeezing too hard. It places each item exactly where the math told it to go.

Why Is This Special?

No Training Needed: You don't have to teach the robot what a "pineapple" is. If you show it a pineapple for the first time, the AI knows what it is and how to pack it. It works with any object (Open-Vocabulary).
Finds the Right Box: It doesn't just pack a box; it can look at a shelf of different-sized boxes and pick the smallest one that fits everything, saving space.
Human-Like: The researchers tested this by asking real humans how they would pack items. They created a "Consistency Score" to measure how much the robot's packing matched human intuition. iPack scored very high, meaning it packed things the way a careful human would.

The Results

The team tested iPack in a computer simulation and with a real robot arm.

Success Rate: The robot successfully packed items about 93% of the time.
Efficiency: It packed items more densely (tighter) than older methods that didn't care about safety rules.
Mistakes: When it failed, it was usually because the camera got a little blurry (making the robot miss the grip) or the robot bumped the side of the box.

In a Nutshell

iPack is a robot system that uses the "brain" of a chatbot to understand what objects are and how to treat them, and the "math" of a puzzle solver to arrange them perfectly. It allows robots to pack groceries or moving boxes with the same care and common sense a human would use, without needing to be manually programmed for every single item in the store.

Technical Summary of iPack: Intuitive Bin Packing with Large Language Models

Problem Statement
While robotics and automation have advanced significantly in warehouse logistics, the specific task of packing diverse items (such as groceries) into bins remains underexplored compared to bin picking. Existing bin packing methods primarily focus on optimizing packing density and object stability within controlled environments where item shapes are known and pre-packed. However, in open-world scenarios like grocery packing, the order of packing is critical for product integrity (e.g., preventing heavy items from crushing fragile ones, or keeping frozen goods away from moisture-sensitive items). Defining explicit rules for these constraints is difficult due to the vast variety of objects and the need for semantic reasoning. Current approaches often require manual rule specification or training on specific datasets, limiting their adaptability to new items or open-vocabulary scenarios.

Methodology
The paper introduces iPack, a novel framework that combines Vision Language Models (VLMs), Large Language Models (LLMs), and Mixed-Integer Linear Programming (MILP) to automate bin packing with human-like semantic constraints. The system operates in three distinct steps:

Perception Step:
- The system uses GroundingDINO to detect objects in an RGB-D scene and SAM 2 to generate segmentation masks.
- To filter out distractors (like the box or robot gripper) and refine object selection, a VLM is prompted with visual numbering to identify grocery items.
- The VLM estimates object properties (weight, size, pose). For standardized items, it leverages an agentic internet search to retrieve precise dimensions; for non-standardized items (e.g., produce), it relies on visual appearance.
- Object tracking is maintained using the Hungarian algorithm, combining mask Intersection over Union (IoU) and centroid distance.
Planning Step:
- An LLM is queried to generate a set of critical item pairs ( $F$ ) representing "product integrity constraints" (e.g., "Item $j$ cannot be placed on top of Item $i$ ").
- These constraints, along with contact avoidance rules (e.g., frozen items not touching bread), are integrated into a Mixed-Integer Linear Programming (MILP) framework (solved via PySCIPOpt).
- The optimizer computes a spatially optimal packing scheme that minimizes container volume (selecting the smallest suitable box from a set) while satisfying non-overlap and integrity constraints.
- To ensure robotic feasibility, the planner incorporates manipulation constraints: it restricts object orientations to stable poses, applies collision padding (5mm) for gripper dimensions and perception noise, and minimizes the sum of vertical ( $z$ ) positions to ensure a stable stack.
Execution Step:
- A robotic manipulator executes the plan by picking and placing items in ascending order of their $z$ -positions (bottom-up).
- The grasp pose is constructed using Principal Component Analysis (PCA) on the segmentation masks projected onto the point cloud. The approach direction is vertical, and the gripper aligns with the smaller principal component of the mask.
- The system uses a standard RRT motion planner for pick-and-place operations.

Key Contributions

Open-Vocabulary Packing: A novel approach that optimizes packing space under dynamically generated product integrity constraints without requiring re-training for new item classes.
Packing Consistency Score ( $C$ ): A new data-driven metric based on human preferences to evaluate how well a packing scheme adheres to human-like stacking rules. This metric was derived from a web-based survey involving 84 participants and 14 object classes.
Comprehensive Evaluation: Extensive validation in both simulation and on a real robot (Franka Research 3), demonstrating the system's ability to handle diverse grocery items and select optimal container sizes.
Open Source: Release of the source code, manipulation stack, and evaluation dataset to facilitate integration into custom scenarios.

Experimental Results

VLM Sensitivity: The authors evaluated four VLMs (GPT-5, GPT-5-mini, Claude Sonnet 4.5, Gemini 3 Flash) for constraint generation. GPT-5 yielded the best Packing Consistency Score ( $C=0.2$ ), indicating the closest alignment with human preferences, while Claude Sonnet 4.5 achieved the highest packing density but with a higher violation score ( $C=1.6$ ).
Baseline Comparison: Compared against offline methods (Elhedhli et al., Dube et al.) and an online method (GOPT), iPack achieved the lowest number of violated constraints ( $C=0.2$ ) and a high packing density ($0.445$). The vanilla version of iPack (without integrity constraints) achieved higher density ($0.478$) but significantly more violations ( $C=2.4$ ), highlighting the trade-off between density and product safety.
Real-World Performance: In end-to-end experiments with 6 to 12 items, the system achieved a 92.6% manipulation success rate. Failures were attributed to RGB-D noise affecting grasp estimation, gripper collisions with container walls, and placement inaccuracies leading to downstream collisions. The Packing Consistency Score increased with the number of items, consistent with the complexity of the planning problem.

Significance and Claims
The paper claims that iPack bridges the gap between traditional, density-focused bin packing and the semantic requirements of real-world logistics. By leveraging the common sense knowledge of pretrained LLMs and VLMs, the system can handle arbitrary objects without manual rule specification or training data collection. The authors emphasize that their approach is zero-shot and open-vocabulary, making it adaptable to various packing domains and user preferences. While acknowledging limitations in manipulation robustness (e.g., slippage, wall collisions), the work demonstrates that semantic constraints can be successfully integrated into optimization frameworks to preserve product integrity in automated grocery packing. Future work is suggested to incorporate vision-language-action models to handle contact-rich behaviors like pushing for error correction.