💻 computer science

Pack it in: Packing into Partially Filled Containers Through Contact

This paper presents a contact-aware packing approach that utilizes a physics-aware perception system and a contact-based multi-object trajectory optimizer to enable robots to successfully place new items into partially filled containers by intentionally interacting with and rearranging existing objects to create necessary space.

Original authors: David Russell, Zisong Xu, Maximo A. Roa, Mehmet Dogar

Published 2026-05-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: David Russell, Zisong Xu, Maximo A. Roa, Mehmet Dogar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to pack a suitcase for a trip. Usually, you start with an empty bag and carefully fold your clothes in. But what if you are trying to add one last shirt to a suitcase that is already stuffed to the brim, and the clothes inside have shifted into a messy, tight jumble? There is no big empty hole to just drop the shirt in.

In the real world, warehouse robots face this exact problem. They need to put new items into boxes that are already half-full, but the items inside have moved around, leaving no clear space. Traditional robots are programmed to be "collision-free," meaning they are told to never touch anything. If there's no empty space, they just give up.

This paper introduces a smarter robot that knows how to play "Jenga" with a suitcase. Instead of avoiding the mess, it learns to push, nudge, and slide the existing items to make room for the new one, all in one smooth motion.

Here is how the system works, broken down into three simple parts:

1. The "Push-and-Insert" Dance (Trajectory Optimization)

Imagine you are trying to slide a book into a tight shelf. You can't just shove it straight in; you have to push the books on the left slightly to the right while you slide the new book in.

The robot does this using a sophisticated math trick called iLQR. Think of this as the robot running a super-fast movie in its head. It simulates thousands of different ways to move its arm. It asks: "If I push this red can to the left, will it bump into the green box? If I nudge the blue box, will it slide out of the way?"

It finds a path where the robot's arm physically interacts with the clutter, shoving things aside just enough to create a tiny gap, and then slides the new object into that gap simultaneously. It doesn't stop to rearrange everything first; it does the rearranging and the packing at the same time.

2. The "Crystal Ball" (Physics-Aware Perception)

Here is the tricky part: When the robot is pushing things around, its own arm and the object it's holding often block the camera's view. It's like trying to watch a game of pool while someone is standing right in front of the table. The robot loses sight of the items it just pushed.

To solve this, the robot uses a "Physics Crystal Ball." Even when it can't see an object, it knows how heavy it is and how it moves. If the robot pushes a box and the camera loses it, the robot calculates: "I pushed it with this much force, so it must have moved to this spot." It combines what it sees with what it knows about physics to keep track of the items even when they are hidden.

3. The "Smart Guess" (Placement Planner)

Before the robot even starts moving, it has to decide where to put the new item. If it picks a spot that is too crowded, the whole plan fails.

The system acts like a game of Tetris. It quickly tests hundreds of different spots in its mind. For each spot, it simulates dropping the new item and watching how the existing items react.

Bad Spot: The new item tries to go in, but it pushes everything up into the air (like trying to force a square peg in a round hole). The system rejects this.
Good Spot: The new item slides in, gently nudging the neighbors sideways, and everything settles flat at the bottom. The system picks this spot.

The Results: How Well Did It Work?

The researchers tested this on a real robot arm in a lab. They created 40 different messy scenarios where a container was already full of random objects (like cans and boxes).

The Winner: Their new system, named PackItIn, successfully packed the item 70% of the time.
The Losers: They compared it to two "dumb" versions:
1. One that didn't know where to put the item (random guessing).
2. One that didn't understand physics (it thought the items were ghosts that wouldn't move).
  Both of these failed almost all the time (only about 17-20% success).

Why did it sometimes fail?
The robot failed mostly because of two things:

The Grip: The robot used a simple vacuum pump to hold the item. When it pushed other objects, the item sometimes spun around or slipped off the suction cup.
The Eyes: Sometimes the camera got blocked so badly that the robot's "physics crystal ball" couldn't guess the positions correctly, leading to a crash.

The Bottom Line

This paper shows that robots don't always need to be gentle and avoid touching things. Sometimes, to get a job done in a crowded space, they need to be a little bit pushy. By combining smart planning, a "crystal ball" for tracking hidden objects, and the ability to push things aside while packing, robots can handle messy, real-world warehouses much better than before.

Problem Statement

The paper addresses the challenge of robotic bin-packing in warehouse automation, specifically the task of inserting a new object into a container that is already partially filled. Unlike traditional bin-packing approaches that assume empty containers or rely on collision-free motion planning, this work tackles scenarios where continuous free space is insufficient. In such "cluttered" environments, objects often shift during transport, creating suboptimal arrangements. The core problem is to physically interact with existing objects—pushing, nudging, or displacing them—to create the necessary space for a new item, all within a single continuous action rather than separate rearrangement and insertion phases.

The authors identify three primary challenges:

Motion Planning: Generating a trajectory that simultaneously inserts the new object and rearranges existing ones, moving beyond simple collision-free paths.
Robust Execution: Handling the discrepancy between physics simulations and real-world contact dynamics, particularly when objects are occluded from vision systems during manipulation.
Feasible Pose Selection: Determining a target placement pose that is physically achievable without causing system failure, acknowledging that a poor choice can render the motion planning problem unsolvable.

Methodology

The proposed system, named PackItIn, integrates three main components within a Model Predictive Control (MPC) framework:

1. Physics-Based Trajectory Optimization (iLQR)
The core motion planner formulates packing as a trajectory optimization problem solved using the iterative Linear Quadratic Regulator (iLQR) algorithm.

Contact Awareness: The system uses MuJoCo as a physics simulator to model contact interactions. It defines a cost function with residuals for position, orientation (upright), end-effector force, joint limits, and obstacle repulsion.
Simultaneous Action: Unlike prior works that separate rearrangement and insertion, this formulation allows the robot to displace existing objects while moving the new object toward its goal in one continuous action.
Cost Decay: To prevent the robot from focusing on displacement indefinitely, obstacle repulsion weights decay over time, prioritizing the final insertion.

2. MPC Framework with Physics-Aware Perception
To bridge the gap between simulation and reality, the trajectory optimization is embedded in an asynchronous MPC scheme.

State Estimation: The system tracks object poses using an OptiTrack vision system. However, to handle occlusions (when the robot or gripped object blocks the camera), it integrates a physics-based state estimator. This estimator predicts the poses of occluded objects based on the robot's motion and the last known states of the objects.
Control Loop: A low-level joint PD controller with feed-forward torque tracks the optimized trajectory, allowing the system to re-optimize at 20 Hz to correct for physical inaccuracies and model mismatches.

3. Placement Planner
Since a target pose is not always known or guaranteed to be feasible, a high-level planner suggests a suitable location.

Sampling and Simulation: The planner samples random poses within the container's bottom plane. For each sample, it "teleports" the new object into the physics simulator and integrates the physics for $M$ steps to resolve collisions.
Cost Evaluation: A cost function evaluates the disturbance caused by the placement. It penalizes large displacements in the vertical axis (indicating the object is being pushed out of the container) and lasting penetrations. The planner selects the pose that minimizes disturbance to the existing arrangement.

The execution is divided into three stages: PRE-PACK (positioning above the target), INSERT (displacing objects and lowering the item), and FINE-TUNE (aligning the item to the exact target pose).

Key Contributions

The paper claims the following contributions:

Generalized Contact-Based Formulation: A trajectory optimization approach based on iLQR that handles objects of any shape (not limited to cuboids) and implicitly manages multi-object interactions through a general-purpose physics simulator.
Simultaneous Rearrangement and Insertion: A method that plans for contact-based interactions to clear space and insert an object in a single continuous action, improving time efficiency compared to two-phase approaches.
Physics-Informed Placement Planner: A module that outputs physically feasible packing poses by simulating interactions, ensuring the chosen goal does not lead to system failure.
Robust MPC Integration: The integration of the above methods into an MPC framework with a physics-aware perception system to handle occlusions and model inaccuracies on real hardware.

Experimental Results

The authors evaluated the system on a Franka-Emika Panda robot with a vacuum pump attachment in a real-world setting and compared it against two baselines:

Uninformed Pose: Uses a random target pose (no placement planner).
Uninformed Physics: Uses the placement planner but ignores the physics of existing objects during trajectory optimization (treats them as static obstacles).

Real-World Performance:

PackItIn achieved a 70% success rate (28/40 trials).
Uninformed Physics achieved 20% success.
Uninformed Pose achieved 17.5% success.
The full method significantly reduced execution time and end-effector force errors compared to baselines.
Failure Modes: Failures were primarily attributed to object detachment (due to the object rotating on the pump tip) and force failures (jamming). The authors noted that rotational misalignment of the object relative to the pump tip was the primary driver of failure.

Simulation Performance:

In simulation, PackItIn achieved a 93% success rate.
The gap between simulation and reality was attributed to the "compliant" nature of the MuJoCo simulator (allowing interpenetration) and the absence of real-world failure conditions like object detachment or force-based safety halts.

Significance and Claims

The paper positions this work as a step toward more capable warehouse automation systems, specifically extending the capabilities of existing stowing and bin-packing systems (such as those by Amazon Robotics) by enabling explicit planning for contact-based interactions.

The authors modestly claim that their system demonstrates the feasibility of using contact-aware trajectory optimization for packing in cluttered environments on real hardware. They emphasize that while the current implementation relies on marker-based tracking and simple geometries for ease of implementation, the underlying method is theoretically applicable to complex geometries.

The paper concludes that to transition this system to industrial settings, future work must address:

Replacing marker-based perception with a more generalizable vision system.
Improving the modeling of the interaction between the object and the pump tip (to prevent detachment).
Developing robust physics estimates for motion planning in higher levels of clutter, potentially requiring dimensionality reduction techniques.