Leveraging Inter-object Affordances for Efficient Planning in Contact-rich Tasks

Robots have long been masters of the factory floor, where they repeat the same precise movements on identical parts. But step into a real kitchen, and the rules change. Here, objects come in every shape, size, and material, and they interact in complex ways. A heavy frying pan cannot be balanced on a thin salt shaker, and a round glass mug will roll off a curved surface. For a robot to navigate this messy reality, it needs more than just a list of steps; it needs to understand the physical "rules of engagement" between objects. This is the challenge of task-and-motion planning, a field that tries to bridge the gap between high-level goals, like "organize the table," and the low-level physics of how a robot arm actually moves. While modern artificial intelligence has made strides in understanding language and images, getting a robot to reliably stack a variety of household items without toppling them remains a significant hurdle.

A team of researchers at the University of Innsbruck has tackled this problem by teaching a planning system to think about the physical relationships between objects before it ever moves a muscle. Instead of relying on simplified models that treat all objects as uniform blocks, they developed a method that accounts for the specific affordances of real-world items. In this context, an affordance is simply what an object allows a robot to do with it based on its shape and material. For instance, a flat cutting board offers a stable surface to hold other items, while the round bottom of a salt shaker does not. The researchers built a system that uses these physical truths to filter out impossible actions before the robot even attempts them, ensuring that a plan to stack a mug on a pan is only generated if the physics actually allow it.

The core of their work involves a new way of defining how a robot interacts with its environment. They created a set of rules that describe how objects can be grasped, supported, lifted, or slid. For example, the system understands that a frying pan can be lifted by its handle but is too heavy and awkward to be grabbed by its rim. It knows that a wooden tray is too heavy to be lifted off a table but can be slid across the surface. By encoding these physical constraints directly into the planning process, the robot avoids wasting time trying to execute actions that are physically impossible, such as trying to balance a large, heavy object on a small, unstable surface. This approach, which the authors call Unified TAMP, allows the robot to generate a sequence of actions that is guaranteed to be feasible in the real world, rather than just theoretically possible.

To test this idea, the researchers set up a simulated kitchen environment featuring a robot arm and a collection of common items: a salt shaker, a glass mug, a cutting board, a frying pan, and a wooden tray. The goal was to organize these scattered items by stacking them neatly onto the tray. They created over a hundred different starting scenarios, varying the number of objects and their initial positions, to see how well their method held up under pressure. They compared their new system against two other approaches: an older version of the planning tool that ignored specific physical properties, and a system that relied on a large vision-language model to guess the right moves based on common sense.

The results were striking. The new system, which incorporated the detailed physical rules, succeeded in creating valid plans nearly every time it was tested, achieving a success rate of up to one hundred percent when the rules were perfectly known. Even when the system had to use a vision model to identify the objects and their properties, it still succeeded ninety percent of the time. In contrast, the older planning method, which treated objects as generic blocks, failed more than half the time because it tried to stack items in ways that defied physics, like placing a heavy pan on a tiny shaker. The vision-language model, despite its ability to understand language and images, struggled significantly, succeeding only about forty percent of the time. It often missed subtle physical details, such as the fact that a round surface cannot support a heavy object, leading to plans that looked logical in text but would fail in reality.

Beyond success rates, the speed of planning was a major differentiator. The new system generated plans in a fraction of a second, roughly two orders of magnitude faster than the vision-language model, which took over a minute on average. This speed comes from the fact that the new system uses a structured, logical framework to eliminate impossible options immediately, whereas the vision model has to reason through every possibility from scratch. The researchers found that as the number of objects increased, the older methods became increasingly slow and error-prone, while their new approach maintained its efficiency and accuracy. This suggests that for robots to operate effectively in complex, real-world environments, they need a solid foundation of physical understanding that goes beyond just recognizing what an object is.

The study also highlighted the limitations of relying solely on artificial intelligence to understand the physical world. While vision-language models are powerful tools for recognizing objects and generating ideas, they lack the rigorous, step-by-step reasoning required to ensure a plan will work in a physical simulation. The researchers showed that by combining the common-sense knowledge of these models with a strict, rule-based planning system, they could get the best of both worlds: the ability to recognize real-world objects and the ability to generate a plan that respects the laws of physics. This hybrid approach allows the robot to handle the messiness of a real kitchen, where a heavy pan cannot be lifted by its edge and a round mug cannot be balanced on a curved surface, without getting stuck or failing.

Ultimately, this work demonstrates that for robots to move from controlled laboratories to our homes, they must be equipped with a deep, symbolic understanding of how objects interact. It is not enough to know that a cup exists; the robot must understand that the cup has a flat bottom that can support a coaster, but a curved side that cannot. By building these physical constraints directly into the planning process, the researchers have created a system that is both fast and reliable. The experiments, conducted in a realistic simulation, show that this method can handle complex stacking tasks with a high degree of success, offering a promising path forward for robots that need to navigate the unpredictable and varied world of human environments.