Zero-shot 2D Grounding with Novel Affordance Types

Imagine you are teaching a robot how to interact with the world. You can tell the robot what a "knife" looks like, and it can find one in a picture. But knowing what a knife is isn't the same as knowing what you can do with it. This is the difference between object recognition and "affordance." Affordance is a fancy word for the hidden clues an object gives us about how to use it. A handle on a mug "affords" grasping; the flat side of a knife "affords" cutting; the peel of an orange "affords" peeling. For robots to be truly helpful in our messy, real-world kitchens and living rooms, they need to understand these actions, not just the objects.

The big challenge has been that most robot brains are trained like students who only study for a specific test. If a robot learns how to "cut" an apple, it might get confused when asked to "cut" a tomato, or it might not know how to "peel" a banana if it's never seen that specific action before. It's like a student who memorizes the answers to a math worksheet but freezes when the teacher asks a similar question with different numbers. Scientists have been trying to build robots that can handle these new, unseen instructions, but until now, the focus has mostly been on teaching them about new objects, not new actions.

This paper introduces a new way to teach robots to handle "novel affordance types"—basically, new ways to use things they've never been explicitly taught. The researchers, Haomeng Zhang and Raymond A. Yeh from Purdue University, noticed that existing AI models are great at finding new objects but terrible at figuring out new actions. They created a new set of challenges (called benchmarks) to test this specific skill. They found that the current state-of-the-art models, which are very smart at recognizing objects, completely fail when asked to find the right spot to "peel" or "sit on" something if those specific actions weren't in their training data.

To fix this, the team built a new system called AffordAnything. Think of it as a detective that doesn't just look at the whole picture but zooms in on tiny clues. The system realizes that where you interact with an object is usually a specific part of it (like the handle of a cup or the edge of a book). Instead of trying to memorize every possible action, this system uses a powerful, pre-trained AI (a Visual Language Model) to break the object down into small patches and figure out which patch matches the action word. For example, if you ask it to "cut," it looks for the sharp edge; if you ask it to "hold," it looks for the handle.

The researchers tested two versions. The first, AffordAnything, is a "training-free" version, meaning it works right out of the box without needing to be re-taught. It already performed significantly better than existing methods. The second version, AffordAnything+, is a "trainable" upgrade that learns how to mix these clues together even better with just a tiny bit of practice. On their new test sets, this upgraded model improved the accuracy of finding the right spot by 12.3% (measured as an absolute increase in IoU@0.4) compared to the best previous method.

The paper explicitly argues against the idea that simply using bigger, smarter AI models or just recognizing objects is enough. They showed that even the most advanced models fail when the action is new. They also demonstrated that while these models can recognize a "knife," they don't automatically know how to "peel" it unless they are specifically designed to look for the relationship between the action and the object's shape. The results suggest that to make robots truly adaptable, we need to stop teaching them a fixed list of actions and start teaching them how to reason about the parts of an object that make an action possible. The authors hope this new way of testing and building these systems will help robots become more flexible and ready for the real world, where instructions are always changing.