💻 computer science

Assessing VLM-Driven Semantic-Affordance Inference for Non-Humanoid Robot Morphologies

This paper investigates the capability of Vision-Language Models (VLMs) to infer semantic affordances for non-humanoid robots using a novel hybrid dataset, revealing that while VLMs generalize well across morphologies, they exhibit a consistent tendency toward conservative predictions characterized by low false positive rates but high false negative rates, particularly in novel tool-use scenarios.

Original authors: Jess Jones, Raul Santos-Rodriguez, Sabine Hauert

Published 2026-04-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jess Jones, Raul Santos-Rodriguez, Sabine Hauert

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a team of robots trying to clean up a messy room. Some look like humans, but others look like strange, round vacuum cleaners, or boxy carts, or even rolling bins. To work together, these robots need to know not just what things are (a cup, a ball, a screwdriver), but what they can do with them based on their own unique bodies.

This is where the concept of "Affordance" comes in. Think of affordance as the "menu of actions" an object offers. A chair affords "sitting" to a human, but to a rolling robot, it might just afford "bumping into."

The paper by Jess Jones and her team asks a big question: Can modern "smart" AI models (called Vision-Language Models or VLMs) figure out this menu for robots that don't look like humans?

Here is the breakdown of their findings, using some everyday analogies:

1. The "Human-Centric" Bias (The Problem)

Imagine you hire a chef who has only ever cooked for humans. If you ask them, "What can I do with this rock?" they might say, "You can't eat it, so it's useless." They might miss that you could use the rock to prop open a door or build a wall.

The researchers found that these AI models are like that chef. They are trained on billions of images of humans interacting with objects.

The Result: When asked what a weird, round robot can do with a ball, the AI often ignores the robot's specific shape. Instead, it defaults to human actions like "throw" or "squeeze," even if the robot can't do those things.
The Consequence: The AI becomes overly conservative. It's like a cautious parent who says, "No, you can't touch that," even when the robot could safely push it. The AI misses valid opportunities (False Negatives) because it's scared of making a mistake, but it rarely suggests something dangerous (Low False Positives).

2. The Experiment (The Test Kitchen)

The team created a special "test kitchen" with two types of ingredients:

Real Data: Videos of actual robots (called DOTS) moving around real objects in a warehouse.
Synthetic Data: Computer-generated videos of robots in weird scenarios.

They tested three different "super-brains" (GPT, Gemini, and Claude) by giving them a simple description of a robot's body. For example: "I am a round robot. I can scoop things up if they are light, but I can't lift heavy boxes."

3. The Findings (The Taste Test)

The "Human" Robot: When the AI looked at a robot that looked like a human, it was pretty good at guessing "Pick up" (grasping). But for things like "Push," it got confused and kept suggesting human actions that didn't fit.
The "Weird" Robots: When the AI was told about the round, scooping robots, it actually did better at guessing "Push" or "Scoop." Why? Because the researchers gave it a clear, physical description. It was like telling the chef, "You have a spoon, not a fork," which helped them stop guessing and start thinking logically.
The "Conservative" Glitch: Even with the descriptions, the AI still played it safe. If it wasn't 100% sure, it would say, "I don't know what to do with this," rather than guessing. This is good for safety (the robot won't break things), but bad for efficiency (the robot sits there doing nothing when it could be working).

4. Where the AI Got Stuck (The Blind Spots)

The researchers found two major "blind spots" in the AI's brain:

Spatial Confusion: They asked the AI about a robot that could only lift things above its head. The AI still said, "You can lift that rock on the floor!" It failed to understand the physical rules of the robot's body.
Material Confusion: They told the AI, "This robot can only cut paper and wood." The AI still suggested the robot could "cut" a golf ball or a screwdriver. It didn't really understand what materials are made of; it just guessed based on patterns.

5. The Big Takeaway

The paper concludes that AI is a powerful tool, but it needs a translator.

Right now, these models are like a brilliant librarian who knows every book in the world but has never seen a robot. If you just ask them, "What can this robot do?", they will guess based on what humans do.

However, if you give them a clear, physical description of the robot (like a recipe card), they can start to make much better guesses. The challenge is that they are still too cautious. They would rather do nothing than risk doing the wrong thing.

In the future: To make these robots truly helpful, we need to teach the AI to be less afraid of guessing wrong, or to give it better "rules of the road" (like task descriptions) so it knows exactly what it's supposed to be doing. This will help a team of diverse robots—some round, some tall, some wheeled—work together to clean our houses, build our cities, and explore our world.

1. Problem Statement

The paper addresses a critical gap in the deployment of Vision-Language Models (VLMs) for robotics: the assumption that VLMs, trained predominantly on human-centric data, can effectively infer affordances (actionable possibilities) for robots with non-humanoid morphologies.

The Challenge: While VLMs excel at understanding human-object interactions, their application to diverse robotic forms (e.g., cylindrical scoops, specialized cutters) remains unexplored.
The Risk: Directly applying human-centric VLMs to non-humanoid robots may lead to:
- Over-generalization: Suggesting actions that are physically impossible for the robot (e.g., "grasping" with a non-dexterous end-effector).
- Conservative Bias: Failing to recognize valid interactions because they deviate from human norms.
- Safety vs. Utility Trade-off: A tendency to avoid false positives (unsafe actions) at the cost of high false negatives (missing valid opportunities).

2. Methodology

A. Experimental Pipeline

The authors propose a Semantic-Affordance Inference Pipeline that integrates VLMs with robotic perception:

Prompt Engineering: The VLM is provided with a natural language description of the robot's physical properties (dimensions, movement capabilities, and specific mechanisms like "scooping" or "cutting").
Visual Input: On-board cameras capture frames (sampled every second).
VLM Inference: The VLM identifies objects in the frame and predicts affordances based strictly on the provided robot description.
Localization: Detected objects are passed to GroundingDINO (a zero-shot object detector) to generate 2D bounding boxes. Combined with robot odometry, these are triangulated into 3D world coordinates.
Semantic Association: To handle VLM stochasticity (e.g., labeling an object as "cup" in one frame and "mug" in another), the system uses sentence transformer embeddings (SBERT) to merge detections of the same object within a spatial threshold ( $d < 0.1$ L2 norm).

B. Dataset Construction

The authors created a novel hybrid dataset comprising 774 labeled object instances across 100 unique object categories (e.g., food, construction materials, sports equipment).

Sources:
- Real-World: Captured using DOTS robots in controlled indoor arenas mimicking household, construction, and logistics environments.
- Synthetic: Generated using Google's Veo3 model to provide diverse non-humanoid perspectives.
Robot Morphologies: The study evaluates six non-humanoid capabilities (Scoop, Push, Lift, Cut, Pick, Collect) against a Humanoid Baseline (limited to Push, Pick, Lift for comparison).
Models Tested: Three leading VLMs (as of Sept 2025): GPT-5, Gemini-2.5-pro, and Claude-opus-4-1.

C. Evaluation Metrics

Performance is measured using F1 scores derived from a confusion matrix of:

True Positives (TP): Correctly identified affordance-object pairs.
False Positives (FP): Incorrectly predicted affordances (safety risk).
False Negatives (FN): Missed valid affordances (utility loss).
Semantic Similarity: A threshold ( $\tau = 0.45$ ) is used to match VLM labels with ground truth labels (e.g., matching "grasp" with "pick").

3. Key Contributions

First Empirical Analysis of Non-Humanoid Affordances: The paper provides the first systematic evaluation of how VLMs generalize affordance inference to robots with fundamentally different embodiments than humans.
Novel Hybrid Dataset: Introduction of a curated dataset combining real-world and synthetic data specifically designed for non-humanoid affordance characterization.
Identification of Systematic Biases: The study quantifies a specific bias pattern: Low False Positives but High False Negatives. VLMs tend to be "conservative," preferring to predict no action rather than an incorrect one.
Pipeline Architecture: A robust framework for grounding VLM semantic outputs into 3D spatial maps using multi-view localization and semantic association.

4. Key Results

A. Performance Trends

Humanoid Baseline: VLMs performed moderately well on "Pick" (grasping) but struggled with "Push." They frequently defaulted to out-of-scope human actions (e.g., suggesting "throwing" a ball instead of "pushing" it), leading to high false negative rates for the target task.
Non-Humanoid Robots:
- Success: Models performed significantly better on affordances explicitly constrained by the robot's physical description (e.g., "Scoop" and "Push"). Providing a concise physical description successfully grounded the reasoning, reducing irrelevant human-centric suggestions.
- Failure: Performance dropped significantly for complex spatial or material constraints, specifically "Lift" and "Cut."

B. Critical Weaknesses Identified

Poor Spatial Reasoning: For the "Lift" capability (defined as lifting objects above the chassis), models like Claude and GPT incorrectly identified objects on the ground as liftable, failing to apply spatial constraints.
Lack of Material Understanding: For the "Cut" capability (restricted to paper, plastic, wood), models frequently suggested cutting inappropriate materials like golf balls or screwdrivers.
Domain Bias: Performance was highest for familiar categories (Food, Sports Equipment) and lowest for Construction Items, confirming that VLM capabilities are limited by the distribution of human-centric training data.
Conservative Bias: Across all non-humanoid morphologies, the dominant error mode was False Negatives. The models tended to omit valid interactions rather than risk an incorrect prediction.

C. Model Comparison

Gemini-2.5-pro showed the strongest overall results for non-humanoid robots (Mean F1: 0.5).
Claude showed a dramatic drop in "Pick" inference capabilities for non-humanoid forms, suggesting lower embodiment generalization.
GPT-5 performed well on human baselines but struggled with spatial constraints in non-humanoid scenarios.

5. Significance and Implications

Safety vs. Utility: The "conservative bias" (low FP, high FN) is beneficial for safety (preventing dangerous actions) but detrimental to utility (robots failing to utilize their full capabilities). The paper argues that effective integration requires complementary approaches to mitigate this over-conservatism without sacrificing safety.
Task-Conditioned Prompting: The authors suggest that adding high-level task descriptions (e.g., "tidy the room") to the prompt could further constrain VLMs and reduce irrelevant suggestions.
Future Directions: The work highlights the need for:
- Task-conditioned prompting to reduce out-of-scope actions.
- Local, fine-tuned models to address inference latency (current API calls take 3–10 seconds).
- Community Benchmarks: A large-scale, public dataset for non-humanoid affordance inference to standardize evaluation.

In conclusion, while VLMs show promise for grounding robot interactions in semantic descriptions, they currently lack the spatial and material reasoning required for complex, non-humanoid manipulation. Their utility in real-world multi-robot systems depends on overcoming their inherent human-centric biases and conservative inference patterns.