Evaluating VLMs' Spatial Reasoning Over Robot Motion: A Step Towards Robot Planning with Motion Preferences

This paper evaluates the spatial reasoning capabilities of four state-of-the-art Vision-Language Models in interpreting robot motion preferences, demonstrating that Qwen2.5-VL achieves high accuracy in zero-shot and fine-tuned settings for enforcing constraints like object proximity and path style, thereby highlighting the potential of integrating VLMs into robot planning pipelines.

The Gist

Imagine you are teaching a robot butler to clean your living room. You don't just want it to pick up a cup; you want it to do so in a specific way. Maybe you want it to take a long, winding path to avoid a fragile vase, or perhaps you want it to sneak close to the sofa to grab a remote without knocking over the lamp.

For a long time, robots have been great at the "how" (moving from point A to point B) but terrible at the "how I want it" (the style, the mood, the specific constraints). They are like a very obedient dog that follows commands but doesn't understand the nuance of "don't step on the cat."

This paper asks a big question: Can modern AI "eyes and brains" (Vision-Language Models) understand these subtle human preferences just by looking at a picture of a robot's path?

Here is the breakdown of their experiment, explained with some everyday analogies.

1. The Setup: The "Robot Dance-Off"

The researchers didn't just ask the AI to guess. They set up a Robot Dance-Off.

• The Choreography: First, they used a standard robot planner to generate 50 different ways a robot could move from the kitchen to the living room. Some paths were straight, some were zig-zags, some went close to the window, and some stayed far away.
• The Visuals: They turned these 50 paths into a single image, like a map where every possible route is drawn in a different colored line (Red, Blue, Green, etc.).
• The Judge: They showed this image to four different "AI Judges" (advanced Vision-Language Models like Qwen2.5-VL and GPT-4o) and gave them a text instruction, like: "Pick the path that goes around the table but stays far away from the painting."

The goal was to see if the AI could look at the map, read the instruction, and point to the correct colored line.

2. The Four Ways to Ask the AI

The researchers tried four different ways to present the paths to the AI, similar to how you might show someone a menu:

1. The "One Big Map" (Single-Image): Show all 50 paths on one picture at once. The AI has to compare them all simultaneously.
2. The "One-by-One" (Multi-Image): Show the AI one path, ask "Is this good?", then show the next, and so on.
3. The "Descriptive Guide" (Visual Context): First, ask the AI to write a detailed description of the room and objects, then ask it to pick the path based on that description.
4. The "Video Gallery" (Screenshot Gallery): Show a strip of images, like a comic book, showing the robot moving along each path step-by-step.

The Result: The "One Big Map" approach was the clear winner.

• Why? Think of it like shopping for shoes. If you look at one shoe, then take it off and look at another, you might forget exactly how the first one felt. But if you lay all the shoes out on the floor, you can instantly compare them and pick the best one. The AI needs to see all the options side-by-side to make a fair comparison.

3. The Winners and Losers

• The Champion: Qwen2.5-VL (a specific AI model) was the best judge. It got about 71% to 75% of the answers right. It was particularly good at understanding spatial relationships, like "stay away from the lamp."
• The Runner-Up: GPT-4o (a very famous model) actually did worse than Qwen in this specific task. It's like having a very smart generalist who is great at many things but missed the specific details of this robot dance-off.
• The Struggle: The AI models were great at understanding proximity (e.g., "stay close to the wall") but struggled more with style (e.g., "make a zig-zag path"). It's easier for an AI to see where something is than to judge the shape of a line.

4. The "Fine-Tuning" Magic Trick

The researchers also tried a little magic trick called Fine-Tuning.
Imagine you have a smart student who knows a lot of general facts but hasn't studied for this specific test. They gave the AI a tiny "crash course" (just 98 examples) on how to pick robot paths.

• The Result: The AI's performance jumped significantly! One model improved its accuracy by over 60% after just a tiny bit of training.
• The Metaphor: It's like taking a brilliant chef who knows how to cook everything, giving them a recipe card for "Spicy Tacos," and suddenly they become a world-class taco chef.

5. The Cost of Thinking

The paper also looked at how much "brain power" (computing cost) this took.

• Showing all paths on one image was the most efficient. It used the least amount of data (tokens) but gave the best results.
• Trying to show the AI many separate images or video strips was like asking it to read a whole library to find one sentence—it was expensive and didn't work as well.

The Bottom Line

This paper shows that we are getting closer to a future where you can talk to your robot like a human. You won't just say "Go to the kitchen." You could say, "Go to the kitchen, but take a slow, winding path so you don't scare the cat, and keep a safe distance from the coffee table."

While the AI isn't perfect yet (it sometimes hallucinates or picks the wrong color), the fact that it can understand these complex spatial preferences just by looking at a map is a huge step forward. It suggests that in the near future, robots won't just be tools that follow orders; they will be assistants that understand our style.

Technical Summary

1. Problem Statement

While Vision-Language Models (VLMs) have shown promise in general robot task planning and human-robot interaction, their specific capability to perform spatial reasoning over robot motion remains under-explored. Specifically, it is unclear whether current foundation models can effectively interpret natural language instructions regarding:

• Object Proximity: Constraints on the distance between the robot and environmental objects (e.g., "stay away from the lamp," "pass between the table and sofa").
• Path Style: Preferences regarding the geometric or topological nature of the trajectory (e.g., "shortest path," "zigzag," "curved," "smooth").

The core challenge is determining if VLMs can act as a scoring mechanism to select the optimal trajectory from a set of candidates generated by traditional motion planners, based on these nuanced user preferences.

2. Methodology

A. Dataset Construction

The authors constructed a custom dataset of 558 language-constrained robot motion planning problems using the iGibson simulation environment (featuring realistic home scenes from the BEHAVIOR dataset).

• Task Types: 126 Navigation tasks (mobile base) and 432 Manipulation tasks (robotic arm).
• Preference Categories:
  • Object Proximity: Instructions to stay close to or far from specific objects.
  • Path Style: Instructions regarding path shape (straight, curved, zigzag, shortest).
• Ground Truth Generation:
  1. Candidate Generation: Used BiRRT (Bidirectional Rapidly-exploring Random Trees) and PRM (Probabilistic RoadMaps) with varying random seeds and cost functions to generate 50 diverse trajectories per problem.
  2. Clustering: Applied K-means clustering on trajectory waypoints to group similar paths.
  3. Selection: Selected the path closest to each cluster centroid to represent diverse geometric/topological options.
  4. Annotation: Manually annotated each path with a unique description to create the ground-truth label for the VLM evaluation.

B. VLM-Based Path Selection Framework

The proposed pipeline involves generating diverse paths, visualizing them, and using a VLM to score/select the best match for a text instruction. The authors evaluated four distinct querying methods:

1. Single-Image Trajectory: All candidate paths are visualized as colored dotted trails in a single image. The VLM scores all paths simultaneously.
2. Multi-Image Trajectory Trail: Each candidate path is presented in a separate image. The VLM scores them individually, and the highest score is selected.
3. Single-Image with Visual Context: The VLM is first asked to generate a structured textual description of the image (objects, relationships) before scoring the paths.
4. Screenshot Gallery: A sequence of screenshots showing the robot executing each path is presented (row-by-row), and the VLM selects the correct row.

C. Models Evaluated

• Qwen2.5-VL-72B: A large model with strong visual localization capabilities.
• GPT-4o: A state-of-the-art multimodal model.
• LLaVa1.5-7B: A conversational VLM.
• Qwen2.5-VL-7B & 3B: Smaller variants for fine-tuning experiments.

3. Key Contributions

1. Benchmarking Spatial Reasoning: Established a rigorous evaluation framework for VLMs on robot motion planning, specifically focusing on path style and object proximity constraints, which are distinct from standard task planning.
2. Query Method Analysis: Systematically compared four different image querying strategies, identifying that Single-Image Trajectory (showing all paths at once) is superior to multi-image or sequential methods.
3. Fine-Tuning Efficacy: Demonstrated that Supervised Fine-Tuning (SFT) on a small dataset (98 examples) significantly boosts the performance of smaller models (LLaVa1.5-7B and Qwen2.5-VL-7B), proving that these architectures can adapt to motion preference instructions with minimal data.
4. Cost-Accuracy Trade-off: Analyzed the relationship between token usage (computation cost) and accuracy, showing a linear correlation where reducing image resolution (token count) degrades performance.

4. Results

A. Accuracy by Query Method

• Best Performance: The Single-Image Trajectory method yielded the highest accuracy.
  • Qwen2.5-VL-72B: Achieved 71.4% zero-shot accuracy overall.
  • GPT-4o: Performed lower than Qwen2.5-VL.
  • Multi-Image & Gallery Methods: Performed poorly. The authors hypothesize that multi-image methods fail because the VLM cannot maintain a consistent relative comparison across separate inference requests.
  • Visual Context: Adding an intermediate step to generate image descriptions did not improve accuracy and likely introduced token redundancy.

B. Accuracy by Task Type

• Object Proximity vs. Path Style: All models performed better on Object Proximity tasks (e.g., "stay away from the window") than on Path Style tasks (e.g., "take a zigzag path").
  • Qwen2.5-VL-72B: 74.4% on Proximity vs. 63.9% on Path Style (Navigation).
  • Manipulation Tasks: Overall success rate was lower (65.5%) compared to navigation (71.4%).
• Model Comparison: Qwen2.5-VL generally outperformed GPT-4o, attributed to its superior object grounding capabilities.

C. Impact of Fine-Tuning

• Fine-tuning on a small dataset (98 examples) resulted in significant gains:
  • Qwen2.5-VL-7B: Accuracy increased by >20%.
  • LLaVa1.5-7B: Accuracy increased by >60%.
• This indicates that even smaller models can learn to interpret motion constraints effectively with minimal data.

D. Computation Cost

• The Single-Image method was the most token-efficient (avg. ~687 tokens for Qwen2.5-VL).
• Accuracy scales approximately linearly with the number of tokens (image resolution), suggesting a direct trade-off between computational cost and reasoning quality.

5. Significance and Limitations

Significance:
This work bridges the gap between high-level semantic instruction and low-level motion planning. It validates that VLMs can serve as effective "judges" in a planning loop, selecting trajectories that satisfy human preferences regarding safety (proximity) and style, which traditional planners often ignore. The findings suggest that integrating VLMs into motion pipelines is a viable path for more intuitive human-robot interaction.

Limitations & Failure Modes:

• Optimization Blindness: VLMs struggle to identify mathematically optimal paths (e.g., "shortest path") compared to classical planners like RRT*.
• Hallucinations: Models occasionally selected paths that did not exist in the image (e.g., choosing a "red" path when no red path was present).
• Complexity: Performance drops on complex path-style descriptions compared to simple proximity constraints.

Future Work:
The authors aim to integrate these capabilities robustly into real-world robot pipelines, potentially requiring improved accuracy or human-in-the-loop interfaces to handle edge cases.