From Vision to Harvest: Benchmarking Vision-Language Models for Multi-Arm Robotic Fruit Harvesting

Technical Summary: From Vision to Harvest

Problem Definition
The paper addresses the challenge of deploying multi-arm robotic systems for fruit harvesting in unstructured orchard environments. While multi-arm systems offer higher throughput than single-arm counterparts by picking concurrently, they face two primary hurdles:

1. Generalization: Traditional systems rely on specialized perception pipelines (e.g., YOLO) that degrade under varying lighting, occlusion, and environmental conditions, often requiring extensive retraining on target data.
2. Coordination Complexity: Planning collision-free trajectories for multiple arms in a shared workspace is an NP-hard problem. Existing solutions often simplify this by partitioning the workspace among arms, which can reduce planning quality and overall throughput.

The authors propose evaluating Vision-Language Models (VLMs) as a zero-shot alternative. The motivation is that human workers successfully harvest by visually reasoning about spatial relationships and sequencing tasks without explicit retraining. The paper investigates whether pretrained VLMs can replicate this high-level reasoning to generate effective, collision-free multi-arm harvesting plans directly from raw RGB-D images.

Methodology
The authors introduce a comprehensive benchmark to evaluate VLMs against a traditional "oracle" pipeline.

• Oracle Pipeline (Baseline): A state-of-the-art, three-stage pipeline serving as an upper-bound reference:

  1. Perception: Uses GroundingDINO for open-vocabulary detection and SAM2 for pixel-level segmentation to isolate fruits.
  2. Localization: Projects segmented depth pixels into 3D coordinates using a pinhole camera model and clusters them via DBSCAN to estimate fruit positions and detach times.
  3. Planning: Utilizes a bi-level mixed-integer programming (MIP) framework to assign fruits to specific arms and generate synchronized, collision-free trajectories.

• Zero-Shot VLM Pipeline:

  • Input: Raw orchard RGB-D images and a structured prompt.
  • Prompt Design: The prompt provides the VLM with a specific role (agricultural robotics expert), physical scale references (e.g., fruit diameter), coordinate system definitions, and robot constraints (e.g., arm ordering on a shared rail). It explicitly asks the model to identify fruits, reason about spatial relationships, and output a JSON object containing harvesting sequences and 3D waypoints in meters.
  • Safety Verification: Since VLMs output waypoints without timing information, a lightweight trajectory verifier checks for "order violations." If an arm assigned to a fruit with a larger X-coordinate must pass an arm assigned to a fruit with a smaller X-coordinate (violating the fixed physical order of arms on the rail), the plan is flagged as a collision.

• Experimental Setup:

  • Datasets: Real-world images from apple and citrus orchards.
  • Models: Evaluated five closed-source (e.g., GPT-4o, GPT-5.5-Pro, Claude Sonnet 3.5) and two open-source VLMs.
  • Metrics: Detection ratio, harvested ratio (at varying spatial thresholds: 1.0m, 0.5m, 0.25m), collision ratio, trajectory distance, and token usage.

Key Contributions

1. First Comprehensive Benchmark: The paper presents the first benchmark specifically designed to evaluate pretrained VLMs on zero-shot multi-arm fruit harvesting planning across real-world crops (apple and citrus).
2. VLM Planning Pipeline: Development of a pipeline that generates multi-arm waypoints directly from raw images, coupled with a collision-checking mechanism that validates feasibility based on kinematic constraints.
3. Empirical Analysis: A systematic evaluation revealing that while frontier VLMs can generate plans, their performance is highly sensitive to physical scale priors, spatial thresholds, and scene complexity.
4. Open Source: The authors commit to open-sourcing the benchmark to facilitate future research.

Results

• Capability: Frontier VLMs (e.g., GPT-5.5-Pro, Claude Opus 4.7) can generate complete harvesting plans that cover most fruits in zero-shot scenarios.
• Precision vs. Recall: There is a significant gap between detecting fruit and precisely localizing them. Models often achieve high detection ratios (e.g., >90% at 1.0m threshold) but suffer drastic drops in harvested ratios at stricter thresholds (e.g., <10% at 0.25m for some models on citrus).
• Safety and Coordination: Collision ratios are substantial for many models (e.g., >80% for Claude Sonnet 3.5 on citrus), indicating that VLMs struggle with the complex spatial coordination required for multi-arm systems without explicit geometric constraints.
• Prompt Sensitivity: Ablation studies show that physical scale priors are critical; removing them causes a sharp decline in localization accuracy. However, structured output (JSON) is essential; without it, models fail to produce executable plans despite detecting fruit.
• Scalability: Performance degrades as the number of fruits increases (higher scene complexity) and as the number of arms increases, highlighting the difficulty of scaling coordination logic.

Significance and Claims
The paper claims that this work highlights both the promise and current limitations of VLMs in agricultural robotics.

• Promise: VLMs demonstrate the ability to generalize across crops and environments without task-specific training, offering a potential path to reduce reliance on manual labor and specialized data collection.
• Limitations: Practical deployment is currently limited by the inability of VLMs to generate accurate 3D waypoints (particularly in depth) and to perform collision-aware coordination for multiple arms.
• Conclusion: The authors conclude that while VLMs are effective for high-level task planning, they are not yet a complete solution for multi-arm harvesting. Future work requires addressing the gap between semantic reasoning and precise metric localization and safety verification. The benchmark serves as a foundation for developing more generalizable and efficient planning systems.