💻 computer science

Vision-guided Autonomous Dual-arm Extraction Robot for Bell Pepper Harvesting

This paper introduces VADER, a vision-guided autonomous dual-arm mobile robot designed to harvest bell peppers in unstructured outdoor environments, featuring a robust perception pipeline trained on a new hierarchical dataset and achieving a harvest success rate exceeding 60% with a cycle time under 100 seconds per fruit.

Original authors: Kshitij Madhav Bhat, Tom Gao, Abhishek Mathur, Rohit Satishkumar, Francisco Yandun, Dominik Bauer, Nancy Pollard

Published 2026-03-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kshitij Madhav Bhat, Tom Gao, Abhishek Mathur, Rohit Satishkumar, Francisco Yandun, Dominik Bauer, Nancy Pollard

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a busy farm where the workers are running out of time and money to pick bell peppers by hand. Picking peppers is tricky; you have to hold the fruit steady with one hand while snipping the stem with the other, all while avoiding leaves that block your view. If you squeeze too hard, you bruise the pepper. If you miss the stem, you rip the plant.

Enter VADER (Vision-guided Autonomous Dual-arm Extraction Robot). Think of VADER not as a scary sci-fi villain, but as a highly skilled, two-armed robot gardener designed to do the heavy lifting so humans don't have to.

Here is how VADER works, broken down into simple concepts:

1. The Setup: A Mobile Kitchen Cart

VADER rides on a rugged, all-terrain vehicle (like a small, smart golf cart) that drives down the rows of pepper plants. Mounted on this cart are two long, flexible robot arms (like human arms with seven joints each).

The Left Arm has a soft, 3D-printed "hand" designed to gently grab the pepper without squishing it.
The Right Arm has a specialized "scissor" tool designed to slice the stem.
The Eyes: Each arm has its own camera, giving the robot a 3D view, much like how humans use two eyes to judge depth.

2. The Brain: Seeing Through the Leaves

The biggest challenge in a real field is that leaves hide the peppers. It's like trying to find a specific red ball in a pile of green leaves.

The "Coarse-to-Fine" Strategy: VADER's brain first takes a quick, blurry look to find where a pepper might be (Coarse). Then, it zooms in with its cameras to get a super-clear picture of the pepper and its stem (Fine).
The 3D Model: Once it sees the pepper, the robot doesn't just see a flat image. It builds a 3D mathematical model of the pepper (imagine wrapping the pepper in a perfect, invisible egg-shaped shell) to figure out exactly how to hold it and where to cut.

3. The Dance: A Coordinated Tango

This is where VADER shines compared to older robots that only have one arm.

The Problem with One Arm: If a robot has only one arm, it has to hold the pepper and cut the stem at the same time. It's like trying to eat soup with a fork while holding the bowl with the same hand. It's clumsy and risky.
The VADER Solution: VADER splits the job. One arm acts as the Anchor, holding the pepper steady. The other arm acts as the Chef, snipping the stem. Because they work together, the pepper doesn't wobble, and the plant doesn't get shaken.
The "Collision Wall": To make sure the two arms don't bump into each other, the robot's brain draws an invisible line down the middle of its workspace. It tells the left arm, "You stay on the left," and the right arm, "You stay on the right." This lets them work at the same time (parallel) instead of waiting for each other, saving about 20 seconds per pepper.

4. The Safety Net: The "Human-in-the-Loop"

Even the best robots get stuck. Maybe a leaf is too thick, or the stem is in a weird spot.

The GELLO Console: If VADER gets confused, it doesn't just crash. It has a "panic button" that switches to Teleoperation. A human operator can pick up a special controller (which looks like a mini-version of the robot arms) and manually guide the robot to finish the job. It's like a pilot taking over the controls of a plane when the autopilot gets confused.

5. The Results: Getting Better Every Day

The team tested VADER in a fake farm (a testbed) and found:

Success Rate: It successfully picked about 6 out of 10 peppers on the first try. (This is actually quite good for a robot in a messy, real-world environment!).
Speed: It takes about 1 minute to find, grab, cut, and store one pepper.
The Dataset: The team also created a massive photo album of over 3,000 pictures of peppers, showing them in different lights and angles, so other robots can learn from their mistakes.

The Big Picture

Think of VADER as a bridge. Right now, most robot farmers only work in perfect, clean greenhouses. VADER is designed to handle the messy, unpredictable reality of outdoor farms. By splitting the job between two arms and having a human ready to help if things go wrong, it proves that robots can eventually do the hard, repetitive work of farming, leaving humans to manage the farm and the strategy.

In short: VADER is a two-armed robot gardener that uses two eyes to find hidden peppers, two hands to hold and cut them safely, and a human backup plan to ensure nothing gets dropped.

1. Problem Statement

The agricultural sector faces a critical labor shortage and rising costs, with bell pepper harvesting alone accounting for up to 50% of total production costs. While automation exists for controlled greenhouse environments, harvesting in unstructured outdoor fields remains a significant challenge due to:

Environmental Variability: Uncontrolled lighting, wind, and terrain.
Occlusion: Heavy leaf coverage obscuring the fruit and the peduncle (stem).
Dexterity Requirements: The need to grasp irregularly shaped fruit without damage while simultaneously cutting the peduncle.
Limitations of Single-Arm Systems: Existing solutions often use a single arm to both hold and cut, requiring complex end-effectors. These systems struggle with occlusion management and often fail to localize the peduncle precisely, leading to missed cuts or plant damage.
Data Gaps: Existing perception datasets lack high-fidelity, human-verified annotations for both the fruit and the peduncle, which are crucial for precise pose estimation and cutting.

2. Methodology: The VADER System

The authors propose VADER (Vision-guided Autonomous Dual-arm Extraction Robot), a mobile manipulation system designed to mimic human bimanual dexterity by decoupling the grasping and cutting tasks.

A. Hardware Architecture

Mobile Base: A Clearpath Warthog UGV (differential drive) equipped with a storage bin between the arms.
Manipulators: Two UFactory XArm7 (7-DOF) robotic arms mounted on the side of the base, spaced 50 cm apart to maximize shared workspace.
End-Effectors:
- Gripper Arm: A tendon-actuated, 3D-printed TPU soft gripper designed to conform to irregular fruit geometries without force sensing.
- Cutter Arm: A single-motor cutter designed to exert shear force for peduncle severance.
Perception: Each arm is equipped with a palm-mounted Intel RealSense D405 RGB-D camera, providing dual-view redundancy to mitigate occlusion.
Compute: NVIDIA Orin AGX for onboard processing.

B. Perception Pipeline

The system employs a coarse-to-fine strategy using a hierarchical dataset:

Instance Segmentation: Two fine-tuned YOLOv11 models detect the pepper fruit and the peduncle separately.
Coarse Pose Estimation: A 6-DOF pose is estimated based on the geometric relationship between the fruit centroid and the peduncle centroid.
Pose Refinement: A Superellipsoid fitting algorithm is applied to the partial 3D point cloud of the fruit. This regularizes the shape (accounting for the bell pepper's cubic-to-spherical geometry) and refines the position and orientation.
Final Pose: The refined fruit center is used to reconstruct the orthonormal frame, determining the final cutting axis.

C. Task and Motion Planning

Grasp Planning: A parametric grasp circle is generated orthogonal to the stem axis. The system samples candidate poses, performs inverse kinematics (IK), and selects the first collision-free configuration closest to the current state.
Dual-Arm Coordination:
- Shared Workspace Division: A virtual "collision wall" is dynamically inserted into the planning scene, splitting the workspace based on the target pepper's horizontal position. This allows the two arms to plan and execute in parallel without reasoning about the other arm's dynamic movement.
- Parallel Execution: Planning for the second arm overlaps with the execution of the first, saving ~20 seconds per cycle compared to sequential planning.
- Optimization: A custom OMPL objective function balances joint-space path length with task-space end-effector distance to prevent inefficient winding paths.
State Machine: Orchestrates the workflow: Home $\to$ Pre-grasp (fine pose update) $\to$ Grasp/Cut $\to$ Storage.

D. Teleoperation & Safety

To handle failure modes (e.g., heavy occlusion), the system integrates GELLO-based teleoperation.

Hardware: 3D-printed kinematic replicas of the XArm7s with Dynamixel motors acting as haptic interfaces (not active drives).
Performance: Operates at 100 Hz control frequency with <30 ms latency, allowing human operators to intervene and recover from failures seamlessly.

3. Key Contributions

Dual-Arm Autonomous Framework: A unified system integrating two 7-DOF arms for coordinated grasp-and-cut operations, specifically addressing the limitations of single-arm outdoor harvesting.
Hierarchical Dataset: A new dataset of 3,258 images spanning indoor and outdoor domains. It features both coarse (fruit-only) and fine (fruit + peduncle) annotations, enabling multi-stage training for robust perception.
Robust Perception & Planning: A novel pipeline combining YOLOv11 segmentation with superellipsoid fitting for precise peduncle localization, coupled with a parallelized dual-arm motion planning strategy using a dynamic collision wall.
Teleoperation Fail-Safe: Integration of a high-fidelity, low-latency teleoperation system for human-in-the-loop recovery, ensuring operational continuity in unstructured environments.

4. Experimental Results

Perception:
- Pepper Detection: F1-score of 0.942, mAP@50 of 0.97.
- Peduncle Detection: F1-score of 0.832, mAP@50 of 0.884.
- Inference speed is under 25 ms per frame (60 Hz capability).
Simulation:
- Achieved a 92.3% harvest success rate.
- Average cycle time: 69.25 seconds.
- Identified that targets must be within 10 cm of the robot's center horizontally to maximize reachability.
Real-World (Indoor Testbed):
- Success Rate: 61% (30 successful harvests out of 47 trials).
- Cycle Time: Average of 69.25 seconds per fruit.
- Failure Analysis: The primary failure mode (13/17) was cutter offset error (missing the peduncle vertically), attributed to the difficulty of calibrating suspended peppers in the mockup. Other failures included storage timeouts and grasp failures.

5. Significance and Future Work

Significance: VADER bridges the gap between greenhouse automation and practical outdoor deployment. By decoupling grasping and cutting, it reduces mechanical complexity and improves reliability. The contribution of a dual-granularity dataset addresses a critical gap in agricultural robotics data, specifically for peduncle localization.
Future Directions:
1. Integrating autonomous navigation for row traversal.
2. Extensive outdoor field testing to validate robustness against diverse weather and lighting.
3. Developing active perception strategies to handle partial occlusion more effectively.

This work demonstrates that a dual-arm approach, supported by robust perception and human oversight, is a viable path toward solving the labor crisis in open-field bell pepper harvesting.