💻 computer science

Robotic Strawberry Harvesting with Robust Vision and Deep Reinforcement Learning based Sim-to-Real Control

This paper presents a closed-loop robotic strawberry harvesting system that integrates a robust HRAttnEdge-YOLO26-seg vision model with simulation-trained deep reinforcement learning control to achieve high success rates in complex agricultural environments while reducing hardware dependency and development costs.

Original authors: Al Bashir, Shao-Yang Chang, Partho Ghose, Prem Raj, Chen-Kang Huang, Azlan Zahid

Published 2026-05-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Al Bashir, Shao-Yang Chang, Partho Ghose, Prem Raj, Chen-Kang Huang, Azlan Zahid

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a robot trying to pick a strawberry in a crowded garden. The leaves are everywhere, the berries are hiding behind stems, and the robot needs to be gentle enough not to crush the fruit but strong enough to pull it off the vine. Doing this automatically is incredibly hard because the robot has to "see" the berry clearly and then "move" its arm smoothly without bumping into anything.

This paper describes a new robot system designed to solve this problem using two main tricks: a super-sharp pair of "glasses" (vision) and a "muscle memory" brain (control) that was trained in a video game before ever touching a real robot.

Here is how the system works, broken down into simple parts:

1. The "Super-Glasses": Seeing Through the Mess

The Problem: In a real greenhouse, strawberries are often hidden behind leaves or clustered together. Standard computer vision is like looking at a messy pile of laundry; it might see a red blob, but it can't tell exactly where the berry ends and the leaf begins. If the robot guesses the edge wrong, it might grab a leaf instead of the fruit.

The Solution: The researchers built a new type of camera software called HRAttnEdge-YOLO26-seg.

The Analogy: Think of standard vision software as a painter using a thick, blunt brush. It gets the general shape of the strawberry but leaves fuzzy edges. This new software is like a master artist using a fine-tipped pen. It has three special tools:
1. High-Resolution Lens: It keeps a "zoomed-in" view of tiny details (like the stem and the edge of the berry) that other systems usually throw away.
2. Focus Filter: It ignores the background noise (leaves and shadows) and focuses only on the parts of the image that look like fruit.
3. Edge Detective: It specifically looks for the sharp lines where the fruit ends, ensuring the robot knows exactly where to grab.
The Result: When tested, this system was much better at drawing the perfect outline around a strawberry, even when it was partially hidden, compared to older methods.

2. The "Video Game Training": Learning to Move Without Breaking Things

The Problem: Teaching a robot arm to move smoothly usually involves "trial and error." You tell the robot to move, it crashes, you fix it, you try again. This is expensive, slow, and risks breaking the robot or the delicate strawberries.

The Solution: The researchers used Deep Reinforcement Learning (DRL), specifically a method called PPO.

The Analogy: Instead of teaching the robot in the real greenhouse, they put it in a virtual video game (Isaac Lab). In this game, they created thousands of fake greenhouses with fake strawberries. The robot played the game millions of times, trying to reach the target. Every time it moved smoothly, it got a "point." Every time it jerked or crashed, it lost points.
The Transfer: Once the robot became a master at the game, they took that "brain" (the policy) and uploaded it to the real robot. Because it had already learned the best way to move in the simulation, it could immediately perform smooth, fluid motions in the real world without needing to crash into anything first.
The Comparison: They tested this against a traditional method (Inverse Kinematics), which is like trying to solve a complex math equation in real-time to move the arm. The traditional method was jerky and often failed. The "video game trained" robot moved like a human dancer—smooth and predictable.

3. The "Assembly Line": Putting It All Together

The team connected these two parts using a standard robot language (ROS).

See: The camera spots a strawberry and draws a perfect outline around it.
Calculate: It finds the center of that outline and figures out exactly where it is in 3D space.
Move: The "video game trained" brain tells the robot arm how to move its joints to reach that spot.
Grab: A soft gripper (like a gentle hand) grabs the berry and pulls it off the vine.
Repeat: The robot drops the berry in a basket and goes to the next one.

The Real-World Test

They took this system into a real greenhouse in Texas.

The Scorecard: The robot successfully reached for the strawberries 96.6% of the time. It successfully grabbed and pulled them 91.3% of the time. Overall, it successfully harvested 84.3% of the attempts.
The Comparison: This was much better than using the old "math equation" method, which was too shaky and unreliable for the messy greenhouse environment.

What They Didn't Do (Limitations)

The paper is very clear about what this system doesn't do yet:

It doesn't know the exact angle of the stem (it just aims for the center of the fruit).
It doesn't actively move the camera to look around leaves if a berry is blocked; it just picks the ones it can see.
It relies on the robot being set up in a specific way; if you move the robot to a different greenhouse, you have to recalibrate the "map" between the camera and the arm.

The Bottom Line

This paper presents a robot that uses smart vision to see clearly through clutter and simulated video-game training to move smoothly. It proves that you don't need to break a real robot to teach it how to pick fruit; you can teach it in a virtual world and then let it do the real work.

Technical Summary: Robotic Strawberry Harvesting with Robust Vision and Deep Reinforcement Learning based Sim-to-Real Control

Problem Statement
Robotic harvesting in controlled-environment agriculture (CEA) faces significant challenges due to the labor-intensive nature of the task and the complexity of the environment. Specifically, strawberry harvesting requires selecting individual fruits, handling them carefully, and removing them within a narrow maturity window. Current systems struggle with three tightly coupled problems: accurate fruit perception in cluttered scenes, stable 3D target estimation for manipulation, and efficient motion generation under uncertainty and workspace constraints. Traditional approaches often rely on inverse kinematics (IK) and general-purpose perception, which can lead to unstable trajectories, sensitivity to calibration noise, and unreliable target generation in dense, occluded agricultural settings. Furthermore, training control policies on physical robots is costly, slow, and prone to hardware damage.

Methodology
The authors propose a closed-loop robotic harvesting system integrating a robust vision module, simulation-trained Deep Reinforcement Learning (DRL) control, and ROS-based real-robot execution.

Vision Module (HRAttnEdge-YOLO26-seg):
To address instance segmentation in cluttered greenhouse scenes, the authors modified the YOLO26-seg architecture into HRAttnEdge-YOLO26-seg. Key architectural improvements include:
- High-Resolution P2 Branch: Integration of a stride-4 feature branch ( $P2/4$ ) to preserve fine spatial details (e.g., calyx and specular edges) often lost in lower-resolution features.
- Segmentation-Path Attention: Application of Cross Stage Partial with Spatial Attention (C2PSA) specifically on the $P2$ and $P3$ feature paths to sharpen mask boundaries without increasing global network computation.
- Edge-Supervised Prototype Learning: Replacement of the standard prototype branch with Proto26Edge, which includes an auxiliary edge head and boundary supervision loss. This encourages crisp object boundaries during training.
- 3D Localization: The system estimates 3D fruit centers by fusing the 2D mask centroids with aligned depth data from an RGB-D camera. A temporal association and smoothing scheme is applied to reduce target jitter caused by depth noise and occlusion.
Control Module (Sim-to-Real DRL):
Instead of relying on IK-based planners, the authors employ Proximal Policy Optimization (PPO) trained in the Isaac Lab simulation environment.
- Training Setup: The policy is trained on a UR10e manipulator in a domain-randomized simulation. The observation space includes relative joint positions/velocities, the commanded end-effector pose, and the previous action. The action space consists of continuous joint-position commands.
- Reward Function: The reward combines position tracking error (Euclidean distance and a fine-grained exponential term near the goal), orientation error, and penalties for rapid action changes and high joint velocities to ensure smooth, stable motion.
- Deployment: The trained policy is exported as a TorchScript model and integrated into a ROS framework. A command-streaming layer converts policy outputs into executable joint trajectories, including a halt mechanism to suppress steady-state oscillation near the target.
System Integration:
The perception, control, and task execution layers are unified via ROS 1. The system operates in a sequential manner: localizing ripe strawberries, prioritizing the nearest target, executing the learned reaching policy, performing a downward pull-based detachment, and transferring the fruit to a collection basket.

Key Contributions

HRAttnEdge-YOLO26-seg: A novel instance segmentation architecture specifically designed for greenhouse strawberry scenes, achieving superior boundary precision through high-resolution feature integration and edge supervision.
Simulation-Trained PPO Controller: Development of a target-conditioned reaching controller for a UR10e arm using Isaac Lab, enabling the generation of smooth, dynamically stable motions without extensive physical trial-and-error.
Integrated Closed-Loop Pipeline: A ROS-based framework that successfully bridges specialized perception and simulation-trained control to execute autonomous harvesting on a physical mobile robot platform.

Results

Vision Performance: On both self-collected and public datasets, the proposed model outperformed baseline YOLO26-seg and other state-of-the-art models (YOLOv8-seg, YOLO11-seg). It demonstrated a 10% to 14% improvement in segmentation performance (mAP75 and mAP50:95), particularly in handling small, partially occluded, and cluttered fruit instances.
Motion Quality: In-house comparisons showed the PPO controller produced significantly more stable and dynamically smoother motion (lower RMS jerk and variability) compared to an IK-based MoveIt baseline.
Harvesting Success: In greenhouse trials involving 281 strawberries across 130 trials, the system achieved:
- 96.6% reaching success rate.
- 91.3% grasp-and-pull success rate.
- 84.3% overall harvesting success rate.
  The drop from reaching to final harvesting was attributed primarily to post-contact fruit handling limitations of the soft gripper rather than perception or reaching failures.

Significance and Claims
The paper claims that this work provides a practical and resource-efficient alternative to conventional planner-dependent reaching in manipulation. By combining task-specific perception with simulation-trained PPO, the system enables reliable closed-loop robotic harvesting in complex agricultural environments. The authors emphasize that their approach reduces hardware dependency and development costs by eliminating the need for exhaustive physical trials during the policy learning phase. They position this as a scalable blueprint for autonomous harvesting that addresses the instability and singularity issues common in IK-based and visual-servoing systems.

Limitations
The authors modestly acknowledge several limitations:

The 3D target generation relies on segmented centroids and local depth rather than full 6D fruit pose or peduncle geometry, which may limit more complex detachment strategies.
The system lacks active perception or viewpoint adjustment for occluded targets.
The reliance on calibrated frame transformations with empirical offsets may limit portability across different hardware configurations or greenhouse layouts.
The current harvesting policy is sequential and reactive, lacking higher-level planning for dense canopies.