🤖 AI

Vision-Based Early Fault Diagnosis and Self-Recovery for Strawberry Harvesting Robots

This paper proposes a vision-based framework for strawberry harvesting robots that integrates the SRR-Net multi-task perception model with real-time end-effector feedback and corrective control strategies to diagnose and self-correct faults such as misalignment, empty grasping, and fruit slippage, thereby significantly enhancing harvesting stability and efficiency.

Original authors: Meili Sun, Chunjiang Zhao, Lichao Yang, Hao Liu, Shimin Hu, Ya Xiong

Published 2026-02-17

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Meili Sun, Chunjiang Zhao, Lichao Yang, Hao Liu, Shimin Hu, Ya Xiong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a strawberry harvesting robot as a very eager, but slightly clumsy, robot chef trying to pick the perfect berries from a bush. In the past, if this robot chef missed the berry, grabbed a leaf instead, or dropped the berry while twisting the stem, it would just freeze, wait for a human to come fix it, and then try again. This made the robot slow and inefficient.

This paper introduces a "smart self-correction system" that turns the robot into a self-aware chef who can spot its own mistakes and fix them instantly without human help. Here is how it works, broken down into simple concepts:

1. The "Super-Eye" (SRR-Net)

Think of the robot's camera as a regular pair of glasses. Usually, the robot needs to put on different pairs of glasses to do different things: one pair to find the berry, another to cut it out of the background, and a third to check if it's red enough to eat. This is slow and confusing.

The researchers built a "Super-Eye" (SRR-Net). Imagine this as a pair of smart glasses that can do three things at once while looking at the world:

Spot the berry: "There's a strawberry!"
Trace its outline: "Here is exactly where the edges are."
Check the ripeness: "Is it red enough to pick?"

By doing all this in one glance, the robot doesn't have to switch tasks, making it much faster.

2. The "Hand-Eye Coordination" Fix

Even with good eyes, the robot's hand (the gripper) might not line up perfectly with the berry. It's like trying to catch a ball while wearing thick gloves; you might miss by a few inches.

In the past, the robot would just try to grab and fail. Now, the robot has a second chance to aim.

The Analogy: Imagine you are reaching for a cup on a table. You reach out, but your hand is slightly to the left. Instead of grabbing the air, your brain says, "Wait, I'm off," and you slide your hand over to the right before you close your fingers.
The Robot's Move: The robot looks at the berry and its own hand simultaneously. If it sees a gap, it calculates the difference and slides its hand to the perfect spot before grabbing. This happens in less than a second.

3. The "Peek-a-Boo" Camera

The robot uses a soft, air-filled gripper (like a balloon) to hold the berry. The problem? You can't feel if you are holding it tightly enough just by "feeling" the air pressure.

To solve this, the researchers put a tiny camera inside the gripper, like a spy camera hidden in a pocket.

The "Empty Hand" Check: As the robot pulls its hand back, the tiny camera takes a picture. If it sees nothing (or a green, unripe berry), the robot knows, "Oops, I missed!" It immediately lets go, resets, and picks a new berry. It doesn't waste time trying to twist a stem that isn't there.
The "Slip" Prediction: This is the coolest part. As the robot twists the stem to snap the berry off, the tiny camera watches the berry closely. It uses a special AI (like a crystal ball) to predict if the berry is about to slip out of the hand before it actually falls.
- If it's stable: The robot finishes the job.
- If it's slipping: The robot quickly re-inflates its hand, grabs the berry again, and tries to twist it off a second time.
- If it's already gone: The robot gives up on that specific berry immediately to save time, rather than wasting seconds trying to fix a lost cause.

The Result: A Faster, Smarter Robot

Before this system, a full picking cycle took about 11 seconds. If the robot made a mistake, it would get stuck or take even longer.

With this new "self-recovery" system:

Mistakes are fixed instantly: If the hand is off-center, it moves. If the grip is loose, it re-grabs.
Time is saved: By stopping bad attempts early, the robot finishes a cycle in about 4 to 7 seconds instead of 11.
Success rate: The robot successfully recovers from slipping berries about 81% of the time.

In summary: This paper teaches a strawberry robot how to "look at its own hands," realize when it's making a mistake, and fix it on the fly. It turns a clumsy robot that needs constant babysitting into a confident, independent worker that can handle the messy, unpredictable reality of a strawberry field.

1. Problem Statement

Strawberry harvesting robots face significant challenges in orchard environments that compromise operational stability and efficiency. The primary issues identified are:

Low Integration of Visual Perception: Traditional systems often separate detection, segmentation, and ripeness estimation, leading to increased model complexity, reduced inference speed, and fragmented fault diagnosis.
Positional Misalignment: Errors in hand-eye calibration and dynamic environmental factors cause misalignment between the end-effector (gripper) and the target fruit, resulting in empty grasps or failed detachments.
Gripping Failures and Slippage: Flexible pneumatic grippers lack reliable tactile sensors. Consequently, robots cannot easily detect "empty grasps" (missing fruit) or "misgrasps" (unripe fruit) during the deflating stage, nor can they predict fruit slippage during the snap-off stage. This leads to wasted cycle time and fruit damage.
Lack of Self-Recovery: Current systems often rely on human intervention to diagnose faults, causing frequent work interruptions and reducing effective field operation time.

2. Methodology

The authors propose a Vision-Based Fault Diagnosis and Self-Recovery Framework centered around three core components:

A. End-to-End Multi-Task Perception (SRR-Net)

Architecture: Based on YOLOv11, the authors developed SRR-Net, a lightweight, end-to-end network with a shared backbone and neck.
Tasks: It simultaneously performs:
1. Object Detection: Identifying strawberries and the robotic gripper.
2. Instance Segmentation: Delineating the boundaries of fruits and the gripper.
3. Ripeness Estimation: A parallel regression branch predicts ripeness scores (0–1.1) using Mean Absolute Error (MAE) loss.
Benefit: Unifies perception and fault diagnosis, reducing computational load while maintaining high accuracy.

B. Relative Error Compensation

Mechanism: Instead of relying solely on pre-calibrated coordinates, the system detects both the target strawberry and the gripper simultaneously within the same camera view.
Process:
1. When the gripper approaches the picking point, SRR-Net extracts 2D coordinates of both objects.
2. Using depth data and hand-eye calibration, 3D coordinates are reconstructed.
3. The relative error ( $\Delta x, \Delta y$ ) is calculated. If the error exceeds a tolerance threshold ( $T=10$ mm), a compensated picking point is generated.
4. The robot arm adjusts its trajectory to re-align with the fruit before grasping.

C. Early Abort Strategy & Self-Recovery

A micro-optical camera embedded in the end-effector provides real-time visual feedback during critical stages:

Deflating Stage (Grasp Verification):
- A MobileNet V3-Small classifier analyzes the gripper view.
- Classes: 0 (Ripe Strawberry), 1 (Empty), 2 (Unripe Strawberry).
- Action: If the fruit is missing or unripe, the system triggers an early abort: the gripper re-inflates, descends, and skips the snap-off/placing stages to immediately select a new target, saving time.
Snap-off Stage (Slippage Prediction):
- A Time-Series LSTM Classifier predicts the probability of slippage based on visual features (normalized areas of strawberry, gripper, background, and strawberry dimensions) extracted from 5 consecutive frames.
- States: Normal (0), Slipping (1), Slipped (2).
- Action:
  - If Slipping is predicted (consecutive frames): The gripper re-inflates and attempts a secondary snap-off.
  - If Slipped is confirmed: The system triggers an early abort to terminate the cycle and reset.

3. Key Contributions

SRR-Net Framework: An integrated multi-task perception model that unifies detection, segmentation, and ripeness estimation, achieving high accuracy without sacrificing real-time performance.
Vision-Based Error Compensation: A novel method using simultaneous target-gripper detection to correct positional deviations in real-time, eliminating the need for complex external sensors.
Early Abort & Recovery Strategy: A robust mechanism using embedded micro-cameras and lightweight classifiers (MobileNet V3, LSTM) to detect grasp failures and slippage early, enabling the robot to self-correct or abort efficiently without human intervention.

4. Experimental Results

Experiments were conducted using the HarvestFlex robot system with datasets including FaultData, GraspData, SnapData, and SlipData.

Perception Performance (SRR-Net):
- Detection: Precision 0.895 / Recall 0.813 (Strawberries); Precision 0.972 / Recall 0.958 (Hands).
- Segmentation: Precision 0.887 / Recall 0.747 (Strawberries).
- Ripeness Estimation: Mean Absolute Error (MAE) of 0.035.
- Speed: Inference speed of 163.35 FPS, comparable to standard YOLOv11-seg.
Error Compensation:
- Reduced mean physical errors from 11.52 mm (x-axis) and 5.15 mm (y-axis) to 3.12 mm and 4.11 mm, respectively.
- Execution time for compensation remained under 1 second.
Fault Diagnosis & Recovery:
- Grasp Adjustment (MobileNet V3): Achieved 100% precision, recall, and F1-score for empty/misgrasp detection.
- Slip Prediction (LSTM):
  - Normal status success rate: 96.88%.
  - Slipping recovery success rate: 81.25%.
  - Slipped detection success rate: 88.89%.
Operational Efficiency:
- Baseline Cycle Time: 11.21 ± 0.15 s.
- With Grasp Adjustment: Reduced to 4.56 s (saving ~6.7 s per cycle).
- With Slip Prediction: Reduced to 7.08 s (saving ~4 s per cycle).
- Recovery Time: Secondary snap-off for slipping fruit took 1.88 s.

5. Significance

This research provides a practical, sensor-light solution for autonomous fruit harvesting. By leveraging vision-based fault diagnosis and self-recovery, the system significantly reduces reliance on expensive tactile sensors and human intervention. The proposed framework enhances the robustness of harvesting robots in dynamic orchard environments, ensuring continuous operation and improving overall harvesting efficiency by minimizing downtime caused by mechanical or perception errors. The integration of multi-task learning with real-time corrective control represents a significant step toward fully autonomous agricultural robotics.