💻 computer science

Learning to Pick: A Visuomotor Policy for Clustered Strawberry Picking

This paper presents a learning-based visuomotor policy for a 4-DoF robotic arm that effectively picks occluded strawberries from clusters by leveraging human demonstrations and an End Pose Assisted Action Chunking Transformer to navigate complex clutter and achieve precise stem access.

Original authors: Zhenghao Fei, Wenwu Lu, Linsheng Hou, Chen Peng

Published 2026-02-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhenghao Fei, Wenwu Lu, Linsheng Hou, Chen Peng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine trying to pick a ripe strawberry from a garden, but instead of a clear path, the fruit is hidden behind a tangle of leaves, stems, and other berries. If you grab too hard, you crush it; if you grab the wrong one, you ruin the harvest. Now, imagine teaching a robot to do this.

This paper is about teaching a robot to be a gentle, dexterous strawberry picker using a method called "learning by watching," rather than trying to program every single move with complex math.

Here is the story of how they did it, broken down into simple concepts:

1. The Problem: The "Tangled Web"

Strawberries don't grow in neat rows like corn. They grow in messy clusters, often hiding behind leaves or other fruit.

The Old Way: Traditional robots try to "see, plan, and act." They take a picture, calculate the exact 3D position of the stem, and plan a perfect path to cut it. But when the strawberry is hidden or the leaves are soft and moving, this math breaks down. It's like trying to solve a complex equation while someone is shaking the table.
The New Way: Instead of calculating the path, the robot learns from a human. Just like a child learns to tie shoes by watching a parent, the robot watches a human operator pick strawberries and learns the feel of the movement.

2. The Robot: The "Four-Armed Acrobat"

The team didn't use a giant, expensive robot with six moving joints (like a human arm). Instead, they used a 4-DoF SCARA arm.

The Analogy: Think of a human arm with six joints (shoulder, elbow, wrist, etc.) as a gymnast who can do backflips. The robot they used is more like a crane on a construction site or a printer head. It moves in a flat plane (left, right, forward, back) and can twist a little.
Why? Strawberries in modern farms often grow on flat tables. You don't need a gymnast to pick them; you need a precise, fast, and cheap crane. This robot is cheaper and faster, but it has to be very smart to navigate the mess.

3. The Teacher: The "Teleoperation Interface"

How do you teach a robot to be gentle? You don't tell it "move 5mm left." You let a human do it.

The Setup: A human operator sits at a computer with a special controller (like a video game controller with two arms). They move the controller, and the robot mimics the movements in real-time.
The Data: While the human picks, two cameras on the robot's "wrist" (one looking up, one looking down) record everything. The robot learns not just where to go, but how to push a leaf aside gently to reveal the fruit.

4. The Brain: The "Future-Seeing" AI

The core of their innovation is a new AI brain called EPACT (End Pose Assisted Action Chunking with Transformers).

The "Action Chunking": Instead of thinking "move left, then move right," the AI thinks in chunks. It predicts a whole sequence of moves (like a 3-second dance routine) all at once. This makes the movement smooth, not jerky.
The "End Pose" Trick (The Secret Sauce):
- The Problem: If the AI just guesses the angles of the robot's joints, a tiny mistake in the shoulder can cause a huge mistake at the hand (the gripper). It's like trying to aim a laser pointer by adjusting the elbow; a small wobble at the elbow makes the dot miss the target by miles.
- The Solution: The AI first guesses where the hand needs to be (the target). Then, it works backward to figure out how the joints should move to get there.
- The Metaphor: Imagine you are walking to a specific tree in a forest.
  - Old AI: "I will move my left foot 2 inches, then my right foot 2 inches..." (It might get lost).
  - EPACT AI: "I need to get to that tree. Okay, I'll walk that way." It focuses on the destination first, then figures out the steps. This ensures the robot's hand actually reaches the strawberry stem, even if the path is tricky.

5. The Results: From "Clumsy" to "Dexterous"

They tested the robot in a lab with fake strawberries tangled in wires and leaves.

The Baseline: A standard robot (using the old math method) failed almost everything.
The New Robot: The learning-based robot succeeded about 72% of the time, even in the messiest scenarios.
The Magic: The robot learned to push leaves aside, dodge obstacles, and gently grab the stem without crushing the fruit. It didn't just "pick"; it "manipulated" the environment, just like a human.

6. The Catch and The Future

The Catch: The robot is still a bit confused by real-world messiness. It works great in the lab, but real fields have different lights, wind, and dirt. It needs more practice (more data) to become a master.
The Future: The authors suggest a hybrid approach. If the strawberry is easy to see, use the fast, old math method. If it's hidden in a cluster, switch to the "learning" robot that can be gentle and clever.

Summary

This paper is about teaching a robot to be a gentle gardener. By letting a human show the robot how to navigate a messy strawberry patch, and by giving the robot a brain that focuses on the goal (the stem) rather than just the steps (the joints), they created a system that can pick fruit in ways traditional robots never could. It's a step toward robots that don't just work in factories, but can work in the delicate, messy world of nature.

1. Problem Statement

Strawberry harvesting is a labor-intensive process that remains difficult to automate due to the fruit's fragility and its natural growth habit. Unlike crops like corn or wheat, strawberries grow in dense, interwoven clusters with leaves, stems, and unripe fruits often occluding the target.

Key Challenges:
- Occlusion: Target fruits are frequently hidden, requiring the robot to navigate through clutter.
- Dexterity: Successful picking requires delicate manipulation to bypass or gently move soft obstacles (leaves/stems) without damaging the fruit.
- Precision: The picking point must be precise (cutting the stem just above the calyx).
- Limitations of Traditional Methods: Conventional "percept-plan-control" pipelines struggle with scene understanding, depth estimation in clutter, and trajectory planning for deformable objects. They often fail to adapt to the dynamic variability of real-world environments.

2. Methodology

The authors propose a learning-based approach using Imitation Learning (IL) to acquire dexterous visuomotor skills directly from human demonstrations, bypassing the need for complex explicit modeling of the environment.

A. System Hardware

Robot: A cost-effective 4-DoF SCARA (Selective Compliance Articulated Robot Arm) arm. This was chosen over 6-DoF arms because strawberries in vertical farming/tabletop setups grow in a planar arrangement, making the 4-DoF design efficient and sufficient.
Teleoperation: A dual-arm human interface allows operators to control the SCARA arm remotely.
Sensing: Two wrist-mounted cameras (one top, one bottom) provide multi-view visual data to assist in handling occlusions.

B. Algorithm: EPACT (End Pose Assisted Action Chunking with Transformers)

The core contribution is a modified version of the Action Chunking Transformer (ACT) algorithm, tailored for this specific task.

Standard ACT: Predicts a sequence of future joint actions (action chunking) to reduce compounding errors and ensure smooth trajectories.
EPACT Innovation: The authors observed that standard ACT, which predicts joint angles directly, can suffer from significant end-effector pose errors due to small joint deviations. To address this, EPACT introduces an End Pose Loss.
- Mechanism: The policy first predicts the 6D end-effector pose. This pose is then decoded into joint angles via a neural network-based Inverse Kinematics (IK) module.
- Loss Function: The training objective includes a standard action reconstruction loss plus a specific End Pose Reconstruction Loss ( $L_{rec\_end\_pose}$ ), forcing the network to prioritize the accuracy of the final tool position.
Two Architectural Variants:
1. EPACT-L (Latent): Uses a unified action head to predict the 5D action vector (4 arm joints + 1 gripper) from the predicted 6D pose, relying on the transformer's latent state to resolve the gripper's open/close state.
2. EPACT-EE (End-Effector): Uses a decoupled action head. One branch predicts the 4D arm joints via the IK module based on the end pose; the other branch predicts the 1D gripper action directly from the transformer's hidden state. This separation is hypothesized to be more robust.

C. Data Collection

Dataset: 511 human demonstrations collected in a controlled lab setting using artificial strawberries (to simulate weight and deformability) and magnetic stems.
Scenarios: Five distinct occlusion scenarios (clusters of 2-3 strawberries) and one non-occluded baseline.
Task: The human operator maneuvers the robot to push aside leaves/stems or detour around them before grasping the stem and cutting.

3. Key Contributions

System Integration: Developed a remotely operated data collection system integrating a 4-DoF SCARA arm with a teleoperation interface, proving the viability of lower-DoF arms for complex agricultural tasks.
Algorithmic Advancement: Introduced EPACT, an enhanced imitation learning framework that incorporates an end-pose loss and an inverse-kinematics module. This significantly improves the precision of the end-effector trajectory compared to standard ACT.
First-of-its-Kind Deployment: Demonstrated the first successful learning and deployment of a dexterous visuomotor policy for picking real, clustered strawberries via imitation learning.
Comprehensive Evaluation: Conducted a rigorous study on the impact of occlusion scenarios and camera viewpoints (single vs. dual view) on policy performance.

4. Experimental Results

The system was evaluated across six scenarios (States 0–5) with 10 trials per policy per scenario.

Performance Comparison:
- Baseline (ACT): Achieved an average success rate of 35.0%.
- EPACT-L: Achieved 66.7% average success rate.
- EPACT-EE: Achieved 71.7% average success rate.
- Improvement: The end-pose assistance improved success rates by 36.7% (EPACT-L) and 41.7% (EPACT-EE) over the baseline.
Generalization: Even in the non-occluded "State 0" (which was not included in the training data), the best policy achieved a 70% success rate, though this highlighted that pure imitation learning still struggles with zero-shot generalization to completely novel, simple tasks.
Camera Views:
- Multi-view (Up + Down): Provided the best performance due to richer 3D geometric understanding and better occlusion handling.
- Single-view: Performance varied by architecture. EPACT-L performed better with the "Up" camera, while EPACT-EE performed better with the "Down" camera, suggesting different architectures prioritize different visual features.
Real-World Transfer: Preliminary tests on real strawberries showed successful transfers (e.g., navigating through stems), though success rates were lower than in the artificial lab setting due to the sim-to-real gap.

5. Significance and Future Directions

Significance: This work demonstrates that imitation learning can solve high-dexterity, occlusion-heavy agricultural tasks that traditional planning algorithms cannot. It validates the use of cost-effective 4-DoF SCARA arms for complex harvesting, potentially lowering the barrier for commercial adoption.
Limitations:
- Data Efficiency: 511 demonstrations are insufficient for robust generalization across all real-world lighting and cluster variations.
- Sim-to-Real Gap: Performance drops when moving from artificial to real strawberries.
Future Work:
- Developing portable data collection devices to scale up human demonstration datasets.
- Utilizing simulation for pre-training followed by real-world fine-tuning.
- Exploring hybrid systems that use traditional planning for simple, unoccluded fruits and imitation learning for complex, clustered scenarios.

In conclusion, the paper presents a robust framework for robotic strawberry picking that moves beyond rigid planning to flexible, learned dexterity, offering a promising path toward sustainable and automated fruit harvesting.