💻 computer science

Text-conditioned Segmentation for Tomato Phenotyping via Procedural Synthetic Data

This paper presents a sim-to-real framework that generates a large-scale procedural synthetic dataset of tomato greenhouses to fine-tune the Segment Anything Model 3 (SAM 3), significantly improving text-conditioned segmentation performance for tomato phenotyping in complex real-world environments.

Original authors: Samy Mounir, Mikolaj Cieslak, Najmeddine Dhieb, Hakim Ghazzai, Jonathan Klein, Katja Froehlich, Soeren Pirk, Wojciech Palubicki, Gianluca Setti, Ahmed M. Eltawil, Dominik L. Michels

Published 2026-07-22

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Samy Mounir, Mikolaj Cieslak, Najmeddine Dhieb, Hakim Ghazzai, Jonathan Klein, Katja Froehlich, Soeren Pirk, Wojciech Palubicki, Gianluca Setti, Ahmed M. Eltawil, Dominik L. Michels

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to understand a garden. You want it to spot every tomato, leaf, and stem so it can help farmers harvest crops without getting tired. This is the world of computer vision, where computers learn to "see" the world through cameras. But here's the tricky part: teaching a computer is like teaching a child to recognize animals. If you only show the child pictures of lions in a zoo, they might get confused when they see a lion in the wild. Similarly, computers struggle to recognize plants in messy, real greenhouses because they haven't seen enough examples.

To solve this, scientists use synthetic data. Think of this as building a video game version of a greenhouse. In a game, you can create millions of perfect pictures of tomatoes, label every single pixel instantly, and never worry about the sun being too bright or a leaf blocking the view. This is called procedural generation, where a computer follows a set of rules (like a recipe) to grow unique, fake plants that look real.

The big question is: If you train a robot on a video game, will it actually work in a real greenhouse? This paper explores that exact mystery. It asks if we can take a super-smart, pre-trained computer brain (called a foundation model) that knows a lot about the world, teach it using our fake video-game tomatoes, and then see if it can suddenly become an expert at spotting real tomatoes. The goal is to save farmers time and money by automating the hard work of counting and checking their crops.

The Video Game Greenhouse and the Super-Brain

The authors of this paper decided to build a digital twin of a commercial cherry tomato greenhouse. Instead of taking thousands of photos of real plants (which takes forever and is expensive), they used a powerful video game engine called Unreal Engine 5 to create a virtual world. Inside this world, they programmed a "digital plant" using a set of rules called L-systems.

Think of L-systems like a genetic code for a video game plant. Instead of drawing every single leaf by hand, the computer follows instructions: "Grow a stem, add a leaf here, branch off there." The authors added special rules to mimic how real farmers manage tomatoes, such as "leaning and lowering" (where the plant is gently tipped over and lowered as it grows tall) and pruning (cutting off old parts). They even added random variations so that no two digital plants looked exactly the same, just like in nature.

From this virtual greenhouse, they generated a massive dataset of 68,000 synthetic images. Each image came with perfect labels: the computer knew exactly which pixels were fruit, which were leaves, and which were stems, down to the individual fruit. It was like having a teacher who never makes a mistake.

The "Segment Anything" Brain

Next, they brought in a superstar AI model called SAM 3 (Segment Anything Model 3). Imagine SAM 3 as a very smart student who has read every book in the library about pictures. It can look at a photo of a dog or a car and instantly point out where the object is, even if it has never seen that specific dog before. This is called zero-shot learning.

However, when the authors asked this super-smart student to look at a dense, messy tomato plant, it got confused. The leaves were overlapping, the fruits were hidden, and the lighting was tricky. The student's performance dropped because it wasn't used to the specific "language" of tomato plants.

The Experiment: Teaching the Student with Fake Data

The team decided to give the student a crash course using their 68,000 fake tomato images. They tried three different ways to teach the model:

Full Fine-Tuning: They made the student relearn everything from scratch using the fake data.
LoRA (Low-Rank Adaptation): They made the student learn only a tiny, specific part of its brain, keeping most of its original knowledge intact.
Weight Interpolation (WiSE-FT): This was the cleverest trick. They took the "fake-data-trained" brain and mixed it with the "original super-smart" brain. It's like blending a new recipe with the original one to get the best of both worlds. They mixed them so that 75% of the brain was trained on the fake tomatoes, but 25% remained the original, general knowledge.

The Results: Does the Fake Training Work?

Here is where things got interesting. When they tested the models on the fake images, the "Full Fine-Tuning" model did the best. It seemed to have memorized the video game perfectly.

But when they took the models out of the video game and tested them on real photos from actual greenhouses, the results flipped. The model that had memorized the fake data the most actually did the worst. It was too focused on the video game rules and couldn't handle the messy reality.

The winner was the Weight Interpolation model (WiSE-FT). By keeping a little bit of the original "super-smart" knowledge, it was able to adapt to the real world much better.

The original, untrained model (Zero-shot) got a score of 52.7% on finding fruits in real images.
The best trained model (Full FT) improved this to 65.6%.
This means the model got 12.9 percentage points better at spotting tomatoes just by learning from the fake data.

The authors also tested the model on a real greenhouse they didn't have labels for (because they couldn't ask the farmers to label every single pixel). They used a "confidence" test: how sure was the model about its answer? The trained model was much more confident and stable, rarely getting confused by shadows or overlapping leaves, whereas the untrained model often missed fruits entirely or hallucinated flowers that weren't there.

The Takeaway

The paper suggests that you don't need to build a robot from scratch to understand a greenhouse. Instead, you can take a general "super-robot" that already knows how to see, and give it a specialized training course using a video game. The key is not to make the robot forget everything it knew, but to gently nudge it toward the specific task of spotting tomatoes.

The authors also made sure to share their work with the world. They released the code for their video game greenhouse, the 68,000 fake images, and the trained robot brain. This means other scientists can use these tools to build better farming robots without having to start from zero.

In short, the paper shows that a little bit of video game practice, mixed with a lot of real-world smarts, can help computers finally learn to see the complex, messy beauty of a tomato plant. It's a step toward a future where robots might one day help harvest our food, saving farmers from back-breaking work.

Problem Statement

Vision-based automation holds significant promise for reducing manual labor in greenhouse crop production and phenotyping. However, the deployment of these systems is constrained by the scarcity of large-scale, annotated training data, particularly for complex agricultural environments. While recent vision foundation models (VFMs) like the Segment Anything Model 3 (SAM 3) demonstrate strong zero-shot generalization, their performance degrades in dense agricultural scenes characterized by overlapping foliage, occluded fruit clusters, and specific crop taxonomies. Furthermore, existing synthetic datasets often rely on static 3D assets or simplified environments that fail to capture the continuous growth dynamics and intensive management practices (e.g., pruning, vine lowering) of commercial greenhouses. Consequently, there is a lack of high-fidelity virtual environments suitable for training robust perception systems, and fine-tuning powerful foundation models on niche agricultural data without sacrificing their generalizability remains an open challenge.

Methodology

The authors propose a sim-to-real framework that combines procedurally generated synthetic data with the fine-tuning of a foundation model. The methodology consists of three primary components:

1. Procedural Synthetic Data Generation:
The authors developed a high-fidelity virtual greenhouse environment using Unreal Engine 5. Central to this is a novel procedural model of cherry tomato plants (Solanum lycopersicum var. cerasiform) built using L-systems. Unlike static models, this system simulates:

Biological Growth: Sympodial branching patterns, vegetative and reproductive development, and compound leaf structures based on established botanical rules.
Greenhouse Management: The model incorporates biomechanical tropisms and algorithmic simulations of commercial practices, specifically the "leaning and lowering" technique where stems are shifted horizontally as they grow, and pruning of spent basal portions.
Stochastic Variation: Geometric parameters (leaf scale, orientation, branching angles) are modulated by stochastic multipliers to ensure no two simulated plants are morphologically identical.
Data Acquisition: A configurable pipeline generates synchronized RGB images, depth maps, and pixel-perfect ground truth annotations (semantic masks for leaf, stem, flower, fruit, background; instance masks; and bounding boxes) across diverse viewpoints and lighting conditions.

2. Foundation Model Fine-Tuning:
The authors fine-tuned SAM 3 on the generated 68,000-image synthetic dataset using three distinct strategies to specialize the model for tomato organs while preserving its zero-shot capabilities:

Parameter-Efficient Fine-Tuning (LoRA): Applied Low-Rank Adaptation to the attention layers of the SAM 3 vision encoder, updating only a small fraction of parameters.
Full Fine-Tuning: Trained all semantic components of SAM 3, including the semantic segmentation head, on the synthetic corpus.
Weight Interpolation (WiSE-FT): To balance domain specialization with general visual priors, the authors interpolated weights between the pre-trained zero-shot checkpoint and the fine-tuned checkpoints using the formula $\theta_{wise}(\alpha) = (1 - \alpha) \cdot \theta_{base} + \alpha \cdot \theta_{ft}$ with $\alpha = 0.75$ . This was applied to both full fine-tuning and LoRA variants.

3. Evaluation Framework:
The models were evaluated on three external real-world mask datasets (TomatoMAP-Seg, LaboroTomato, Rob2Pheno) and a custom real-world dataset captured in a commercial Saudi Arabian greenhouse (which lacked ground truth and was used for qualitative and label-free confidence analysis). The evaluation focused on semantic segmentation performance, specifically measuring macro fruit Intersection over Union (IoU) and label-free confidence metrics.

Key Contributions

Large-Scale Synthetic Dataset: The generation of a photorealistic, procedurally generated synthetic dataset for tomato phenotyping with pixel-perfect ground truth annotations (RGB, depth, semantic, and instance masks) that captures commercial greenhouse management practices.
Fine-Tuning Pipeline: A proposed text-conditioned foundation model fine-tuning pipeline that leverages synthetic data to specialize SAM 3 for greenhouse crop organs via LoRA, full fine-tuning, and weight interpolation.
Performance Gains: Demonstration that combining synthetic data with SAM 3 fine-tuning significantly improves segmentation performance on real-world data, achieving a 12.9 percentage point increase in macro fruit IoU (from 52.7% to 65.6%) compared to the zero-shot baseline.
Open Release: The public release of the procedural model, the generated synthetic dataset, and the fine-tuned SAM 3 weights on Hugging Face to facilitate community benchmarking.

Results

Real-World Transfer: The Full FT SAM 3 variant achieved the best aggregate performance across external real-world datasets, reaching a macro fruit IoU of 0.6562 and an image-weighted fruit IoU of 0.7372. The WiSE-FT LoRA SAM 3 variant achieved the highest performance on the LaboroTomato dataset (0.7728 IoU).
Synthetic vs. Real Inversion: A critical finding was the "synthetic-real inversion." Supervised baselines (DeepLabV3+, SegFormer) and Full FT models ranked highest on the synthetic validation set but did not necessarily transfer best to real data. Conversely, LoRA variants, which scored lower on synthetic validation, transferred substantially better to real imagery. This suggests that optimizing solely for synthetic metrics is insufficient for model selection.
Label-Free Confidence: On the unlabeled real greenhouse dataset, the WiSE-FT variant demonstrated superior robustness compared to the zero-shot baseline, showing higher all-pixel confidence (0.6114 vs. 0.3685), better Test-Time Augmentation (TTA) agreement (0.9045 vs. 0.7345), and more realistic foreground coverage. The zero-shot model exhibited systematic failures, including foreground under-coverage and "flower hallucination."
Prompt Simplicity: The fine-tuned models successfully responded to concise, organ-level text prompts ("stem," "leaf," "fruit"), producing precise, non-overlapping semantic masks without complex prompt engineering.

Significance and Claims

The paper claims that its primary significance lies in demonstrating that procedurally generated synthetic data can effectively bridge the sim-to-real gap when paired with foundation model fine-tuning. The authors emphasize that the synthetic data does not need to teach the model general visual understanding from scratch; rather, it steers an already general model (SAM 3) toward the specific structural complexities of tomato canopies.

The work highlights that real-domain evaluation is essential for reliable model selection, as synthetic-domain metrics alone are poor predictors of real-world generalization. By providing an out-of-the-box tool (fine-tuned weights and data), the authors aim to accelerate the development of scalable agricultural vision systems without requiring researchers to train custom models from scratch. The paper concludes that while synthetic data is a powerful enabler, the specific fine-tuning strategy (e.g., weight interpolation vs. full fine-tuning) must be carefully selected based on real-world performance rather than synthetic validation scores.