Tomato Multi-Angle Multi-Pose Dataset for Fine-Grained Phenotyping

This paper introduces TomatoMAP, a comprehensive dataset of 64,464 multi-angle, multi-pose tomato images with detailed annotations for seven regions of interest and 50 growth stages, which was validated to demonstrate that a cascading deep learning framework achieves fine-grained phenotyping accuracy and speed comparable to human experts.

The Gist

Imagine you are a detective trying to solve a mystery, but instead of looking for fingerprints, you are looking at a tomato plant to figure out exactly how healthy it is, how big its fruit will be, and what stage of life it's in.

For a long time, scientists had to do this "detective work" by looking at plants with their own eyes. But humans get tired, they get distracted, and sometimes two experts will look at the same tomato and disagree on whether it's "half-ripe" or "almost ripe." This is called observer bias, and it makes scientific data messy and hard to trust.

This paper introduces a new solution: TomatoMAP. Think of it as a super-powered, robot assistant that never gets tired, never argues, and sees the plant from every possible angle.

Here is a breakdown of how they built this "Tomato Robot" and what it can do, using some everyday analogies:

1. The "Tomato Photo Booth" (The Data Collection)

Imagine a tomato plant sitting in the middle of a room. Instead of taking one photo, the scientists built a special machine that acts like a 360-degree photo booth.

• The Setup: They put the plant on a spinning turntable (like a lazy Susan at a restaurant).
• The Cameras: They hung four cameras around the plant at different heights (looking down, straight on, and from the side).
• The Spin: As the plant spins, the cameras snap pictures at every 30-degree turn.
• The Result: Instead of one flat picture, they get a "movie" of the plant from every angle. They did this for 101 different plants over several months, capturing 64,464 images. It's like having a complete 3D model of the plant's life story, from a tiny sprout to a fruit-bearing giant.

2. The "Labeling Party" (The Annotations)

Now, they have thousands of pictures, but computers can't understand them unless you tell them what they are looking at. This is where the "labeling" happens.

• The Human Team: A group of expert botanists (plant doctors) looked at the photos and drew boxes around specific parts: "That's a leaf," "That's a flower cluster," "That's a baby fruit."
• The AI Helper: They didn't just do this manually for every single photo. They taught a computer to do the heavy lifting. First, the humans labeled a few thousand photos. Then, the computer learned from them and labeled the rest. Finally, the humans double-checked the computer's work.
• The Detail: They didn't just say "fruit." They got super specific. They labeled fruits by size (2mm, 4mm, 6mm) and ripeness (green, turning red, fully red). They even labeled tiny "side shoots" and "flower buds." It's like sorting a pile of LEGO bricks not just by color, but by the exact shape and size of every single piece.

3. The "Three-Layer Cake" (The AI Models)

The scientists didn't just build one big, clumsy robot brain. They built a three-layer team that works together, like a relay race:

• Layer 1 (The Classifier): This is the "General." It looks at the plant and says, "Okay, this plant is at stage 80 of its life (BBCH scale)." It gives a broad overview.
• Layer 2 (The Detective): Based on that stage, this layer zooms in. It says, "Since it's stage 80, I'm going to look specifically for fruit clusters." It finds the specific parts of the plant.
• Layer 3 (The Surgeon): This layer is the most precise. It doesn't just draw a box around the fruit; it traces the exact outline of every single pixel of the fruit. It separates the fruit from the leaf perfectly, even if they are touching.

4. The "Face-Off" (Human vs. AI)

The big question was: Is the robot as good as the human expert?
To find out, they set up a competition. They took 295 new photos and asked five human experts to label them. Then, they asked the AI to label the same photos.

• The Result: The AI and the humans agreed almost perfectly. In fact, the AI was actually more consistent. If you asked the same human to label the same photo twice, they might make a small mistake the second time because they were tired. The AI never gets tired; it gives the exact same answer every time.
• The Takeaway: The AI isn't replacing the humans; it's giving them a super-tool that removes the "human error" and saves them hours of boring work.

Why Does This Matter?

Tomatoes are a huge part of our food supply. By using this dataset and these AI tools, scientists can:

• Breed better tomatoes: Find the plants that produce the most fruit or resist disease faster.
• Save money: Stop paying people to stand in greenhouses counting leaves all day.
• Be fair: Get data that is the same no matter who is looking at it.

In short: The scientists built a high-tech, spinning photo booth to take thousands of pictures of tomatoes. They taught a computer to recognize every tiny detail of the plant, from a 2mm bud to a ripe red fruit. They proved that this computer is just as good as a team of human experts, but it never gets tired, never argues, and works 24/7. This is a giant leap forward for growing food smarter and faster.

Technical Summary

1. Problem Statement

Traditional plant phenotyping for Solanum lycopersicum (tomato) faces significant challenges:

• Observer Bias & Inconsistency: Manual measurement is subjective, labor-intensive, and prone to human error, limiting reproducibility.
• Data Limitations: Existing datasets often consist of single-pose, single-angle images with low instance counts per class. They lack 3D topological information (e.g., interleaf distances, stem-leaf angles) and fail to capture the full growth cycle comprehensively.
• Fine-Grained Complexity: Tomato plants exhibit high phenotypic plasticity and subtle variations in traits (fruit color, flower stages, shoot structure) that are difficult to distinguish using standard 2D analysis or coarse-grained models.
• Scalability: Traditional methods cannot scale to the high-throughput data required for modern breeding and genomic selection.

2. Methodology

A. Data Acquisition System

The authors developed a specialized IoT-based imaging station to capture multi-angle, multi-pose data under controlled greenhouse conditions:

• Hardware: A synchronized array of four OV5647 5-megapixel CMOS cameras. Three have 90° lenses, and one has a 170° fisheye lens.
• Configuration: Cameras are mounted at vertical inclination angles of 45°, 90°, 135°, and 180°.
• Rotation: Plants are placed on a turntable with 30° rotational increments, capturing 12 distinct poses per plant to achieve a full 360° view.
• Subjects: 101 individual tomato plants monitored over 163 days (August to January) at 32 time points.
• Resolution:
  • Low-Resolution (LR): 1080 × 1440 pixels for object detection and classification (64,464 images).
  • High-Resolution (HR): 3648 × 5472 pixels captured via a Panasonic Lumix DSLR for semantic and instance segmentation (3,616 images).

B. Dataset Structure (TomatoMAP)

The dataset is organized into three subsets supporting a cascading model architecture:

1. TomatoMAP-Cls: For BBCH-based growth stage classification (50 fine-grained classes).
2. TomatoMAP-Det: For 3D-aware object detection of 7 Regions of Interest (ROIs): whole plant, leaf, panicle, batch of flowers, batch of fruits, axillary shoot, and shoot.
3. TomatoMAP-Seg: For pixel-wise semantic and instance segmentation of flowers (by size: 2mm–12mm) and fruits (by ripeness: nascent to fully ripe).

C. Annotation Protocol

• Classification: Uses the BBCH scale (modified for tomatoes) to define 50 developmental stages.
• Detection: Employs a progressive AI-assisted workflow. Initial manual labeling (1,780 images) trained a model to pre-annotate subsequent batches, which were then corrected by domain experts. This iterative process ensured high-quality bounding box annotations.
• Segmentation: Uses the Interactive Semi-Automatic Annotation Tool (ISAT) with Segment Anything Model 2 (SAM2) to generate pixel-wise masks, refined by experts.

D. AI Validation Framework

The authors validated the dataset using a cascading deep learning framework:

• Classification: MobileNetV3-Large (Lightweight CNN).
• Object Detection: YOLOv11-Large (CSPDarknet backbone).
• Segmentation: Mask R-CNN with ResNet-50/101 and FPN backbones.
• Evaluation: Performance was measured against five domain experts using Cohen's Kappa and inter-rater agreement heatmaps on 295 external images.

3. Key Contributions

• TomatoMAP Dataset: The first comprehensive dataset for tomatoes featuring multi-angle (4 cameras), multi-pose (12 angles), and time-series data. It contains 64,464 LR images and 3,616 HR images with over 649,000 annotated instances.
• Fine-Grained Granularity: Unlike previous datasets focusing on leaf disease or simple fruit counting, TomatoMAP provides detailed annotations for 50 BBCH growth stages and 7 specific biological ROIs, including complex structures like axillary shoots and panicles.
• Cascading Architecture: The dataset is explicitly designed to support a hierarchical AI workflow where classification informs detection, which in turn guides segmentation, mimicking a biological analysis pipeline.
• FAIR Principles: The data is machine-friendly, standardized, and publicly available via the e!DAL archive, ensuring interoperability and reuse.

4. Results

• Model Performance:
  • Classification: MobileNetV3 achieved 79.19% accuracy on 50 BBCH classes, significantly outperforming random guessing (2%).
  • Detection: YOLOv11 showed strong diagonal dominance in confusion matrices, with accuracy up to 0.96 for leaves and whole plants. Performance was slightly lower for overlapping clusters (flowers/fruits) but still robust.
  • Segmentation: Mask R-CNN achieved a best AP-50 of 63.59% (using ResNet-50, 1x schedule, 2.4e-4 learning rate). Performance was highest for mid-to-late developmental stages (mature fruits, >6mm flowers) and lower for early-stage, low-contrast objects (nascent fruits, 2mm buds).
• AI vs. Human Consistency:
  • Cohen's Kappa: The AI model demonstrated "almost perfect agreement" with human experts, comparable to the agreement level between different human experts (HI vs. HI).
  • Consistency: The AI model showed 100% internal consistency across inference replicates, whereas human annotators exhibited variability due to subjectivity.
  • Bias Reduction: The study confirms that AI reduces phenotyping bias, particularly in boundary delineation and repetitive tasks.

5. Significance

• Accelerated Breeding: By providing a scalable, high-accuracy method for phenotyping, TomatoMAP enables rapid selection of agronomically desirable genotypes based on complex traits like fruit quality and stress resistance.
• 3D Phenotyping: The multi-angle, multi-pose design preserves critical 3D structural features (topology, angles) that 2D datasets miss, offering a more realistic representation of plant architecture.
• Standardization: The use of the BBCH scale and standardized IoT protocols sets a new benchmark for reproducibility in plant science.
• Cost & Time Efficiency: The validation proves that AI can replace labor-intensive manual phenotyping without sacrificing accuracy, drastically reducing the time and cost of large-scale breeding programs.
• Resource Availability: The release of a large-scale, multi-task dataset (Classification, Detection, Segmentation) fills a critical gap in the computer vision community for agricultural applications.