📄 plant biology

OneGrow: Unified Temporal Plant Image and Mask Generation

OneGrow is a unified latent flow-matching model that jointly generates temporal wheat images and their organ-level segmentation masks, enabling multi-task synthesis and consistent long-term forecasting by leveraging a shared autoencoder and pseudo-labels from a large-scale dataset.

Original authors: Boss, M., Volpi, M., Roth, L.

Published 2026-09-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Boss, M., Volpi, M., Roth, L.

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

Farmers and scientists have long relied on cameras to watch crops grow, hoping that a simple photograph could reveal how healthy a plant is or how much grain it will eventually produce. This practice, known as phenotyping, turns images into data about plant traits. However, raw photos tell only part of the story. To truly understand a plant's development, researchers need to know exactly which parts are leaves, which are stems, and which are the seed heads, and they need to see how these parts change over time. Getting this level of detail is incredibly difficult. Manually drawing outlines around every leaf and stem in thousands of photos is slow and expensive work. Furthermore, tracking the same plants day after day requires complex equipment to ensure the images line up perfectly, a task few research platforms can manage consistently. As a result, there is a shortage of high-quality data that pairs clear photos of crops with precise maps of their internal structures across an entire growing season.

To solve this problem, a team of researchers has developed a new artificial intelligence system called OneGrow. Instead of trying to analyze existing photos, this system learns to create them. It is a generative model, a type of computer program that learns the patterns of real data and then produces new, realistic examples that follow those same rules. OneGrow is unique because it does not just make pictures; it makes pairs of pictures and their corresponding structural maps simultaneously. It learns from wheat crops, a staple food for much of the world, and it understands how the plant changes from day to day. By training on a mix of single-day photos and multi-year time-lapse sequences, the system learns to imagine what a wheat field looks like at any stage of growth, complete with a detailed breakdown of its leaves, stems, and heads.

The core of this achievement is a unified approach that treats the photo and the map as two sides of the same coin. In previous attempts, models could either predict how a plant would look in the future without knowing its structure, or they could generate a map for a single day without understanding the passage of time. OneGrow bridges this gap. It uses a shared digital representation for both the image and the mask, allowing the system to switch roles effortlessly. If a researcher provides a photo, the system can draw the map. If they provide a map, the system can paint a realistic photo. If they provide a few days of photos, the system can predict what the plant will look like days later. This flexibility comes from a mechanism the researchers call a "reveal," which acts like a set of instructions telling the computer which parts of the scene are already known and which parts it needs to invent.

The training process was a careful balancing act. The researchers fed the system two different types of data. The first was a massive collection of single-day wheat photos, where the computer had to guess the structure of the plant. Since human experts had not labeled every single one of these, the team used a separate, highly accurate AI tool to create "pseudo-labels," or computer-generated maps, to serve as the ground truth. The second data source was a multi-year dataset containing thousands of images of the same wheat plots taken over six years. This allowed OneGrow to learn the rhythm of the seasons, understanding how a small seedling transforms into a tall, dense crop. By combining these two sources, the model learned to handle the wide variety of lighting and weather conditions found in real fields while also grasping the slow, steady progression of plant growth.

When tested, the system demonstrated a remarkable ability to generate coherent sequences. In one experiment, researchers gave the model a few days of photos and asked it to forecast the next few days. The system produced images that showed the canopy growing denser and the plants maturing, all while keeping the structural maps consistent with the new images. In another test, the model took a simple color-coded map of a plant's parts and generated a full-color photograph that matched the layout perfectly, placing leaves and stems exactly where the map indicated. The system even managed to fill in missing parts of an image, taking a small patch of a photo and expanding it into a full scene that remained true to the surrounding context. These results were not just visually convincing; they were quantitatively strong. When the researchers compared the generated maps against real human-drawn maps, the system achieved a high level of accuracy, particularly for leaves and seed heads, though it found the thin stems slightly more challenging to define.

The implications of this work extend beyond simply creating pretty pictures. Because the system can generate realistic images paired with perfect structural maps, it offers a way to create endless training data for other AI tools. Researchers can use these synthetic pairs to teach other computers how to identify diseases or count seeds, covering growth stages and weather conditions that are rare or missing in real-world datasets. The model also opens the door to translating abstract plant growth simulations into realistic visuals, allowing scientists to see how different genetic traits or environmental changes might look in the field. While the system is currently specialized for wheat and relies on computer-generated labels for its training, it represents a significant step toward a future where we can simulate and understand crop development with a depth and speed that was previously impossible. The work suggests that by teaching machines to see the structure within the image, we can unlock a more complete understanding of how our food grows.

Technical Summary: OneGrow

Problem Statement

Image-based crop phenotyping relies on paired data containing both imagery and organ-level segmentation masks to study plant development effectively. However, such data is scarce due to two primary bottlenecks:

Annotation Cost: Organ-level segmentation is labor-intensive, meaning labeled data covers only a small fraction of collected imagery.
Temporal Scarcity: Capturing the same plots repeatedly over a season to track development is operationally difficult and rarely undertaken.

Existing generative models for plants generally fail to address both needs simultaneously. Some synthesize temporal imagery without structural labels, while others generate labeled image-mask pairs but lack a temporal dimension. To our knowledge, no plant-focused model jointly generates organ-labeled imagery over time, particularly for crops like wheat where observations are often sparse rather than densely sampled.

Methodology: OneGrow

OneGrow is a unified latent flow-matching model designed to jointly model wheat images and their organ-segmentation masks over time. It leverages a "reveal" mechanism to handle various generation tasks within a single architecture.

Core Architecture

Shared Latent Representation: OneGrow utilizes a single, frozen image Variational Autoencoder (VAE) to encode both RGB images and color-coded segmentation masks. Masks are converted to RGB images using a fixed palette before encoding. This allows the model to treat images and masks as different "views" within the same latent space without needing modality-specific encoders.
Flow-Matching Transformer: The model employs a transformer backbone (DiT) that processes all latent patches of a "window" simultaneously. It predicts a joint velocity field to transport noise to clean data along a linear interpolant path.
View Collection: A training window is defined as a collection of views indexed by modality (image or mask) and frame (time). Each view consists of non-overlapping patches.
The Reveal Mechanism: A binary reveal mask ( $r$ $r$ ) is applied per patch.
- $r=1$ : The patch is revealed as clean context ( $\tau=1$ ).
- $r=0$ : The patch is generated ( $\tau=\tilde{\tau}$ ).
  This mechanism allows the model to handle spatially partial context (e.g., some patches observed, others generated) and defines the specific task (segmentation, synthesis, forecasting) based on which views are revealed.

Training Strategy

Data Sources: The model is trained jointly on two sources:
1. GWFSS: A large single-frame dataset with broad appearance variation. Human-annotated masks are held out for evaluation; the rest are pseudo-labeled by a pre-trained segmentation model.
2. FIP1: A multi-year temporal dataset containing aligned growth windows of winter wheat.
Pseudo-Labeling: Since human annotations are scarce for temporal data, the model trains on pseudo-labels generated by a state-of-the-art segmentation model (ViT-Adapter/Mask2Former).
Loss Function: The model minimizes a reveal-weighted mean squared error (MSE) on the velocity field, focusing only on generated patches.
Temporal Roll-out: For sequences longer than the training window, a sliding-window roll-out is used. Each new frame is generated autoregressively, conditioned on the preceding frames and the target mask, ensuring temporal consistency.

Conditioning

The model accepts metadata conditioners, including the acquisition domain and per-frame photometric statistics (exposure and color), represented as register tokens. Classifier-free guidance (CFG) is applied during sampling.

Key Contributions

Unified Model: OneGrow is a plant-focused model that jointly generates organ-labeled imagery over time, unifying tasks like segmentation, synthesis, and forecasting.
Pseudo-Label Training: It demonstrates effective joint training on a diverse single-frame source and a multi-year temporal source using pseudo-labels, reducing the dependency on expensive human annotations.
Flexible Reveal Mechanism: The per-patch reveal system allows a single model to perform diverse tasks (image-to-mask, mask-to-image, spatial completion, forecasting) simply by changing the configuration of observed vs. generated content.

Results

The model was evaluated on held-out human-annotated data (GWFSS) and temporal data (FIP1).

Segmentation (Image-to-Mask): OneGrow achieved a mean Intersection-over-Union (mIoU) of 0.697 on the GWFSS test set, compared to 0.740 for the original challenge winner. The performance gap was most notable in the "stem" class, which is thin and difficult to segment.
Mask Adherence (Cycle Consistency): When generating images from masks and re-segmenting them, OneGrow achieved a cycle-consistency mIoU of 0.749, matching the challenge winner's performance on real images (0.740). This indicates the model places organs accurately according to the conditioning mask.
Image Generation Quality:
- On GWFSS, OneGrow outperformed a retrieval baseline (Mask-NN) in paired LPIPS (0.426 vs. 0.548) but lagged in set-level metrics (FD-DINOv2, CMMD), likely because the baseline retrieves exact real training images.
- On FIP1, OneGrow outperformed the baseline in CMMD and LPIPS, with comparable FD-DINOv2 scores.
Qualitative Tasks: The model successfully demonstrated temporal forecasting (continuing growth trajectories), mask-guided spatial completion (outpainting), and joint generation of image-mask pairs.

Significance and Limitations

Significance:
OneGrow provides a method to generate realistic, organ-labeled image sequences, which can augment training data for segmentation and recognition tasks, particularly for under-represented growth stages or imaging conditions. It bridges the gap between process-based crop models (which provide structural masks) and realistic imagery.

Limitations:

Pseudo-Label Bias: The model inherits biases from the teacher segmentation model, particularly regarding thin stems.
Domain Specificity: The model is trained exclusively on wheat; transfer to other crops is untested.
Inference Cost: As an iterative generative model, it is more computationally expensive at inference than dedicated segmentation networks.
Evaluation Scope: The temporal mask-conditioned generation is currently supported only by qualitative results; quantitative validation for this specific capability is pending.
Performance Gap: OneGrow does not surpass the dedicated segmentation challenge winner in pure image-to-mask accuracy, as this was not the primary optimization target (mask-conditioned synthesis and forecasting were emphasized).