Using Vision Language Foundation Models to Generate Plant Simulation Configurations via In-Context Learning

This paper introduces a novel framework utilizing vision-language foundation models (Gemma 3 and Qwen3-VL) to automatically generate JSON simulation configurations for digital twin agriculture by interpreting drone imagery, demonstrating their potential to scale functional-structural plant modeling while highlighting current limitations in visual reasoning and reliance on contextual priors.

The Gist

Imagine you are a master chef who wants to recreate a specific dish you saw in a photograph. But instead of just guessing the ingredients, you need to write a perfect, computer-readable recipe (a JSON file) that a robot chef can follow to grow that exact dish in a virtual garden.

This paper is about teaching AI chefs (specifically Vision Language Models) to do exactly that: look at a photo of a real cowpea field and automatically write the "recipe" (configuration file) needed to build a perfect 3D digital twin of that field.

Here is a breakdown of their journey, using some everyday analogies:

1. The Problem: The "Manual Entry" Bottleneck

In agriculture, scientists use Digital Twins (virtual copies of real fields) to run "what-if" experiments. For example, "What happens if we water this crop less?"

• The Old Way: To make a digital twin, a human expert has to manually measure the photo, count the plants, guess the sun's angle, and type all that data into a complex code file. It's like trying to build a Lego castle by reading the instructions one brick at a time while blindfolded. It's slow, boring, and hard to scale.
• The Goal: Can we just show the AI a photo and say, "Build me the code for this," and have it do it instantly?

2. The Experiment: The "Training Camp"

The researchers created a massive synthetic training camp.

• The Fake Garden: They used a 3D simulator to generate thousands of fake cowpea fields. They knew the "ground truth" (the exact recipe) for every single one.
• The Test: They took photos of these fake fields and asked different AI models (like Gemma and Qwen) to look at the photo and write the recipe back.
• The Teaching Method (In-Context Learning): They tried five different ways to teach the AI:
  1. Zero-Shot: Just saying, "Write a recipe in JSON format." (Like telling a student, "Write a story," with no examples).
  2. With a Template: Giving them a blank form to fill out.
  3. With Examples: Showing them three photos and their correct recipes.
  4. With Visual Examples: Showing them photos and recipes side-by-side.
  5. With Cheat Sheets: Giving them hints like, "There are 14 plants, and the sun is at 60 degrees." (This is like giving the student the answer key for the easy parts so they can focus on the hard parts).

3. The Results: The "Smart but Flaky" Student

The results were a mix of "Wow!" and "Oops."

• The Good News: The AI got surprisingly good at the basics. It could count the plants and guess the sun's position reasonably well, especially when given the "Cheat Sheet" (grounding info). It learned that if the plants look small, they are young; if they look big, they are older.
• The Bad News: The AI struggled with the "secret sauce."
  • The "Hallucination" Trap: Sometimes, when the AI couldn't clearly see a detail in the photo, it didn't say, "I don't know." Instead, it made things up to fit the format. It's like a student who, when they don't know the answer to a math problem, just writes down a random number that looks like it fits the pattern.
  • The "Copycat" Problem: When the AI was given examples (few-shot learning), it sometimes just copied the numbers from the examples instead of looking at the new photo. It relied on the context rather than the image.
  • The "Blind" Surprise: In a weird twist, when they tested the AI by hiding the photo and just asking it to guess based on the text prompt, it sometimes did better than when it had the photo. This suggests that for some tasks, the AI was just memorizing the "average" of the training data rather than actually "seeing" the image.

4. The Real World Test: From Fake to Real

They took their best-performing AI and showed it a photo of a real cowpea field (not a fake one).

• The Gap: The AI did okay, but it made more mistakes than on the fake data. Real photos are messy (shadows, dirt, weird angles), while the fake photos were perfect.
• The Visuals: When they used the AI's generated recipe to build the 3D model, the result looked like a cowpea field, but sometimes the plants were in the wrong rows or the wrong size. It was like a chef who got the ingredients right but arranged them on the plate in a slightly weird way.

5. The Takeaway: A Promising Prototype

This paper is the first time anyone has tried to use AI to go straight from "Photo" to "3D Simulation Code" for plants.

• The Analogy: Think of this as teaching a robot to be a landscape architect. Right now, the robot is an intern. It can draw a decent sketch if you give it a ruler and a template, but it still needs a human to check the details.
• The Future: The authors say that to make this truly useful, we need to give the AI better "training materials" (more detailed examples) and maybe teach it to admit when it's unsure, rather than guessing.

In short: They built a bridge between looking at a photo and building a 3D simulation. The bridge is a bit wobbly right now, but it's the first time anyone has tried to build it, and it proves that the AI can learn the job—it just needs a little more practice and better instructions.

Technical Summary

Here is a detailed technical summary of the paper "Using Vision Language Foundation Models to Generate Plant Simulation Configurations via In-Context Learning."

1. Problem Statement

Functional-Structural Plant Models (FSPMs) are powerful tools for simulating biophysical processes in agriculture (e.g., photosynthesis, water use) and creating "Digital Twins." However, their deployment at scale is hindered by:

• High Complexity: FSPMs require detailed 3D representations of fields, including plant positions, canopy structure, and environmental parameters.
• Low Throughput: Manually configuring simulation parameters (often in structured formats like JSON or XML) is time-consuming and requires expert knowledge.
• The Gap: While Vision Language Models (VLMs) have advanced in image analysis and text generation, their ability to directly translate remote sensing images (e.g., drone orthophotos) into complex, structured simulation configuration files (JSON) for 3D plant reconstruction has not been tested.

2. Methodology

A. Dataset Generation

The authors created a synthetic benchmark and a real-world validation set:

• Synthetic Dataset: Generated using the Helios 3D procedural plant library.
  • Source: Real-world drone orthophotos of cowpea (Vigna unguiculata) plots were used to extract spatial features.
  • Process: An image processing pipeline (ExG vegetation index, OpenCV blob detection) extracted plant counts and locations. These were combined with randomized biophysical parameters (leaf pigments, soil data) to generate 1,120 synthetic images across 4 Days After Planting (DAP) stages (10, 30, 50, 70, 90).
  • Output: Each image corresponds to a ground-truth JSON configuration file containing metadata, environment settings, field layout, plant properties (PROSPECT model parameters), and camera settings.
• Real Dataset: Drone orthophotos from a 2025 cowpea breeding experiment in California. Ground truth was manually annotated for plant count and location.

B. Vision Language Models (VLMs)

The study evaluated state-of-the-art open-source VLMs:

• Models: Gemma 3 (4B, 12B, 27B parameters) and Qwen3-VL (4B, 8B, 30B parameters).
• Fine-tuning: A parameter-efficient fine-tuning (PEFT) using LoRA was performed on the Qwen3-VL 32B model (141.9M trainable parameters) using 1,788 synthetic images.

C. In-Context Learning (ICL) Strategies

Five distinct prompting methods were tested to generate JSON configurations from images:

1. Zero-shot: Basic role definition and format restriction ("Output as JSON").
2. Schema-guided: Added a JSON schema to define keys and data types.
3. Few-shot (Text): Added 3 examples of JSON outputs with reasoning steps.
4. Few-shot (Multimodal): Added 3 image-JSON pairs to the context.
5. Grounding: Added explicit metadata hints (e.g., exact plant count, sun position, coordinate conversion formulas) to the prompt.

D. Evaluation Metrics

Performance was assessed across three categories:

1. JSON Integrity: Syntax error rate, key-missing rate, and BLEU-4 score (similarity to ground truth).
2. Geometric Evaluations: Mean Absolute Error (MAE) for Days After Planting (DAP), plant count, and Chamfer Distance for plant spatial localization.
3. Biophysical Evaluations: MAE for leaf pigments (chlorophyll, carotenoids, anthocyanins), water mass, and leaf structure.

3. Key Results

• Impact of Grounding: Providing explicit grounding information (Method 5) significantly reduced errors across all metrics. It acted as a "checkpoint," allowing models to bypass complex visual inference for known variables.
• Model Performance:
  • Qwen3-VL generally outperformed Gemma 3, particularly in geometric tasks (DAP and plant location), even with smaller parameter counts (e.g., Qwen 4B vs. Gemma 27B).
  • Fine-tuning: LoRA fine-tuning improved performance on baseline prompts but did not always outperform the original model when rich context (few-shot examples) was provided.
• Geometric vs. Biophysical:
  • Models successfully estimated structural metadata (plant count, sun azimuth) but struggled with biophysical traits (leaf pigments), showing high MAE regardless of model size.
  • DAP Estimation: Errors generally decreased as model size increased, but adding few-shot examples did not consistently lower MAE.
• Sim-to-Real Gap:
  • When applied to real drone images, models exhibited higher syntax errors and key-missing rates compared to synthetic data.
  • Visual Hallucination: Without grounding, models sometimes generated incorrect plot structures (e.g., double-row plants instead of single-row).
• Ablation Study (Blind Baseline): When the target image was removed from the prompt, the model sometimes achieved lower error rates than when the image was present (specifically for plant count and sun elevation). This indicates the models often rely on contextual priors (distributions from few-shot examples) rather than genuine visual inference, treating the image as noise when visual cues are ambiguous.

4. Key Contributions

1. First Benchmark of its Kind: This is the first study to utilize VLMs to generate structural JSON configurations for 3D plant simulations directly from images.
2. Synthetic-to-Real Framework: Introduced a scalable pipeline using procedural generation (Helios 3D) to create a high-fidelity synthetic dataset for training and evaluating VLMs in agriculture.
3. Comprehensive ICL Analysis: Systematically evaluated how different in-context learning strategies (from zero-shot to grounding) affect the generation of complex, multi-modal simulation parameters.
4. Identification of Limitations: Highlighted the "contextual bias" phenomenon where VLMs default to prompt priors rather than visual evidence, a critical insight for deploying AI in scientific simulation.

5. Significance and Future Work

• Scalability: This approach offers a pathway to automate the creation of Digital Twins for agriculture, potentially enabling large-scale "what-if" experiments without manual parameterization.
• Current Limitations: Current VLMs cannot yet match human-annotated ground truth or basic computer vision pipelines in accuracy, particularly for biophysical traits.
• Future Directions: The authors suggest improving accuracy by:
  • Extending context windows (e.g., to 128K tokens) to include more detailed examples.
  • Providing domain-specific references (e.g., color books for leaf pigments).
  • Further fine-tuning on curated synthetic datasets.

In conclusion, while VLMs show promise in interpreting structural metadata for digital twins, they currently rely heavily on provided context and struggle with pure visual inference for complex biophysical parameters. The proposed framework provides a foundation for bridging the gap between remote sensing imagery and actionable 3D simulation configurations.