Does Your VFM Speak Plant? The Botanical Grammar of Vision Foundation Models for Object Detection

1. Problem Statement

Vision Foundation Models (VFMs) and Open-Vocabulary Detectors (OVDs) promise zero-shot object detection without task-specific training data. However, their application in agricultural environments faces significant hurdles:

• Complexity: Agricultural objects (flowers, pods, leaves) are small, visually similar to surrounding structures, and vary by growth stage.
• Prompt Sensitivity: VFM performance is highly sensitive to text prompt construction. Standard class names (e.g., "flower") often yield poor results.
• Annotation Barrier: Traditional fine-tuning (e.g., LoRA) requires labeled datasets and GPU resources, which are often unavailable to agricultural end-users.
• The Gap: There is a lack of systematic understanding regarding how to construct optimal prompts for VFMs in agriculture without manual annotation or fine-tuning.

2. Methodology

The authors propose a systematic prompt optimization framework that decomposes text prompts into interpretable linguistic axes and optimizes them without labeled real-world data.

A. Prompt Decomposition (The "Botanical Grammar")

Prompts are decomposed into eight axes:

1. Taxonomy: Specificity of the species (e.g., cowpea, legume, vigna unguiculata).
2. Color: Visual descriptors (e.g., yellow, purple).
3. Anatomy: Structural details (e.g., open petals, stamens).
4. Phenology: Growth stage (e.g., bud, blooming).
5. Negation: Excluding confusers (e.g., not a leaf).
6. Size: Relative scale (e.g., tiny, large).
7. Grammar: Sentence structure (e.g., a single, close-up of).
8. Emoji: Visual tokens (e.g., 🌸, 🌻).

B. Two-Phase Optimization Pipeline

1. Phase 1: One-Factor-at-a-Time (OFAT) Analysis:
   • The system varies one axis while keeping others at baseline to isolate the impact of each component on four models: YOLO World, SAM3, Grounding DINO, and OWLv2.
   • This identifies the most influential axes for each specific architecture.
2. Phase 2: Combinatorial Optimization:
   • Top-performing values from Phase 1 are combined programmatically.
   • The pipeline generates base combinations, appends negation clauses to top candidates, and finally appends emoji tokens to the single best prompt.
   • Background Regularization: For YOLO World, an empty string ("") is added as a background class to suppress false positives.

C. Generalization via LLM

• LLM-Guided Translation: To test cross-task generalization, the authors use an LLM (Qwen-4B) to translate the optimal axis structure from cowpea flowers to cowpea pods.
• The LLM preserves the structural logic (e.g., "anatomy descriptor" or "negation") but updates the specific values (e.g., changing "petals" to "peas") to fit the new target object.

D. Datasets & Metrics

• Data: Synthetic data generated via Helios plant simulation (500 flower images, 499 pod images) and real field data (268 flower images, 186 pod images).
• Metric: mAP@0.5 (mean Average Precision at IoU 0.5) is used instead of mAP@0.5:0.95, as accurate detection counts are prioritized over precise bounding box localization in this context.
• Baseline: Supervised YOLOv11 trained on real data serves as the upper bound.

3. Key Contributions

1. Model-Specific Prompt Divergence: The study reveals that there is no universal "best" prompt. Different architectures respond to different linguistic features (e.g., OWLv2 is highly sensitive to phenology terms like "closed bud," while Grounding DINO prefers bare nouns).
2. Synthetic-to-Real Transfer: The framework demonstrates that prompts optimized entirely on synthetic data transfer effectively to real-world field imagery, often matching or exceeding prompts discovered via optimization on labeled real data.
3. Annotation-Free Optimization: The pipeline achieves substantial performance gains over naive baselines without requiring any manual annotation or model fine-tuning.
4. LLM-Augmented Discovery: Using an LLM to translate prompt axes uncovered non-obvious, high-value descriptors (e.g., "mottled" for pods) that manual translation would likely miss.

4. Key Results

• Performance Gains:
  • YOLO World: +0.357 mAP@0.5 gain on synthetic flower data (from 0.141 baseline to 0.498 with optimized prompt + emoji).
  • OWLv2: +0.362 mAP@0.5 gain on synthetic flower data.
  • SAM3: Achieved 0.649 mAP on synthetic data with optimized prompts.
• Cross-Task Generalization (Flowers to Pods):
  • Prompts optimized on synthetic flower data transferred to real pod detection.
  • SAM3: The LLM-translated prompt ("a single mottled pea pod...") achieved 0.429 mAP on real pod data, significantly outperforming the manually structured prompt (0.100) and the naive baseline (0.039).
• Synthetic vs. Real Optimization:
  • For the majority of model-object combinations, prompts derived from the synthetic pipeline performed as well as or better than those derived from real-data optimization (e.g., YOLO World flower: 0.374 synthetic vs. 0.353 real).
• Model Sensitivity:
  • Grounding DINO showed the least sensitivity to prompt engineering, often performing best with simple nouns ("flower").
  • OWLv2 showed extreme sensitivity; changing "flower" to "closed bud" increased mAP from near-zero to 0.348 on real data.

5. Significance and Implications

• Democratizing AI in Agriculture: This work provides a pathway for domain experts (farmers, agronomists) to deploy state-of-the-art VFMs without needing ML engineering expertise, labeled datasets, or expensive GPU clusters for fine-tuning.
• Synthetic Data Viability: It validates the use of synthetic plant simulation for tuning prompt engineering strategies, reducing the dependency on costly field data collection for initial model setup.
• Understanding VFM "Reasoning": The findings suggest that VFMs encode visual concepts in specific, non-intuitive ways (e.g., preferring "closed bud" over "flower" due to pretraining scale biases). Understanding these "botanical grammars" is crucial for effective deployment.
• Limitations & Future Work: The paper notes that confidence score miscalibration remains a bottleneck. VFMs produce uncalibrated scores across different prompts, making it difficult to set a universal threshold for deployment without some proxy data. Future work should focus on generalizing this framework to more crop species and developing self-calibrating confidence mechanisms.

In conclusion, the paper establishes that rigorous, domain-aware prompt engineering is a critical prerequisite for unlocking the zero-shot capabilities of VFMs in complex agricultural settings, offering a practical, annotation-free alternative to traditional fine-tuning.