Are vision-language models ready to zero-shot replace supervised classification models in agriculture?

Technical Summary: Are Vision-Language Models Ready to Zero-Shot Replace Supervised Classification Models in Agriculture?

Problem Statement
Vision-language models (VLMs) have demonstrated strong zero-shot and few-shot capabilities in general-domain visual tasks, prompting interest in their deployment for agriculture where labeled data is often scarce or expensive. However, agricultural image understanding presents unique challenges, including high intra-class variability, fine-grained visual differences, and a strong dependence on contextual information (e.g., crop type, phenological stage, geography). While supervised, task-specific models have historically addressed these challenges, they require substantial annotation effort and struggle with generalization. It remains unclear whether current off-the-shelf VLMs are reliable enough to replace these supervised systems for critical agricultural decision support, particularly given the high cost of misclassification in management actions.

Methodology
The authors present a large-scale, systematic benchmark evaluating a diverse set of open-source and closed-source VLMs against a supervised baseline.

• Datasets: The evaluation utilized 27 datasets from the AgML collection, comprising 248,000 images across 162 classes. These cover three primary task categories: plant disease identification, pest and damage detection, and plant/weed species classification.
• Models:
  • Foundation Models: Open-source models (SigLIP2, LLaVA-NeXT, Qwen2.5-VL, Gemma-3, Deepseek-VL) and closed-source models (GPT-5 Nano, GPT-5, Gemini-3 Pro, Claude Haiku 4.5).
  • Baseline: A supervised, fine-tuned YOLO11 model served as the task-specific performance benchmark.
• Evaluation Protocols:
  • Multiple-Choice Question Answering (MCQA): Models were prompted to select from constrained label sets. Variants included options restricted to within-species classes, across-species classes, and the inclusion of a "None of the above" option to test rejection capabilities.
  • Open-Ended Question (OEQ): Models were asked to generate free-form responses describing diseases, pests, or species without predefined options.
• Metrics: Performance was measured using two complementary approaches:
  1. Exact Match Accuracy: Based on fuzzy string matching (preprocessing, substring checks, and sequence similarity) with a threshold of 0.6.
  2. LLM Judge Accuracy: A semantic evaluation using a reasoning-capable LLM (GPT-OSS-20B) to determine if the predicted response and ground truth were semantically equivalent, filtering for high-confidence matches.

Key Results

1. Supervised Baseline Superiority: The supervised YOLO11 baseline consistently achieved markedly higher accuracy than any zero-shot foundation model across all tasks.
2. Prompting Format Impact: MCQA prompting yielded substantially higher performance than OEQ. The best-performing VLM (Gemini-3 Pro) reached approximately 62% average accuracy under MCQA, whereas raw OEQ accuracy was typically below 25%.
3. Evaluation Methodology Sensitivity: The choice of scoring method significantly altered results. LLM-based semantic judging increased OEQ accuracy for top models (e.g., from 21% to 30%) and shifted model rankings, demonstrating that surface-level string matching fails to capture semantic correctness in agricultural contexts.
4. Model Performance:
   • Closed-Source: Gemini-3 Pro was the top performer overall.
   • Open-Source: Qwen-VL-72B was the best open-source model, approaching closed-source performance under constrained prompting but still trailing proprietary systems.
5. Task-Level Differences: Plant and weed species classification was consistently easier for models. Pest and damage identification remained the most challenging category, highlighting the difficulty of diagnosing stressors from single images without contextual priors.

Significance and Claims
The paper concludes that current off-the-shelf VLMs are not yet suitable as standalone agricultural diagnostic systems to replace supervised models. The persistent performance gap suggests that general-purpose multimodal capabilities do not directly translate to reliable, fine-grained agricultural recognition.

However, the authors argue for a nuanced role for VLMs in agriculture:

• Assistive Components: VLMs can function effectively when paired with constrained interfaces, explicit label ontologies, and domain-aware evaluation strategies.
• Contextual Dependence: The results underscore that agricultural diagnosis often requires more than generic recognition; it necessitates combining fine-grained perception with domain knowledge and structured decision processes.
• Evaluation Rigor: The study emphasizes that evaluation methodology (prompting format and scoring) meaningfully affects reported conclusions, urging future benchmarks to report both surface-form and semantic-judge scores with transparent auditing artifacts.

The work aims to inform practitioners considering VLM-based tools and researchers developing next-generation multimodal models, grounding these conclusions in a broad, transparent benchmark rather than speculative claims.