TomaMMU: A Comprehensive Multimodal Understanding Benchmark for Tomato Leaf Diseases

Technical Summary: TomaMMU and TomaBench

Problem Statement
Current automated plant disease recognition systems predominantly frame diagnosis as a visual classification problem. While effective in controlled laboratory settings, these systems struggle in real-world agricultural environments characterized by visual variability, background clutter, occlusions, and inconsistent illumination. Existing benchmarks often rely on synthetic annotations or uniform laboratory backgrounds, failing to capture the complexity of field conditions. Furthermore, there is a significant gap in evaluating the multimodal understanding and reasoning capabilities of Vision-Language Models (VLMs) regarding plant pathology. Current models often rely on superficial correlations rather than robust disease representations, leading to poor generalization and a lack of reliable diagnostic reasoning required for expert-level decision support.

Methodology
The authors introduce TomaMMU (Tomato leaf disease MultiModal Understanding), a large-scale dataset and TomaBench, a hierarchical benchmark designed to evaluate VLMs on tomato disease understanding.

• Data Curation Pipeline: The dataset was constructed through a three-stage pipeline:
  1. Data Collection: Images were sourced from "in-the-wild" farm environments (LeafNet, TOM2024) and in-house collections, totaling 28,808 high-quality images. This contrasts with previous datasets that rely heavily on laboratory settings. The dataset covers 15 categories (14 diseases and 1 healthy class) and 213,119 human-annotated visual question-answer (VQA) pairs.
  2. Human Annotation: Domain experts performed rigorous manual verification, extracting structured metadata including symptom descriptions, disease types, and quantities. Mislabeled data was removed to ensure agronomic reliability.
  3. Question-Answer Generation: The pipeline generates both Multiple-Choice Questions (MCQs) and Open-Ended Questions (OEQs). MCQs are generated by creating three distractors for each fact, while OEQs require short, factual responses.
• Benchmark Taxonomy (TomaBench): The evaluation framework organizes 42,626 QA pairs into a three-level hierarchical taxonomy:
  • Basic Perception: Includes Healthy-Diseased Classification (HDC), Leaf Count (LC), and Crop Species Identification (CSI).
  • Pathology Understanding: Focuses on Symptom Identification (SI) and Pathogen Classification (PC).
  • Expert Diagnosis: Involves Disease Classification (DC) and Scientific Name Classification (SNC).
• Evaluation Strategy: The study evaluates 14 state-of-the-art VLMs (including Proprietary models like Gemini 2.0 Flash, Generative FMs like LLaVA and Qwen, and domain-specific models like BioCLIP and SCOLD) in a zero-shot setting. Additionally, the authors fine-tuned a model named TomaLLaVA (using SCOLD as the image encoder and LFM2.5 as the LLM) on the TomaMMU training set to assess the impact of domain-specific adaptation.
• Metrics: Performance is measured using Accuracy and F1 scores for MCQs. For OEQs, the study employs ROUGE-L for lexical similarity and an LLM-as-a-judge (GPT score) for semantic correctness, graded on a scale of 1–5.

Key Contributions

1. TomaMMU Dataset: A comprehensive VQA dataset containing 28,808 images and up to 213,119 VQA pairs across 14 diseases and 1 healthy category. It integrates real-world field imagery with structured, human-verified metadata.
2. TomaBench Benchmark: A hierarchical evaluation framework covering seven specific agricultural tasks (HDC, DC, PC, CSI, SNC, LC, SI) organized into three diagnostic levels (Basic Perception, Pathology Understanding, Expert Diagnosis).
3. Systematic Evaluation: A comprehensive assessment of 14 leading VLMs, revealing significant limitations in fine-grained recognition and factually grounded reasoning for agricultural tasks.
4. Domain Adaptation Demonstration: The introduction of TomaLLaVA, which demonstrates that simple fine-tuning on TomaMMU substantially narrows the performance gap, achieving near-perfect results on perception tasks and strong performance on diagnostic reasoning.

Results

• Zero-Shot Performance: Existing state-of-the-art VLMs, even proprietary models like Gemini 2.0 Flash, struggle with the benchmark. The best-performing zero-shot model achieved an average accuracy of roughly 70.3% on MCQs, with significant drops in complex tasks like Pathogen Classification (PC), Symptom Identification (SI), and Scientific Name Classification (SNC).
• OEQ Challenges: Models performed poorly on open-ended questions, with average GPT scores around 2.3–2.7 (out of 5). Error analysis categorized failures into perceptual errors (hallucinations/refusals), core misses (partial relevance but wrong diagnosis), and granularity omissions (lack of specific detail).
• Fine-Tuning Impact: Fine-tuning TomaLLaVA on the TomaMMU training set resulted in a substantial performance boost. The model achieved an average accuracy of 96.09% and an F1 score of 0.958 on MCQs, outperforming the best zero-shot model (LFM2.5-VL) by +34.8% in accuracy.
• Task-Specific Gains: While fine-tuning significantly improved performance on basic perception and disease classification, improvements on complex pathology understanding and expert diagnosis tasks were more modest, highlighting the inherent difficulty of multi-statement reasoning.

Significance and Claims
The paper claims that TomaMMU and TomaBench address a critical gap in agricultural AI by providing a standardized, real-world benchmark that moves beyond simple image classification to evaluate multimodal reasoning. The authors assert that current VLMs struggle to translate visual perception into reliable diagnostic knowledge without domain-specific adaptation. The results suggest that while general-purpose models are insufficient for expert-level agricultural diagnosis, targeted fine-tuning on high-quality, human-annotated datasets like TomaMMU can bridge this gap, enabling AI systems to support farmers and experts with accurate, context-rich disease identification. The work aims to motivate future research in domain adaptation for agriculture and provides a platform for developing more robust agricultural decision-support systems.