💻 computer science

Beyond Accuracy Rankings: Cross-Dataset Stability and Computational Cost in Plant Disease Classification

This study evaluates the trade-off between predictive performance and computational efficiency of ResNet-50, MobileNetV2, and EfficientNet B0 across internal and external plant disease datasets, demonstrating that while ResNet-50 achieves the highest accuracy, MobileNetV2 offers superior throughput and lower latency, suggesting that model selection should balance both metrics rather than relying solely on accuracy rankings.

Original authors: Nitin Kumar, Sakshi Dua

Published 2026-11-10

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Nitin Kumar, Sakshi Dua

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the quiet corners of a farm, the difference between a bountiful harvest and a ruined crop often comes down to a single, timely decision: identifying a sick leaf before the infection spreads. For centuries, this task has relied on the trained eye of a farmer or an agricultural expert, a skill honed by experience but vulnerable to fatigue, similar symptoms between different diseases, and the chaotic variables of the natural world. To bring a steadier hand to this work, scientists have turned to computer vision, teaching machines to recognize plant diseases by showing them thousands of photographs. These digital systems, built on deep learning, have become remarkably good at spotting illness when tested on the very images they were taught to recognize. However, a critical question remains: if a system learns to diagnose a disease in a perfectly lit laboratory, will it still work when the light is harsh, the background is cluttered with weeds, or the photo is taken with a different camera? The reliability of these tools depends not just on how well they memorize their training data, but on how they hold up when faced with the messy reality of the field.

This is the central puzzle tackled by researchers Nitin Kumar and Dr. Sakshi Dua from Lovely Professional University. They set out to look beyond the simple ranking of which computer model gets the highest score on a test. Instead, they wanted to understand how three popular types of artificial intelligence architectures—ResNet-50, MobileNetV2, and EfficientNet-B0—perform when the rules of the game change. They trained these models on a massive, curated collection of over 70,000 images of healthy and diseased leaves, a dataset known as the New Plant Diseases Dataset. This collection is clean and uniform, much like a photo studio. Once the models were trained, the researchers did not stop there. They took the same models and tested them on a completely different set of images called PlantDoc, which consists of photos taken in real-world fields with uneven lighting and complex backgrounds. The goal was to see if the models could transfer what they learned from the clean studio to the chaotic field, and to measure how much computing power each model required to do the job.

The results revealed a stark and instructive divide between performance in the lab and performance in the wild. When tested on the clean, internal data they were trained on, all three models performed with high competence, correctly identifying the vast majority of leaves. ResNet-50, the largest and most complex of the three, achieved the highest accuracy, correctly classifying over 96 percent of the images. However, when the researchers switched to the real-world PlantDoc images, the performance of all three models plummeted. The accuracy for the best-performing model dropped to just over 21 percent. This dramatic fall highlights a common pitfall in artificial intelligence: a system that excels in a controlled environment often struggles when the background, lighting, or image quality changes. The models had likely learned to rely on subtle cues from the clean backgrounds of their training data, cues that were absent in the field photos.

Yet, the story does not end with a simple failure. When the researchers looked deeper, using different ways to measure success that accounted for how well the models handled each specific type of disease rather than just the overall average, a different picture emerged. While ResNet-50 still had the highest raw accuracy on the field images, the MobileNetV2 model, which is designed to be lightweight and fast, actually performed better at identifying the specific disease categories correctly. It achieved a higher score in a metric that treats every disease type as equally important, suggesting it was more consistent across the board. This finding challenges the assumption that the most powerful or complex model is always the best choice. In this case, the model built for efficiency showed a surprising resilience in the face of real-world unpredictability.

Beyond accuracy, the study placed a heavy emphasis on the cost of running these models. In the world of agriculture, where computers might be running on solar-powered devices in remote fields, speed and memory usage matter as much as intelligence. The researchers measured how long it took each model to analyze a single image and how many images it could process in a second. Here, MobileNetV2 was the clear winner. It required the least amount of memory, processed images the fastest, and could handle more than 200 images per second. In contrast, ResNet-50, while slightly more accurate on the internal test, was significantly slower and required ten times more memory to run. EfficientNet-B0 fell somewhere in the middle, offering a balance but not the extreme efficiency of MobileNetV2. The data suggests that for practical applications, where resources are limited and speed is essential, the smaller, faster model might be the more viable option, even if it is not the absolute champion in a perfect laboratory setting.

The researchers were careful to note the limits of their findings. The external test used a relatively small number of images, and the specific hardware used to measure speed might not perfectly reflect the performance on every mobile phone or edge device. They also pointed out that the drop in performance was likely due to a mix of factors, including changes in background, lighting, and image quality, rather than a single cause. They did not claim to have solved the problem of cross-dataset failure, but rather provided a clear, controlled comparison that shows how different models behave under pressure. The study concludes that choosing the right tool for plant disease classification requires looking at more than just a single accuracy number. It demands a balance between how well a model predicts, how consistently it handles different types of diseases, and how efficiently it uses the computing resources available to it. In the end, the best model is not necessarily the one with the highest score on a test, but the one that offers the most reliable and efficient performance when the lights go out and the work begins in the field.

Technical Summary: Beyond Accuracy Rankings: Cross-Dataset Stability and Computational Cost in Plant Disease Classification

Problem Statement
While Convolutional Neural Networks (CNNs) have demonstrated high accuracy in plant disease classification on specific, controlled datasets (e.g., PlantVillage), their performance often degrades when applied to images captured under realistic, uncontrolled field conditions. Furthermore, model selection in agricultural applications is frequently driven solely by internal validation accuracy, neglecting two critical factors: (1) the stability of performance across different datasets (cross-dataset generalization) and (2) the computational cost required for deployment. Existing literature often lacks controlled comparisons of different architectures using identical protocols to evaluate these trade-offs.

Methodology
This study employs a comparative experimental design to evaluate three distinct CNN architectures: ResNet-50, MobileNetV2, and EfficientNet-B0. All models utilize a frozen-feature transfer learning approach, where ImageNet-pretrained backbones are fixed, and only the classification head is trained.

Datasets:
- Source (Training/Internal Validation): The "New Plant Diseases Dataset (Augmented)" containing 70,295 images across 38 classes. The data was split 80:20 (56,236 training, 14,059 validation) using a seeded random split.
- Target (External Evaluation): A mapped subset of the PlantDoc dataset, consisting of 146 images from 17 classes that could be mapped to the source vocabulary. This dataset represents field conditions with variable backgrounds and lighting.
Training Protocol: Models were trained using the Adam optimizer (learning rate 0.001) and categorical cross-entropy loss. Data augmentation (random crops, flips, rotations, color jitter) was applied only to the training set.
Evaluation Metrics:
- Predictive Performance: Accuracy, Macro-F1, and Weighted-F1 scores were calculated for both internal and external evaluations.
- Computational Efficiency: Measured via total parameter count, batch-one inference latency (ms), throughput (images/second), and peak GPU memory usage.
- Statistical Analysis: Pairwise McNemar tests were conducted to determine if differences in prediction correctness between models were statistically significant.

Key Contributions
The paper makes three primary contributions to the field of agricultural AI:

Standardized Comparison: It provides an equal comparison of ResNet-50, MobileNetV2, and EfficientNet-B0 using a common frozen-feature transfer learning protocol, isolating architectural differences from training strategy variations.
Cross-Dataset Evaluation: It contrasts source-dataset internal validation against external evaluation on a mapped PlantDoc subset, highlighting the necessity of testing models on data distributions different from the training set.
Multi-Criteria Analysis: It evaluates architectures based on complementary metrics, explicitly analyzing the trade-offs between predictive performance (accuracy/F1) and computational efficiency (latency/throughput/parameters).

Results

Internal Validation: On the source dataset, ResNet-50 achieved the highest performance with 96.33% accuracy and a 96.22% Macro-F1 score. MobileNetV2 and EfficientNet-B0 followed with slightly lower scores.
Cross-Dataset (PlantDoc) Evaluation: Performance dropped significantly for all models on the external PlantDoc subset.
- Accuracy: ResNet-50 achieved the highest accuracy (21.23%), followed by MobileNetV2 (20.55%) and EfficientNet-B0 (18.49%).
- Macro-F1: MobileNetV2 outperformed the others with a Macro-F1 of 22.48% (as reported in Table III), while ResNet-50 scored 20.93% and EfficientNet-B0 17.82%.
- Statistical Significance: McNemar tests revealed no statistically significant differences in prediction correctness between the models on the external subset ( $p > 0.05$ ).
Computational Efficiency:
- MobileNetV2 demonstrated the superior efficiency profile: it had the fewest parameters (2.27 million), the lowest measured inference latency (4.97 ms), and the highest throughput (201.2 images/s).
- ResNet-50 was the most resource-intensive, with 23.59 million parameters and 6.59 ms latency.
- EfficientNet-B0 fell between the two in terms of parameters (4.06 million) but had the highest latency (8.16 ms) and lowest throughput (122.6 images/s) among the three.

Significance and Claims
The authors conclude that model rankings are highly dependent on the evaluation metric and the dataset used. A model that excels in internal validation (ResNet-50) does not necessarily maintain that dominance in external, real-world scenarios, nor does it offer the best computational efficiency.

The study asserts that model selection should not rely solely on internal validation accuracy or parameter count. Instead, a multi-criterion evaluation combining predictive performance (including Macro-F1 for class balance) and computational cost is essential for choosing appropriate CNN architectures for plant disease classification. The results suggest that MobileNetV2 offers a compelling balance of efficiency and robustness for field deployment, despite ResNet-50's superior internal accuracy.

Limitations
The paper explicitly acknowledges several limitations:

The external evaluation was restricted to a small subset (146 images, 17 classes), limiting generalizability to other farms, camera systems, or disease types.
Computational benchmarks were conducted in a specific CUDA environment without specifying the exact GPU model, meaning latency and throughput figures may not directly translate to mobile or edge devices.
The study does not isolate the specific causes of performance degradation (e.g., background bias, illumination, annotation differences) but rather documents the phenomenon of cross-dataset instability.
The source data split was random rather than stratified, and the potential for duplicate images between training and validation sets was not fully explored.

Technical Summary: Beyond Accuracy Rankings: Cross-Dataset Stability and Computational Cost in Plant Disease Classification

More like this