Improving Reliability in Plant Disease Semantic Segmentation by Quantifying Benchmark Contamination and Duplicate Family Exposure
This paper audits the PlantSeg benchmark to reveal that 10.7% of images suffer from cross-split visual family contamination, which artificially inflates model performance by 11–15%, and demonstrates that constructing group-disjoint splits can effectively restore evaluation independence without sacrificing disease class coverage.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the fields of agriculture and computer vision, researchers are teaching artificial intelligence to see plant diseases. The goal is to help farmers spot infections early, measure how severe they are, and protect crops before a small spot of rot becomes a lost harvest. To do this, scientists build software that looks at photos of leaves and draws precise outlines around the sick parts, a process known as semantic segmentation. But for these programs to be trustworthy, they must be tested on images they have never seen before. If a computer learns to recognize a disease by memorizing a specific photo and then gets tested on that same photo, it will get a perfect score without actually learning anything useful. This is why scientists rely on benchmarks: carefully curated collections of images split into a training set for learning and a test set for checking the work. The entire system depends on the assumption that the test images are truly independent, representing new, unseen situations rather than recycled versions of the training data.
A team of researchers recently turned their attention to PlantSeg, a large and widely used collection of plant disease images that has become a standard for testing these AI models. They suspected that the test images might not be as independent as everyone thought. Instead of just checking if two files were identical copies, they looked for "visual families"—groups of images that came from the same original scene, the same leaf, or the same plant, even if they were cropped, rotated, or slightly altered. They found that about 10.7 percent of the images in the benchmark were related in this way, with some appearing in the training set and others in the test set. This meant that the computer was essentially being tested on variations of what it had already studied, giving it an unfair advantage.
To understand how much this hidden connection inflated the scores, the researchers ran a series of careful experiments. They took four different types of AI models and re-evaluated them, this time removing the related images from the test set to create a "clean" version that truly represented new data. The results were consistent: every model scored lower on the clean test set than on the original, contaminated one. The drop in performance was significant, with one popular model seeing its accuracy fall by nearly three percentage points. The researchers used statistical controls to prove that this drop wasn't just because the test set became smaller or because rare diseases were removed; the loss of performance was directly tied to the removal of the familiar, related images. This confirmed that the original benchmark scores were overly optimistic, painting a rosier picture of how well these models could handle real-world, unseen fields.
The team then investigated exactly how the computer was using these related images. They designed an experiment where they trained models on data that either included or excluded the related images, while keeping the number of diseases and the total amount of data exactly the same. When they tested these models on completely new, unrelated images, the results were the same regardless of whether the model had seen the related images during training. However, when they tested the models on images that were related to the training data, the models that had seen the related images performed dramatically better, gaining an advantage of between 11 and 15 percent. This proved that the models weren't just getting better at general plant disease recognition; they were specifically memorizing the visual patterns of those particular families of images. The "intelligence" was narrow, relying on familiarity rather than true understanding.
Recognizing that simply deleting these related images would hurt the ability to test rare diseases, the researchers proposed a better solution. Instead of throwing away data, they rebuilt the benchmark from the ground up. They treated every visual family as a single unit and assigned the entire group to either the training set or the test set, ensuring that no family was split between the two. This "group-disjoint" approach kept all 115 types of plant diseases represented in the test set while guaranteeing that the test images were truly independent. They tested this new structure with a standard model and found it worked perfectly, producing reliable scores without sacrificing the ability to study rare conditions.
The study concludes that the reliability of plant disease AI depends on how carefully we check for these hidden connections. The original PlantSeg benchmark, while valuable, contained a significant amount of data leakage that made models look more capable than they truly were. By auditing for visual families rather than just duplicate files, the researchers showed that the path to trustworthy AI in agriculture requires a more rigorous approach to data separation. The solution isn't to discard data, but to organize it so that the computer is forced to learn the general rules of disease, rather than just memorizing the specific pictures it has already seen. This ensures that when these tools are deployed in a real field, they can truly recognize a new disease on a new leaf, rather than just recognizing a familiar face.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.