🤖 machine learning

Beyond Toy Benchmarks: A Systematic Evaluation of OOD Detection Methods For Plant Pathology Classification

This paper systematically evaluates six OOD detection methods on the fine-grained Plant Pathology 2021 dataset, demonstrating that energy-based fine-tuning outperforms baselines by restructuring the embedding space and calibrating scores, while also revealing practical training instabilities in constrained optimization methods that are often overlooked in standard benchmarks.

Original authors: Devesh Shah

Published 2026-05-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Devesh Shah

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot gardener to identify sick apple leaves. You show it thousands of pictures of healthy leaves, leaves with scab, leaves with rust, and so on. The robot gets really good at this. But what happens when the robot sees a picture of a flower, a car, or a texture pattern that it has never seen before?

This is the problem of Out-of-Distribution (OOD) detection. In the real world, robots shouldn't just guess; they should be able to say, "I don't know what this is."

This paper is a systematic test of six different ways to teach a robot to say "I don't know" when it encounters something strange, using a real-world dataset of apple diseases. Here is the breakdown in simple terms:

The Problem: The Overconfident Robot

Most AI models are trained like students taking a multiple-choice test. Even if the question is nonsense (like "What is the capital of Mars?" when the test is about apples), the student is forced to pick an answer. They might guess "Washington D.C." with 99% confidence just because they have to pick something.

In AI, this is called the Softmax Baseline. The model is forced to distribute 100% of its "confidence" across the known diseases. So, when it sees a flower, it might confidently say, "That's definitely a leaf with Rust!" This is dangerous because the robot thinks it knows the answer when it actually has no idea.

The Experiment: Testing Six "Safety Nets"

The researchers tested six different methods to fix this overconfidence, ranging from simple tweaks to complex training strategies. They used three types of "strange" images to test the robot:

Cars: Very different from leaves (Easy to spot).
Textures: Just patterns, no objects (Easy to spot).
Flowers: Look very similar to leaves (Hard to spot).

Here is how the six methods performed:

The Baseline (Softmax): The robot is overconfident. It fails to spot the flowers, thinking they are diseased leaves.
Independent Classifiers: Instead of forcing the robot to pick one winner, they gave it five separate "yes/no" buttons (one for each disease). If none of the buttons light up, the robot knows it's an unknown object. This was a surprisingly good, simple fix.
The "Known Stranger" (OOD Class): They explicitly taught the robot a sixth category called "Unknown." This worked perfectly, but it's unrealistic because, in the real world, you don't have a labeled box of "unknown" things to show the robot beforehand.
Outlier Exposure: They showed the robot pictures of "weird stuff" (like cars) while training and told it, "If you see this, be unsure." This helped, but the robot still got confused by flowers.
Energy-Based Fine-Tuning (The Winner): This method changes how the robot "feels" about an image. Instead of just picking a winner, it calculates an "energy score."
- Normal leaves have low energy (they feel familiar and calm).
- Strange things (like flowers) have high energy (they feel chaotic and unfamiliar).
- The Result: This method was the best. It didn't just change the scoring rule; it actually reorganized the robot's internal "brain map" (embedding space) so that strange things naturally drifted far away from the known diseases. It could tell the difference between a leaf and a flower much better than the others.
Wild Data Optimization (WOODS): This is the most realistic method. It assumes the robot is deployed in the field and starts collecting a pile of random photos (some leaves, some flowers, some rocks) without labels. It tries to learn from this messy pile.
- The Catch: While this is the most realistic scenario, it was the hardest to train. The math involved was unstable. The researchers had to add "brakes" (caps on penalty weights) to stop the training from crashing. It worked better than the baseline, but it was tricky to get right.

Key Takeaways from the Paper

Realism vs. Performance: The methods that assumed we had perfect "strange" data to show the robot during training worked best. But in the real world, we don't have that. The "Energy-Based" method was the best balance: it worked great without needing labeled "strange" data.
The Flower Problem: The paper found that flowers were actually harder for the AI to reject than cars. Even though humans see a car as totally different from a leaf, the AI's internal math sometimes found cars harder to distinguish than flowers. This shows that what looks "different" to us isn't always what looks "different" to a computer.
Training Instability: When trying to use the most advanced methods (like WOODS) on a medium-sized real-world dataset, the training process was very fragile. Small changes in settings caused the model to break. This is a practical problem that simple computer benchmarks (using tiny, perfect datasets) often miss.
Accuracy Didn't Drop: A major worry was that teaching the robot to say "I don't know" would make it worse at identifying actual diseases. The paper found that it didn't. The robot stayed just as good at identifying sick leaves while becoming much better at spotting strangers.

The Bottom Line

You can teach an AI to be humble and admit when it doesn't know something, even in a complex, real-world job like farming. The best approach found here was using "Energy-Based" scoring, which reshapes how the AI views the world to keep strange things far away from the known ones. However, the most advanced methods are currently like high-performance race cars: they work great, but they are very sensitive to how you tune the engine.

Technical Summary: Beyond Toy Benchmarks – A Systematic Evaluation of OOD Detection Methods for Plant Pathology Classification

Problem Statement

Deep learning systems for computer vision have achieved remarkable performance on benchmark tasks, yet their reliable deployment in real-world scenarios is hindered by the "closed-world" assumption. Standard classification pipelines, optimized via softmax-based objectives, force probability mass across known classes, leading models to assign high confidence to unrelated inputs (Out-of-Distribution or OOD data). While extensive literature exists on OOD detection, the majority of methods are evaluated on small, visually homogeneous datasets (e.g., CIFAR-10/100) that fail to capture the complexity of real-world deployment. Specifically, these benchmarks often lack the semantic variation, covariate shifts, and "near-OOD" cases (e.g., visually similar but distinct categories) common in domain-specific applications like agriculture. This paper addresses the gap between theoretical OOD detection and practical application by evaluating methods on the Plant Pathology 2021 dataset, a fine-grained task involving apple leaf disease classification where natural distribution shifts (e.g., flowers, non-plant objects, occlusions) are prevalent.

Methodology

The study conducts a systematic evaluation of six OOD detection methods, progressing from simple post-hoc scoring to complex constrained optimization, under varying assumptions regarding access to OOD data during training. The experiments utilize a ResNet-18 backbone fine-tuned on a 5-class apple disease subset (15,675 images).

Experimental Setup

In-Distribution (ID) Data: Five single-label disease classes (Scab, Healthy, Frog Eye Leaf Spot, Rust, Powdery Mildew).
Out-of-Distribution (OOD) Data: Three external datasets with varying semantic distance:
- Stanford Cars: Far-OOD (no visual overlap).
- Describable Textures Dataset (DTD): Far-OOD (structural difference).
- Flowers102: Near-OOD (visually similar to leaves, representing a critical failure mode).
Auxiliary Data: ImageNet-O is used for methods requiring labeled OOD data during training. Unlabeled "wild" data (a mix of ID and OOD) is used for constrained optimization methods.

Evaluated Methods

E1: Softmax Baseline: Standard cross-entropy training. OOD score is the negative maximum softmax probability.
E2: Independent Binary Classification: Replaces the softmax head with five independent sigmoid outputs trained with Binary Cross-Entropy (BCE) to remove the zero-sum competition of softmax.
E3: Explicit OOD Class: Adds a sixth output class for OOD, trained on labeled ID and OOD data. Serves as an approximate upper bound.
E4: Outlier Exposure (OE): Fine-tunes the baseline classifier with an auxiliary loss encouraging uniform predictions on labeled OOD data (ImageNet-O).
E5: Energy-Based OOD Detection:
- E5a (Post-Hoc): Uses the free energy of logits ( $E(x) = -T \log \sum e^{f_c(x)/T}$ ) as a scoring function without retraining.
- E5b (Energy Fine-Tuning): Fine-tunes the model with a margin-based energy shaping loss to explicitly separate ID and OOD energy distributions.
E6: WOODS: A constrained optimization framework that trains on unlabeled "wild" data (unknown mixture of ID and OOD) without assuming access to clean labeled OOD data. It minimizes the rate of OOD samples classified as ID, subject to constraints on ID false alarm rates and classification accuracy.

Key Results

The evaluation reveals several critical insights regarding OOD detection in real-world agricultural settings:

Performance Hierarchy: E5b (Energy Fine-Tuning) emerged as the strongest practical method, achieving the highest AUROC and lowest FPR95 across all OOD sets (including the challenging Flowers102 near-OOD set) without requiring labeled OOD data at the initial training stage.
Baseline Improvements:
- E2 (Independent BCE) significantly outperformed the Softmax baseline (E1) despite using no OOD data, suggesting that removing softmax normalization is a potent, low-cost improvement.
- E5a (Post-Hoc Energy) provided immediate gains over E1, confirming that the energy score itself is a superior metric to softmax probability.
Near-OOD Challenge: Contrary to intuition, "Far-OOD" datasets (Stanford Cars) were not consistently easier to detect than "Near-OOD" datasets (Flowers102) across all methods. This highlights that perceptual similarity does not always correlate with feature representation difficulty.
Embedding Space Analysis: Qualitative analysis of the embedding space (via cosine distance to 5-nearest neighbors) showed that E5b significantly reshapes the feature space, increasing the separation between ID and OOD distributions (Wasserstein distance increased from 0.02 in E1 to 0.07 in E5b). This suggests gains come from both a better-calibrated scoring function and a reorganized embedding space.
In-Distribution Stability: Across all methods, balanced accuracy on the ID test set remained stable (ranging 0.916–0.944), indicating that OOD-aware training does not necessarily degrade task performance.

Practical Challenges and Instabilities

The paper documents significant training instabilities when scaling constrained optimization methods to moderate-sized datasets (approx. 10k images), findings largely absent from existing literature:

Learning Rate Sensitivity: For E4, E5b, and E6, the learning rate is the most critical hyperparameter. High learning rates cause the OOD loss to dominate too early, collapsing classification performance.
WOODS Instability: The Augmented Lagrangian Method (ALM) used in WOODS proved highly sensitive. Initial attempts with standard hyperparameters caused penalty weights to grow unboundedly, leading to a 20% drop in validation accuracy. Stabilization required reducing the dual learning rate, loosening constraint tolerances, and introducing a hard cap on penalty weights—a modification not present in the original WOODS formulation.

Significance and Claims

The authors claim that principled OOD detection is achievable on real-world, domain-specific data, but that standard benchmark evaluations often fail to capture the practical challenges that emerge in deployment.

Beyond Benchmarks: The study demonstrates that methods performing well on small, homogeneous benchmarks may not translate directly to complex agricultural settings, particularly regarding near-OOD detection and training stability.
Practical Viability: Energy-based fine-tuning (E5b) is presented as a robust, practical solution that balances high detection performance with the operational constraint of not requiring labeled OOD data during initial training.
Implementation Reality: The paper emphasizes that theoretical frameworks (like WOODS) require careful, dataset-specific engineering (e.g., hyperparameter tuning, penalty clamping) to function stably on moderate-sized datasets, a nuance often overlooked in literature focused on large-scale or synthetic benchmarks.

The work serves as a reference for practitioners aiming to deploy OOD detection beyond standard benchmarks, highlighting that while OOD-aware training is feasible, it demands rigorous attention to training dynamics and hyperparameter stability in real-world contexts.