🤖 machine learning

Multi-Scale ViT Inference with Habitat-Fit Priors and kNN Retrieval for Multi-Species Plant Identification

This paper presents DS@GT ARC's third-place solution for the PlantCLEF 2026 challenge, which achieves robust multi-species plant identification in high-resolution quadrat images by combining a multi-scale DINOv2 ViT classifier with FAISS kNN retrieval, geographic habitat priors, and temporal fusion, while demonstrating that alternative training-centric approaches yielded no performance gains.

Original authors: Alper Erten, Murilo Gustineli, Adrian Cheung

Published 2026-07-17

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Alper Erten, Murilo Gustineli, Adrian Cheung

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery, but instead of a crime scene, your "crime scene" is a square patch of wild meadow, roughly the size of a small dining table. Your job is to name every single type of plant growing there. Now, imagine the twist: the only training manual you have ever seen contains pictures of single, perfect flowers or leaves, isolated on a white background. You've never seen a messy, crowded meadow in your training data. This is the challenge of multi-species plant identification: figuring out who is who in a chaotic crowd when you've only ever studied individuals in isolation.

To tackle this, scientists use Vision Transformers (ViTs), which are like super-smart digital eyes that break an image into tiny puzzle pieces to understand what they are seeing. They also use k-Nearest Neighbors (kNN), a method that works like a "look-alike" search engine: if you show it a picture of a leaf, it finds the most similar pictures in its database and says, "Hey, these look just like that one!" Finally, they use priors, which are just educated guesses based on the rules of the world, like knowing that cacti don't grow in the Arctic or that certain mosses only like high mountains. This paper tells the story of a team that combined these tools to solve the "messy meadow" puzzle, turning a chaotic jumble of green into a precise list of species.

The Detective's Toolkit: How the Team Solved the Messy Meadow

The team, known as DS@GT ARC, entered the PlantCLEF 2026 competition, a high-stakes contest to identify plants in high-resolution photos of 0.5-meter square plots. These photos are huge—about 3,000 by 3,000 pixels—making them too big to feed into a standard computer brain all at once. The team's solution wasn't about building a bigger, more complex brain; it was about being a smarter detective who knows how to look at the clues.

The Strategy: Zooming In and Out
Imagine trying to identify a crowd of people by looking at the whole group from a helicopter. You might see a sea of heads, but you can't tell who is wearing a red hat. If you zoom in too close, you only see one person and miss the context. The team's first trick was Multi-Scale Tiling. They chopped each giant photo into smaller squares (tiles) at different zoom levels: some zoomed out (3x3 grid), some zoomed in (6x6 grid). This gave them 86 different "views" of the same meadow. They ran their main AI model, a fine-tuned DINOv2 ViT-L/14 (a powerful digital eye trained on millions of images), over every single tile. Then, they used a "max pooling" rule: if any tile spotted a specific plant with high confidence, the whole photo got credit for that plant. This ensured they didn't miss tiny, hidden flowers just because they were small in the big picture.

The "Look-Alike" Search Engine
The AI model is great, but sometimes it gets confused. To help it, the team added a kNN Retrieval System. Think of this as a massive library of 860,000 plant photos. When the AI looks at a tile, it also asks the library, "Who does this look like?" The library returns the top 20 most similar images. If the AI is unsure, but the library says, "This looks exactly like a Geum reptans," the team blends that advice into the final answer. It's like having a second opinion from a seasoned botanist who has seen millions of plants.

The "Habitat Fit" Rule: The Ultimate Reality Check
The most powerful part of their solution wasn't a new algorithm, but a set of rules based on geography. They realized that just because a plant could be in a photo doesn't mean it should be. They introduced Habitat-Fit Demotion.

Geography: If a photo was taken in South-Western Europe, they automatically lowered the chances of plants that only grow in the tropics or the Arctic.
Altitude: If the photo was taken at sea level, they demoted plants that only grow on high mountain peaks.
This acted like a filter, quietly whispering to the AI, "Hey, that cactus is unlikely here; maybe you're seeing a rock instead." This step alone made the biggest difference in their accuracy, proving that knowing where you are is just as important as knowing what you see.

The Time Traveler: Temporal Fusion
The test photos weren't just one-off snapshots; many were taken of the same spot at different times of the year. The team used this to their advantage. If a spot was visited in spring and summer, and the spring photo showed a flower that the summer photo missed (because it had died back), they combined the evidence. It's like solving a mystery by checking the suspect's alibi from three different days. If the evidence is consistent across visits, they trust it more. If the visits look very different (like a lush summer vs. a bare winter), they use a special "similarity" check to make sure they aren't getting tricked by the changing seasons.

What Didn't Work (And Why That Matters)

The team didn't just guess; they tried many fancy ideas and threw them away when they didn't work. This is just as important as the success stories.

The "Synthetic" Trap: They tried to create fake training data by mixing and matching digital plant parts (pseudo-quadrats) to teach a new type of decoder. It failed completely. The AI got confused because the fake data didn't look like the real, messy meadows.
The "Cross-Region" Mistake: They tried to teach the AI using a massive dataset of European landscapes (LUCAS) that wasn't the same as the competition's specific plants. The AI learned the wrong lessons and performed worse.
The "Segmentation" Glitch: They tried to use a tool that cuts out individual plants from the background (SAM 3) to get cleaner images. It didn't help because the AI was trained on grid squares, not on these weirdly shaped cutouts. The mismatch caused more noise than signal.

The Final Score

The team's final system, which combined the multi-scale zooming, the look-alike search, and the strict habitat rules, achieved a macro-F1 score of 0.43902 on the private leaderboard, earning them third place in the competition. Interestingly, one of their unselected configurations actually scored even higher (0.45777), but they couldn't see that score until after the competition ended.

The paper suggests that for this specific, messy problem, the best approach isn't necessarily a more complex neural network. Instead, the winners were the ones who carefully designed how to look at the data (multi-scale), used the right "cheat codes" (geographic and altitude priors), and knew when to ignore fancy new tricks that didn't fit the real-world rules. They proved that sometimes, the smartest move is to listen to the environment as much as the image.

Technical Summary: Multi-Scale ViT Inference with Habitat-Fit Priors and kNN Retrieval for Multi-Species Plant Identification

Problem Statement
This paper addresses the PlantCLEF 2026 challenge, a multi-species plant identification task requiring systems to predict every species present in high-resolution (≈3000×3000 pixel) top-down photographs of 0.5×0.5 m vegetation quadrats. The task presents two primary difficulties:

Severe Single-to-Multi-Label Domain Shift: Training data consists of ~1.4 million single-label images of isolated plant specimens (often close-ups of specific organs), whereas test data comprises cluttered, multi-species ground-cover scenes with varying scales and phenological stages.
Resolution Constraints: Test images are too large for standard Vision Transformer (ViT) inputs (typically 518×518), necessitating tiling and aggregation strategies that do not lose fine-grained species cues.

The evaluation metric is a hierarchical macro-averaged sample $F_1$ score, averaged across transects to mitigate bias from over-sampled sites.

Methodology
The proposed pipeline, developed by the DS@GT ARC team, is an inference-driven approach centered on a fine-tuned DINOv2 ViT-L/14 backbone. The system processes data through the following stages:

Multi-Scale Tile Aggregation:
- Each quadrat image is center-cropped and resized to multiple scales ( $s \in \{3, 4, 5, 6\}$ ), generating 86 non-overlapping 518×518 tiles per image.
- A fine-tuned ViT-L/14 classifier (trained on the PlantCLEF dataset) processes each tile, producing a 7,806-dimensional softmax distribution.
- Per-image predictions are aggregated via max pooling across all tiles and scales, preserving the strongest local evidence for each species.
FAISS kNN Retrieval Ensembling:
- To complement the classifier, a k-Nearest Neighbors (kNN) retriever is built using FAISS.
- Embeddings are generated using the ViT-L backbone augmented with a LoRA adapter trained with an ArcFace objective to cluster same-species embeddings.
- For each tile, the top-20 nearest neighbors are retrieved from a database of ~860,000 training embeddings. Their similarities are converted into a soft class distribution and blended with the classifier's output.
- Blending weights are visit-aware: $\beta=0.70$ for multi-visit quadrats and $\beta=1.00$ (kNN disabled) for single-visit quadrats to avoid noise from unreliable retrievals.
Source-Aware Temporal Fusion:
- Leveraging the test set's multi-visit structure, predictions for quadrats sharing the same location identifier are fused.
- Aggregation strategies vary by data source: max pooling is used for most sources, while similarity-weighted aggregation is applied to the RNNB source (coastal sand dunes) to handle high temporal variability (e.g., seasonal changes) without overconfident predictions.
Habitat-Fit Demotion and Geographic Masking:
- Habitat-Fit: A post-processing step rescales probabilities based on geographic and altitude priors derived from the training data. A "fit factor" penalizes species whose known distribution (latitude/longitude centroid and elevation band) does not match the specific source's ecological regime.
- Geographic Mask: A global mask removes ~3,000 species whose training centroids fall outside South-Western Europe (France, Spain, Italy, Switzerland).
- Admission Rule: A two-stage thresholding process retains high-probability classes and enforces a minimum prediction count for specific sources where the model tends to under-predict.

Exploratory Directions (Null Results)
The paper explicitly reports three complementary directions that yielded no performance improvement:

TileQ-Decoder: A label-as-query transformer decoder trained on synthetic pseudo-quadrats in CLS embedding space failed to outperform simple aggregation, suffering from "FAISS-undersampling under-confidence."
Cross-Region Transformer: A noisy-student distillation approach using the unlabeled LUCAS dataset failed due to distributional mismatch between the teacher's pseudo-labels and the target domain.
SAM Instance-Crop Augmentation: Using Segment Anything Model (SAM 3) to generate instance-aware crops did not improve performance, likely due to a distributional mismatch between segmented crops and the grid tiles used for calibration.

Results

Performance: The final selected submission achieved a private leaderboard macro- $F_1$ of 0.43902, placing third overall. An unselected configuration of the same pipeline achieved a higher private score of 0.45777.
Ablation Studies:
- Habitat-Fit Demotion: The single largest contributor to performance (+0.04075 $F_1$ ), highlighting the value of ecological priors.
- Multi-Scale Aggregation: Combining scales 3, 4, 5, and 6 outperformed single-scale or larger scale sets (e.g., up to 8), suggesting a balance between detail capture and noise reduction.
- Backbone Scaling: The ViT-L/14 backbone consistently outperformed the ViT-B/14 by +0.022 to +0.052 $F_1$ across all configurations.
- kNN Retrieval: Provided a significant boost (+0.01256 $F_1$ ) when blended appropriately.
- Temporal Fusion: Provided a modest but consistent gain (+0.00367 $F_1$ ), heavily dependent on source-specific tuning.

Significance and Claims
The paper claims that for large-scale multi-species recognition under severe domain shifts, inference-time strategies and dataset-aware priors can outperform complex training-centric pipelines.

Dataset Structure over Model Complexity: The most significant gains were achieved not by increasing model size or complexity, but by explicitly encoding the dataset's structure (multi-visit temporal patterns, geographic/altitude distributions, and source-specific ecological regimes).
Limitations of External Data: The null results from the LUCAS distillation and SAM augmentation suggest that naive incorporation of external data or augmentations can degrade performance if they introduce distributional drift relative to the target domain.
Aggregation vs. Decoding: The failure of the TileQ-Decoder reinforces that for this specific task, preserving and aggregating cached per-tile probabilities is more effective than re-processing features through a learned multi-label decoder.

The work demonstrates that a carefully tuned inference pipeline, leveraging multi-scale tiling, retrieval-based ensembling, and ecological priors, is a robust solution for automating biodiversity assessment in vegetation quadrats.

The Detective's Toolkit: How the Team Solved the Messy Meadow

What Didn't Work (And Why That Matters)

The Final Score

Technical Summary: Multi-Scale ViT Inference with Habitat-Fit Priors and kNN Retrieval for Multi-Species Plant Identification

More like this