💻 computer science

Towards Robust Deep Learning-based Rumex Obtusifolius Detection from Drone Images

This paper addresses the challenge of detecting *Rumex obtusifolius* across different data domains by demonstrating that while traditional CNNs struggle with domain shifts between ground and drone imagery, self-supervised Vision Transformers (ViTs) achieve superior robustness and performance, supported by the release of a new UAV-based dataset named AGSMultiRumex.

Original authors: Fabian Dionys Schrag, Mehmet Ozgur Turkoglu, Konrad Schindler, Ralph Lukas Stoop

Published 2026-04-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Fabian Dionys Schrag, Mehmet Ozgur Turkoglu, Konrad Schindler, Ralph Lukas Stoop

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a dog to find a specific type of weed (called Rumex obtusifolius) in a meadow.

The Setup: Two Different Worlds
The researchers had a problem. They had a huge library of photos of this weed, but they were all taken by a robot driving on the ground, looking at the plants from a low angle, close up. This is their "Source Domain."

However, they needed the dog to find the weed from a drone flying high in the sky, looking straight down. This is their "Target Domain."

The problem is that a photo taken from a robot's wheel level looks completely different from a photo taken from a drone 12 meters up. The lighting, the angle, and the background (like fences or cars) are totally different. It's like trying to teach someone to recognize a cat by showing them only pictures of cats in a bathtub, and then asking them to identify a cat on a roof.

The Failed Attempt: The Old School Method
First, the researchers tried using standard "deep learning" models (specifically called ResNets). Think of these as very smart, but rigid, students who memorized the "bathtub cat" photos perfectly.

When they tried to use these models on the drone photos, they failed miserably. Even if they let the models "study" the new drone photos a little bit (a process called fine-tuning), the models still couldn't generalize. They were too stuck on the specific details of the ground-robot photos.

The Fix: The "Translation" Tools
To help the old-school models, the researchers tried two special techniques called Domain Adaptation.

The Analogy: Imagine trying to translate a book from English to French. The old models were trying to translate word-for-word and getting confused. The researchers added "translation tools" (Moment Matching and Maximum Classifier Discrepancy) that forced the models to look at the general shape and structure of the sentences rather than just the specific words.
The Result: This helped the old models do better, but they were still struggling. They needed a lot of extra "translation rules" to work.

The Star Player: The Vision Transformer (ViT)
Then, the researchers tried a different kind of model: Vision Transformers (specifically DINOv2 and DINOv3).

The Analogy: If the ResNet models were the rigid students who memorized the bathtub photos, the Vision Transformers are like a polyglot traveler who has already seen millions of photos of cats in bathtubs, on roofs, in trees, and in snow. Because they were trained on such a massive, diverse set of images before the researchers even started, they already understand what a "plant" or a "weed" looks like in general, regardless of the angle or lighting.
The Result: These models didn't even need the special "translation tools." They just looked at the drone photos and said, "Ah, that's a Rumex weed," with high accuracy. When the researchers gave them a tiny bit of extra study time (using a technique called LoRA to make the learning efficient), they became incredibly good at the job.

The Outcome
The researchers built a new dataset of drone photos from Swiss meadows (called AGSMultiRumex) to test this.

The old models (ResNets) needed heavy help to get a score of about 0.45 out of 1.0.
The new models (ViTs), with just a little fine-tuning, scored 0.81 out of 1.0.

The Catch (Limitations)
Even the super-smart models had trouble in a few specific cases:

Confusing Lookalikes: In some flights, big dandelions looked so much like the weed that the models got confused. Since the training photos didn't have many dandelions, the models didn't know the difference.
Weird Objects: In one flight, a car was partially mistaken for a weed. Why? Because the training photos never showed a car, so the model had no idea what a car was. It just saw a weird shape and guessed "weed."

The Bottom Line
The paper concludes that for farming tasks where you need to switch from ground robots to drones, you don't need to build a new model from scratch. Instead, you should use these massive, pre-trained "polyglot" models (ViTs). They are naturally robust enough to handle the change in perspective, making them the best tool for the job. The researchers also made their new drone dataset public so others can test their own models against it.

1. Problem Statement

The paper addresses the challenge of Domain Adaptation (DA) in agricultural computer vision. Specifically, it focuses on detecting Rumex obtusifolius (broad-leaved dock), a persistent weed in grasslands.

The Core Issue: Deep learning models trained on one dataset (Source Domain) often fail to generalize to new environments (Target Domain) due to covariate shift. In agriculture, this shift arises from differences in:
- Viewpoint: Ground-level (robot) vs. Nadir (drone/UAV).
- Sensors: Different cameras and resolutions.
- Conditions: Lighting, vegetation composition, and phenological stages.
Current Limitations: Existing models (e.g., YOLO variants) trained on specific datasets (like RumexWeeds) fail to generalize to other datasets, often dropping to mAP50 values as low as 30%. Standard fine-tuning of Convolutional Neural Networks (CNNs) on source data does not sufficiently bridge the gap to UAV imagery.

2. Methodology

Datasets

Source Domain (RumexWeeds): A dataset of high-resolution images (1920×1200 px, ~1mm GSD) captured by a ground robot (Clearpath Husky) with an oblique 75° angle. It contains 5 subsets from different locations/dates in Denmark.
Target Domain (AGSMultiRumex): A novel, publicly released dataset comprising 15 UAV flights over Swiss meadows.
- Acquisition: Zenmuse P1 camera on a DJI Matrice 300 RTK, nadir view, 12m altitude, 0.105 cm/px GSD.
- Complexity: Includes diverse vegetation (dandelions, herbs), landscape elements (fences, cars), and varying weather/lighting.

Task Reformulation

To apply established DA techniques, the authors reformulated the object detection task as an image classification task:

Tiling: Original images are split into 518×518 pixel tiles (~0.5 m²).
Labeling:
- 1 (Positive): Tile overlap with Rumex bounding box > threshold ( $r_{th}=0.1$ ).
- 0 (Negative): No overlap.
- 2 (Unclear): Partial overlap; excluded from training.
Strategy: A sliding window approach with four passes (corners) ensures full coverage and acts as implicit data augmentation.

Model Architectures

The study compares two backbone families:

Convolutional Neural Networks (ResNets): ResNet50, 101, and 152. Initialized with ImageNet weights.
Vision Transformers (ViTs): DINOv2 and DINOv3 (Small, Base, Large scales). These are pretrained with self-supervised learning (DINO framework) on massive unlabeled datasets, lacking manual annotations.

Training & Adaptation Strategies

Vanilla Supervised Training: Standard fine-tuning (Freezing vs. Unfreezing backbone layers).
Unsupervised Domain Adaptation (UDA): Applied to ResNets using Moment Matching (M2S2DA for single-source, M3SDA- $\beta$ for multi-source). This aligns the first and second statistical moments (mean and covariance) of source and target feature distributions.
Parameter-Efficient Fine-Tuning (PEFT): Applied to ViTs using LoRA (Low-Rank Adaptation). This freezes the pretrained weights and trains only low-rank decomposition matrices, reducing computational cost while enabling adaptation.

3. Key Contributions

New Dataset: Release of AGSMultiRumex, a challenging UAV-based dataset with 15 diverse flights, designed specifically to test robustness and domain shift in weed detection.
Empirical Comparison of Architectures: A systematic evaluation showing that Vision Transformers (ViTs) pretrained with self-supervised objectives (DINOv2/v3) inherently handle domain shifts better than CNNs (ResNets).
Efficiency of LoRA: Demonstration that parameter-efficient fine-tuning (LoRA) on large ViTs achieves state-of-the-art performance without the computational cost of full fine-tuning.
Analysis of Failure Modes: Detailed identification of specific challenges, such as confusion with large-leaf dandelions and false positives from non-vegetation objects (e.g., cars) due to source dataset bias.

4. Experimental Results

Performance Metrics

Performance is measured by the F1-score (specifically for the minority class, Rumex) across all 15 target flights.

Model / Strategy	Median F1 ( $\tilde{F}_1$ )	Observations
ResNet (Vanilla)	~0.15 – 0.20	Poor generalization; fails even with full fine-tuning.
ResNet + M3SDA- $\beta$	~0.45	Explicit domain adaptation improves results significantly but remains moderate.
DINOv3 (Frozen)	~0.50 – 0.60	Strong zero-shot transfer; outperforms adapted ResNets despite no fine-tuning.
DINOv3 + LoRA	~0.81	Best Performance. High precision and recall across most flights.

Key Findings

ViT Superiority: Large-scale self-supervised pretraining (DINOv3) provides rich, general-purpose representations that are intrinsically robust to domain shifts. They surpass even ResNets trained with complex DA techniques.
LoRA Effectiveness: Fine-tuning DINOv3 with LoRA (Rank 32) yields the best results ( $\tilde{F}_1 \approx 0.81$ ), proving that adapting large foundation models is feasible and highly effective for agriculture.
Precision vs. Recall Trade-off:
- ResNet + DA: High recall but low precision (many false positives).
- Frozen ViT: High precision but low recall (misses many weeds).
- ViT + LoRA: Balances both, achieving >0.8 median recall and precision.
Persistent Challenges: Certain flights (e.g., Halden ID 3, Waldegg ID 4) remain difficult due to extreme lighting, early phenological stages, or confounding species (dandelions) not present in the source data.

5. Significance and Conclusion

Paradigm Shift: The paper suggests that for agricultural vision tasks with significant domain shifts, leveraging large, self-supervised pretrained ViTs is a more effective strategy than developing complex domain adaptation techniques for traditional CNNs.
Practical Impact: It demonstrates that existing, publicly available datasets (like RumexWeeds) can be effectively reused for new contexts (UAVs) if the right architecture (ViT) and adaptation method (LoRA) are chosen, reducing the need for massive, task-specific labeled datasets.
Future Directions: While architecture is crucial, the study highlights that dataset bias (e.g., lack of confounding species like dandelions or cars in the source) remains a bottleneck. Future work must focus on curating more diverse training data to handle edge cases and rare confounders.

In summary, the authors successfully bridge the gap between ground-based and aerial weed detection, achieving a robust F1 score of 0.81 by combining self-supervised Vision Transformers with parameter-efficient fine-tuning, while releasing a valuable new dataset for the research community.