Multi-label Instance-level Generalised Visual Grounding in Agriculture

This paper introduces gRef-CW, the first benchmark dataset for generalised visual grounding in agriculture that includes negative expressions, and proposes Weed-VG, a modular framework designed to overcome the domain gap and effectively localise crop and weed instances under challenging field conditions.

The Gist

Imagine you are a farmer standing in a massive, crowded field. You have a robot assistant that can see everything, but right now, it's a bit clumsy. If you ask it, "Find the big corn plant in the top left corner," it might point to a weed, or it might get confused because there are hundreds of tiny plants that look almost identical.

This paper is about teaching that robot to become a super-precise field guide. The authors, researchers from James Cook University, realized that while AI is great at answering questions about pictures (like "What is in this photo?"), it's terrible at pointing to specific things based on a description, especially in messy, real-world farms.

Here is the breakdown of their solution, explained simply:

1. The Problem: The "Needle in a Haystack" Issue

In a normal photo, a computer can easily find a "cat" or a "car." But in a farm field:

• Everything looks the same: A tiny weed looks just like a tiny crop seedling.
• Sizes vary wildly: Some plants are huge, while others are smaller than a pixel.
• The "Ghost" Problem: Sometimes you ask for something that isn't there (e.g., "Find the pumpkin in this field of corn"). Old AI models would just guess and point to a random plant, saying, "Here it is!" even if it wasn't there.

The researchers found that existing AI models were failing miserably at this. They needed a new way to train and test these robots.

2. The New Tool: gRef-CW (The "Farm Dictionary")

To fix the AI, you need better practice material. The authors created a massive new dataset called gRef-CW.

• Think of it as a giant flashcard deck: It contains over 8,000 high-resolution photos of real farms.
• The Annotations: They didn't just label "corn" or "weed." They wrote thousands of specific sentences like, "The small weed in the bottom right" or "No crops are present here."
• The Twist: Crucially, they included "negative" cards. These are sentences describing things that aren't in the picture. This teaches the AI to say, "I don't see that," instead of guessing.

3. The Solution: Weed-VG (The "Smart Detective")

They built a new framework called Weed-VG to solve the problem. Imagine a detective solving a crime, but instead of a crime, it's finding a specific plant. The detective uses a two-step process:

Step A: The "Is it Even Here?" Check (Existence Detection)

Before the detective tries to find the specific suspect, they first ask: "Is this person even in the building?"

• If the answer is No, the detective stops immediately and says, "Not found." This prevents the AI from pointing at random weeds when you asked for a specific crop that isn't there.
• If the answer is Yes, the detective proceeds to Step B.

Step B: The "Which One?" Check (Instance Ranking)

Now that they know the target exists, the detective looks at all the candidates. They use a special scoring system to rank them:

• Word vs. Sentence: They check if the specific words match (e.g., "tiny") AND if the whole sentence makes sense (e.g., "in the top left").
• The "Interpolation" Trick: Because plants can be tiny or huge, the AI uses a mathematical "stretching" technique. Imagine trying to fit a small puzzle piece into a big hole; this method gently smooths out the edges so the AI doesn't get confused by the size difference.

4. The Results: From Clumsy to Precise

When they tested their new "Smart Detective" (Weed-VG) against the old AI models:

• Old Models: They were like a drunk person in a crowd, pointing at random people and saying, "That's him!" even when the person wasn't there. They got about 10-30% of the answers right.
• Weed-VG: It got over 62% of the answers right. More importantly, when asked to find something that wasn't there, it correctly said "No" 78% of the time (compared to less than 3% for the old models).

The Big Picture

This paper is a huge step forward for Precision Agriculture.

• Why it matters: If a robot can accurately find only the weeds and ignore the crops, farmers can spray herbicides only on the weeds. This saves money, reduces chemical pollution, and helps the environment.
• The Analogy: Before this, the robot was like a toddler who sees a red ball and a red apple and thinks they are the same. Now, with Weed-VG, the robot is like a seasoned gardener who can spot the difference between a sprouting crop and a weed, even in a crowded, messy garden, and knows exactly when to say, "I don't see what you're looking for."

In short, they built a new "textbook" for farm robots and a new "brain" that teaches them to look before they leap, ensuring they don't accidentally spray the good crops while trying to kill the weeds.

Technical Summary

Here is a detailed technical summary of the paper "Multi-label Instance-level Generalised Visual Grounding in Agriculture."

1. Problem Statement

The paper addresses the critical gap in Visual Grounding (VG) within the domain of Precision Agriculture. While Vision-Language Models (VLMs) have advanced in tasks like image captioning and Visual Question Answering (VQA), their ability to localize objects based on natural language queries (VG) remains unexplored and ineffective in agricultural settings.

Key Challenges Identified:

• Domain Gap: Existing VG models (e.g., GroundingDINO, MDETR) fail in agricultural imagery due to significant domain shifts. They struggle with the high visual similarity between crops and weeds, extreme scale variations (from tiny seedlings to large plants), and crowded scenes.
• Absence of Targets: Standard VG assumes the referred object exists. In agriculture, a query might refer to a weed that is not present in a specific field image. Current models often hallucinate objects or fail to correctly identify "no target" scenarios.
• Lack of Benchmarks: There was no dedicated dataset for Generalised Visual Grounding (gVG) in agriculture that includes negative expressions (queries for non-existent objects) and instance-level annotations.

2. Key Contributions

A. The gRef-CW Dataset

The authors introduce gRef-CW, the first generalised referring-expression dataset for crop and weed visual grounding.

• Scale: Contains 8,034 high-resolution field images with over 78,000 crop/weed instances and 82,000 annotations.
• Annotation Complexity:
  • Instance-level: 78,288 expressions describing specific plants (e.g., "Medium weed in the top right").
  • Image-level: 4,304 expressions describing the scene as a whole.
  • Negative Expressions: A unique feature where the dataset includes queries for objects that are absent from the image (generated via "Replace" and "Swap" strategies).
• Characteristics: Captures extreme scale variation (tiny to large), high scene density (up to 30+ instances per image), and fine-grained ambiguity between 8 crop types and 1 weed class.

B. The Weed-VG Framework

The authors propose Weed-VG, a modular framework designed to integrate with existing grounding models (specifically GroundingDINO) to handle agricultural challenges. It introduces two core innovations:

1. Hierarchical Relevance Scoring (HRS):

   • Level 0 (Existence Detection): A global classifier determines if the referred object exists in the image at all.
   • Level 1 (Instance Relevance): If the object exists, the model ranks candidate regions.
   • Constraint Enforcement: A logical constraint ensures that instance localization is only performed if the global existence score is positive. This prevents the model from hallucinating objects when they are absent.
   • Multi-level Similarity: Combines sentence-level and word-level text similarities with a learnable weight to refine relevance scores.

2. IoU-Driven Interpolation (InterpIoU):

   • Addresses the issue of extreme scale variation (objects occupying <0.01% of the image).
   • Standard IoU losses produce unstable gradients for tiny objects. The authors introduce an auxiliary loss term using a linearly interpolated bounding box ($B_{int} = (1-\alpha)B_{pred} + \alpha B_{gt}$) to provide smooth, non-zero gradients, stabilizing training for small instances.

3. Distance and Size-Aware Matching:

   • A custom matching cost function that penalizes not just IoU overlap, but also center distance and relative size discrepancies, crucial for distinguishing similar-looking plants of different sizes.

3. Methodology Details

• Architecture: The framework uses a pre-trained GroundingDINO (Swin Transformer backbone + BERT text encoder) as the proposal generator.
• Training Strategy:
  • Two-Stage Training:
    1. Stage 1: Fine-tune the last decoder layer and box head using the InterpIoU loss to improve spatial localization.
    2. Stage 2: Freeze the backbone and train the projection layers and HRS module using hierarchical labels (existence + instance relevance) for 60 epochs.
  • Loss Function: A unified loss combining Hierarchical Multi-label Constraint Enforcement (HMCE) and InterpIoU. The HMCE loss dynamically weights the existence and instance losses based on image type (e.g., prioritizing existence detection for empty images).
• Data Augmentation: Uses Copy-Paste augmentation to increase instance diversity, alongside random rotations and color jittering.

4. Experimental Results

The framework was evaluated on gRef-CW against state-of-the-art baselines (MDETR, GroundingDINO-T/L, SAM3).

• Overall Performance:

  • Weed-VG achieved a Top-1 Accuracy of 62.42% and mIoU of 57.25% on the test set.
  • This significantly outperforms the best baseline (SAM3), which scored 34.88% Top-1 and 32.76% mIoU.
  • Crucially, Weed-VG improved both retrieval (Recall@0.5: 55.44%) and ranking, whereas baselines often improved one at the expense of the other.

• Scale Robustness:

  • Baselines collapsed on "Tiny" instances (Top-1 < 2%).
  • Weed-VG achieved 54.66% Top-1 on tiny crops, reducing the performance gap between tiny and large instances from ~58 points (in baselines) to ~17 points.

• Negative Expression Handling (Neg-Acc):

  • This metric measures the ability to correctly identify when a target is absent.
  • Baselines performed near random chance (Neg-Acc ~3–7.5%).
  • Weed-VG achieved 78.35% Neg-Acc, demonstrating that the hierarchical constraint effectively prevents hallucination in "no-target" scenarios.

• Ablation Studies:

  • Removing the Hierarchical Constraint caused Neg-Acc to drop from 78.35% to 41.60%, proving its necessity for handling absent targets.
  • Removing InterpIoU dropped mIoU by ~9.5%, confirming its importance for small object localization.
  • Joint use of sentence and word-level features was shown to be superior to using either alone.

5. Significance and Conclusion

This paper represents a significant step forward in Precision Agriculture and Generalised Visual Grounding.

• Practical Impact: The ability to accurately localize specific weeds while distinguishing them from crops, and to correctly identify when a weed is not present, is vital for automated selective weeding, reducing herbicide use, and optimizing resource management.
• Scientific Contribution: It establishes a new benchmark (gRef-CW) that highlights the limitations of current VLMs in complex, real-world environments.
• Methodological Innovation: The Weed-VG framework demonstrates that decomposing grounding into "Existence Detection" and "Instance Ranking," coupled with scale-aware regression, is a robust strategy for handling the unique challenges of agricultural imagery (small objects, high density, and visual ambiguity).

The authors conclude that while current models struggle with the "long-tail" distribution of agricultural objects, the proposed modular approach provides a clear baseline for future development in agricultural AI.