💻 computer science

AgroVG: A Large-Scale Multi-Source Benchmark for Agricultural Visual Grounding

This paper introduces AgroVG, a large-scale multi-source benchmark comprising over 10,000 image-query pairs across six agricultural target families to evaluate visual grounding as a generalized set prediction task, revealing significant performance gaps in current models' ability to handle small, occluded, and variable object counts in agricultural scenarios.

Original authors: Haocheng Li, Juepeng Zheng, Zenghao Yang, Kaiqi Du, Guilong Xiao, Gengmeng Pu, Haohuan Fu, Jianxi Huang

Published 2026-05-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Haocheng Li, Juepeng Zheng, Zenghao Yang, Kaiqi Du, Guilong Xiao, Gengmeng Pu, Haohuan Fu, Jianxi Huang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot gardener how to work in a massive, messy field. You want to give it simple instructions like, "Pick only the ripe apples on the left," or "Spray the weeds, but ignore the healthy crops."

For a long time, AI researchers have been good at teaching robots to recognize what is in a picture (e.g., "That is an apple"). But they haven't been very good at teaching robots to listen to specific instructions about where things are, especially when there are dozens of similar things crowded together, or when the thing you asked for isn't there at all.

This paper introduces AgroVG, a new "test drive" (benchmark) designed to see if AI models can actually follow these complex gardening instructions.

Here is a breakdown of what they did, using simple analogies:

1. The Problem: The "Needle in a Haystack" Challenge

In a normal photo, asking an AI to "find the cat" is easy. But in a farm field:

The Haystack: There might be 50 wheat heads that look almost identical.
The Needle: You might ask for "the three tallest wheat heads on the right."
The Trap: Sometimes you ask for "the red apples," but there are no red apples in the picture. A smart robot should say, "I don't see any," but a dumb robot might hallucinate and point at a green leaf anyway.

Existing tests didn't check if robots could handle these three tricky situations at once: finding one specific item, finding many specific items, or correctly saying "none" when the item is missing.

2. The Solution: The AgroVG "Driving Test"

The authors built a massive test bank called AgroVG. Think of it as a giant, standardized driving test for agricultural robots.

The Size: It contains over 10,000 test questions.
The Variety: It covers six different "driving scenarios" (families): crops vs. weeds, fruits, wheat heads, pests (bugs), plant diseases, and tree canopies.
The Two Levels of Difficulty:
- Level 1 (The Box): The robot draws a square box around the target. This is like saying, "The target is somewhere in this square."
- Level 2 (The Mask): The robot draws a precise outline around the target. This is like saying, "The target is exactly this shape, no more, no less." This is much harder because leaves and bugs have weird, jagged shapes.

3. How They Tested It

They didn't train the robots on this test. Instead, they took 26 different AI models (including big famous ones like GPT-4 and specialized open-source models) and asked them to take the test "cold" (zero-shot).

They asked questions like:

"Find the biggest bug." (Single target)
"Find all the weeds on the left." (Multiple targets)
"Find the blue flowers." (When there are no blue flowers in the picture).

4. The Results: The Robots Struggled

The results were a bit of a wake-up call. Even the smartest AI models failed to pass the test with flying colors.

The "Set" Problem: When asked to find all the weeds, the robots usually found the biggest, most obvious ones but missed the smaller, hidden ones. They were like a student who only answers the easy questions and skips the rest.
The "Hallucination" Problem: When asked to find something that wasn't there (like "find the red apples" in a picture of green grass), many robots confidently drew boxes or masks around random green leaves. They were too eager to please and invented targets that didn't exist.
The "Precision" Problem: When asked to draw a precise outline (Level 2), the robots were often sloppy. They might draw a mask that covered the whole plant instead of just the diseased leaf, or they missed the edges entirely.

The Bottom Line: The best model only got about 35% of the "find multiple items" questions right, and less than 17% of the "precise outline" questions right.

5. Why This Matters (According to the Paper)

The paper argues that before we can trust robots to go into a field and spray pesticides or pick fruit based on voice commands, we need a way to measure if they are actually listening and seeing correctly.

AgroVG is that measuring stick. It shows us that while AI is getting better at "seeing" pictures, it is still very bad at "listening" to complex instructions in messy, real-world environments. It highlights that we need robots that are not just good at spotting things, but also good at knowing when nothing is there, and good at counting and grouping things correctly.

In short: We built a very tough exam for farm robots. They took the exam, and they mostly failed. This tells us we have a long way to go before we can let them work alone in the fields.

Technical Summary: AgroVG: A Large-Scale Multi-Source Benchmark for Agricultural Visual Grounding

Problem Definition

Agricultural AI systems require the capability to ground natural-language instructions in complex crop environments to enable tasks such as selective weeding, disease monitoring, and targeted harvesting. While visual grounding (localizing objects described by text) has advanced in general domains, existing benchmarks fail to address the specific challenges of agricultural imagery. These challenges include:

Dense, Repetitive Instances: Single frames often contain dozens of visually similar targets (e.g., wheat heads, clustered fruits), making the assumption of a unique target per query invalid.
Small, Occluded, and Irregular Targets: Pests, disease lesions, and leaves often have non-convex boundaries and are partially hidden, rendering bounding boxes insufficient for pixel-level evaluation.
Target-Absent Expressions: Realistic field instructions may refer to objects that are not present in the image, requiring models to abstain rather than hallucinate predictions.

Current agricultural benchmarks either lack instance-level grounding protocols, are restricted to single-source datasets, or do not evaluate the joint requirements of localization accuracy, set completeness, and existence-aware abstention.

Methodology

The authors introduce AgroVG, a multi-source benchmark formulated as a generalized set-prediction problem. Given an image and a referring expression, the model must return all matching target instances (one, many, or none) as either bounding boxes or instance masks.

Dataset Construction

Scale and Sources: AgroVG comprises 10,071 annotation-grounded image-query pairs derived from 10 source datasets (e.g., PhenoBench, CropAndWeed, GWHD, IP102, PlantSeg).
Target Families: The benchmark covers six agricultural target families: crop/weed, fruit, wheat head, pest, plant disease, and tree canopy.
Annotation Grounding: Every query is traceable to verified instance annotations. Positive queries are linked to specific target sets, while target-absent queries are verified to have empty target sets within the benchmark's family space.
Tasks:
- T1 (Bounding-Box Grounding): Supported across all six target families. Models predict sets of axis-aligned bounding boxes.
- T2 (Instance-Mask Grounding): Supported on three families with reliable instance-level pixel annotations (crop/weed, fruit, plant disease). Models predict binary masks.
Query Regimes: Both tasks evaluate three regimes:
- Single-target: $K=1$ instance.
- Multi-target: $K>1$ instances.
- Target-absent: $K=0$ (no matching instance exists).
Quality Control: The construction pipeline includes audit-driven sampling, expert visual review to remove corrupted images or misaligned annotations, and split-aware query generation to prevent data leakage.

Evaluation Protocol

The evaluation uses task-specific metrics to diagnose failures across localization, set completeness, and abstention:

T1 Metrics: Uses Set-F1 (based on maximum-cardinality matching at IoU thresholds 0.50 and 0.75) for multi-target queries, Strict Accuracy for single-target queries, and Empty Accuracy for target-absent queries.
T2 Metrics: Uses Query-Level Mask Coverage (mIoU, cIoU, Dice) for positive queries and Empty Accuracy for target-absent queries. This approach avoids per-instance mask matching in favor of evaluating the union of referred regions.

Key Results

The authors conducted a zero-shot evaluation of 26 model configurations, including closed-source MLLMs (e.g., GPT-4o, GPT-5.4), open-source VLMs (e.g., InternVL3.5, Qwen3-VL), and specialized grounding systems (e.g., OWLv2, Grounding DINO, LISA, SAM).

Performance Gaps: The benchmark reveals significant limitations in current models.
- Set Completeness: The best multi-target Set-F1 score on T1 is only 0.3540 (InternVL3.5-8B), indicating models struggle to retrieve all valid instances in dense scenes.
- Mask Grounding: The best T2 positive-query mask success rate at IoU@0.75 remains below 0.17. The best All-mIoU is below 0.32.
- Abstention Calibration: Models exhibit poor calibration on target-absent queries. Some models (e.g., GPT-4o) abstain well but localize poorly on positive queries, while others (e.g., Florence-2) detect positive targets well but hallucinate predictions when targets are absent (0% Empty Accuracy).
Trade-offs: There is a clear trade-off between positive-query mask quality and target-absent rejection. Models like SAM 3 achieve high Empty Accuracy (93.4%) but weak positive-query mask quality, whereas segmentation MLLMs like LISA-13B produce better masks but frequently hallucinate on absent targets.
Diagnostic Insights: Performance varies significantly across target families and query types. Rank-based queries (e.g., "leftmost," "largest") and side-selection queries remain substantially harder than unique-target queries, suggesting that open-vocabulary detection alone is insufficient for generalized agricultural grounding.

Significance and Claims

The paper positions AgroVG as a diagnostic benchmark rather than a measure of deployment readiness. Its primary contributions are:

Formulation: It casts agricultural visual grounding as a generalized set-prediction problem, explicitly addressing the need for multi-target completeness and existence-aware abstention.
Comprehensive Benchmark: It provides the first unified benchmark spanning six agricultural target families with both bounding-box and instance-mask grounding protocols under single-, multi-, and target-absent regimes.
Diagnostic Value: The evaluation reveals systematic failures in current state-of-the-art models regarding set completeness, abstention calibration, and fine-grained mask grounding. These findings highlight that reliable agricultural grounding requires joint progress in target existence detection, cardinality estimation, and geometric coverage.

The authors emphasize that AgroVG serves as a standardized testbed to identify failure modes (such as hallucination on absent targets or incomplete set retrieval) that are critical for safety in precision agriculture but are often overlooked in standard single-object grounding benchmarks. The dataset and code are released to support future research in developing more reliable, instruction-aware, and existence-aware agricultural AI systems.