RefAerial: A Benchmark and Approach for Referring Detection in Aerial Images
This paper introduces RefAerial, a large-scale benchmark for referring detection in aerial images characterized by diverse scales and complex scenes, and proposes a novel Scale-Comprehensive and Sensitive (SCS) framework that leverages mixture-of-granularity attention and a two-stage decoding strategy to overcome performance degradation caused by intrinsic scale variations.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are playing a game of "Where's Waldo?", but instead of looking at a flat picture in a book, you are looking down from a helicopter at a busy city street.
In the world of computer vision, this game is called "Referring Detection." It's the ability for a computer to look at an image and find a specific object based on a sentence you give it, like "Find the red car parked next to the blue bus."
For years, computers have been great at this game, but only when playing with ground-level photos (like selfies or street-level shots). In those photos, the objects are big, close up, and usually the main focus.
This paper introduces a new, much harder version of the game: The Aerial View.
The Problem: The "Helicopter" Challenge
The authors realized that existing computers get confused when they look down from the sky. Here is why:
- The "Ant" Problem: In a ground photo, a person might take up half the screen. In an aerial photo, a person is just a tiny speck, like an ant on a sidewalk. The computer struggles to see them.
- The "Crowded Room" Problem: A ground photo usually has one or two main things. An aerial photo of a city intersection might have hundreds of cars, people, and bikes all mixed together. It's like trying to find a specific friend in a stadium full of people wearing similar clothes.
- The "Long Story" Problem: To find a specific car in a crowd, you need a very detailed description. Ground photos usually work with short clues like "the red car." Aerial photos need long, complex stories like "the white sedan driving in the middle lane, with a white car and an electric bike rider ahead of it."
Existing AI models, trained on easy ground photos, completely fail when they try to play this hard aerial version. They get lost in the noise.
The Solution: RefAerial and the "Super-Scanner"
To fix this, the team did two main things:
1. They Built a New Training Ground (RefAerial)
They created a massive new dataset called RefAerial. Think of this as a giant library of 10,000+ high-resolution drone photos.
- The Scale: It has over 115,000 labeled examples.
- The Difficulty: It includes photos taken from different heights, at night, in the rain, and with dozens of targets in a single image.
- The Secret Sauce: They didn't just hire people to label these manually (which would take forever). They built a "Robot Assistant" (REA-Engine). This tool uses advanced AI to draft descriptions and draw boxes, and then human experts act as "editors" to fix mistakes. It's like having a fast typist who writes a draft, and a human editor who polishes it.
2. They Built a New AI Brain (The SCS Framework)
They realized that the old AI brains were too "myopic" (short-sighted). They needed a new brain that could see both the big picture and the tiny details simultaneously. They called this the SCS Framework (Scale-Comprehensive and Sensitive).
Here is how it works, using a Flashlight Analogy:
The Old Way: Imagine trying to find a lost earring in a dark room with a single flashlight. If you shine it on the whole room, the light is too dim to see the tiny earring. If you zoom in too much, you miss the earring because it's in a different corner.
The New Way (MoG Attention): The new AI uses a "Mixture of Granularity" flashlight. It shines multiple beams at once:
- One beam is wide and fuzzy (to see the whole room and big furniture).
- One beam is narrow and sharp (to spot tiny details like the earring).
- It combines all these views so it understands the scene at every scale at the same time.
The Two-Stage Hunt (CtS Strategy):
- Stage 1 (The Broad Sweep): The AI first guesses, "Okay, the target is probably somewhere in this general neighborhood." It gets a rough idea.
- Stage 2 (The Fine Tuning): Once it has the neighborhood, it zooms in and says, "Ah, I see it! It's the specific white car, not the white truck next to it." It refines the answer from "rough guess" to "perfect precision."
The Result
When they tested this new system:
- On the Aerial Dataset: It crushed the competition. While other models got lost in the crowd, the new system found the tiny targets with high accuracy.
- On the Old Ground Dataset: It actually got better at the easy games too, proving that this new way of thinking helps even when the objects are big.
In a Nutshell
This paper is about teaching computers to play "Where's Waldo?" from a helicopter. They built a harder training course (RefAerial) and a smarter detective (SCS Framework) that can look at a scene with both a wide-angle lens and a microscope simultaneously, allowing it to find tiny, specific things in a chaotic, crowded sky view.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.