💻 computer science

RealBirdID: Benchmarking Bird Species Identification in the Era of MLLMs

The RealBirdID benchmark introduces a rigorous evaluation framework for fine-grained bird species identification that requires multimodal large language models to either correctly identify species or abstain with evidence-based rationales for unanswerable cases, revealing that current state-of-the-art models struggle with both high accuracy and calibrated abstention.

Original authors: Logan Lawrence, Mustafa Chasmai, Rangel Daroya, Wuao Liu, Seoyun Jeong, Aaron Sun, Max Hamilton, Fabien Delattre, Oindrila Saha, Subhransu Maji, Grant Van Horn

Published 2026-03-31

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Logan Lawrence, Mustafa Chasmai, Rangel Daroya, Wuao Liu, Seoyun Jeong, Aaron Sun, Max Hamilton, Fabien Delattre, Oindrila Saha, Subhransu Maji, Grant Van Horn

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a birdwatching app on a user's phone. The user snaps a photo of a bird in a tree and asks, "What kind of bird is this?"

In the old days, if the photo was blurry, the bird was hiding behind a leaf, or the bird was making a sound that couldn't be heard, the app would just guess. It might confidently say, "That's a Red-winged Blackbird!" even though it was actually a completely different bird that just looked similar in that one blurry shot. This is called "hallucinating"—making things up with confidence.

RealBirdID is a new "test" created by researchers to see if modern AI models are smart enough to say, "I don't know, and here is why."

Here is the breakdown of the paper using simple analogies:

1. The Problem: The Overconfident Student

Imagine a student taking a very hard biology test.

The Old Way: If the student doesn't know the answer, they still guess. If the question is "What bird is this?" and the photo is too dark, the student guesses anyway. They get it wrong, but they don't get a penalty for guessing.
The Real World: In real life (like in a hospital or a nature reserve), guessing wrong can be dangerous. If a doctor guesses a disease without enough info, it's bad. If a bird app guesses the wrong species, it might mislead a conservationist.
The Goal: We want the AI to act like a honest expert. If the photo is too blurry, the expert should say, "I can't tell, the image is too low quality." If the bird is making a sound we can't hear, they should say, "I need to hear it to be sure."

2. The Solution: The "RealBirdID" Exam

The researchers built a special exam called RealBirdID. It's like a two-part test for AI:

Part A (The Easy Questions): Clear, sharp photos of birds. The AI should answer correctly.
Part B (The Impossible Questions): Photos where the bird is hidden, the photo is blurry, or the bird is a type that only looks different when it sings.
- The Catch: For these impossible questions, the AI isn't allowed to guess. It must abstain (refuse to answer) and give a reason.
- Example: Instead of guessing "Red-tailed Hawk," the AI should say, "I can't identify this because the tail is cut off by a branch."

3. The Results: The AI is Still a "Know-It-All"

The researchers tested the smartest AI models available (including the latest versions of GPT and Gemini) on this exam. Here is what they found:

They are bad at the hard stuff: Even the best AI models struggled to identify the birds in the clear photos (getting less than 13% right on the hardest species).
They are terrible at saying "I don't know": When the photo was impossible to solve, the AI usually just guessed anyway. It didn't know when to stop.
They give bad excuses: Sometimes the AI did say "I don't know," but the reason was wrong.
- Real Reason: "The bird is too far away."
- AI Excuse: "The image quality is low." (Even if the image was actually sharp, the AI just panicked and blamed the quality).
They ignore the ears: Many birds look identical but sound different. The AI models, which are mostly trained on pictures, completely ignored the fact that they needed to hear the bird to solve the puzzle. They acted like they were deaf.

4. The "Map" Analogy

The researchers tried to help the AI by giving it a "geographic map" (telling it, "This bird was spotted in Florida, so it can't be a penguin").

Result: This helped the AI guess the bird correctly more often.
But: It made the AI worse at knowing when to say "I don't know." It became overconfident because it had the map, even when the photo was still too blurry to see the bird.

5. Why This Matters

This paper is a wake-up call. It tells us that just because an AI can talk and see, it doesn't mean it knows when to stop.

Current AI: Like a confident tourist who points at a random building and says, "That's the Eiffel Tower!" even if they are in Tokyo.
What we need: A guide who says, "I'm not sure, the fog is too thick, let's wait for better weather."

The Bottom Line:
The RealBirdID benchmark is a tool to force AI developers to build systems that are humble. We don't just want AI that is good at guessing; we want AI that is good at knowing its own limits, so it doesn't mislead us when the answer isn't clear.

1. Problem Statement

Fine-grained visual recognition (FGVR) for bird species identification faces a critical gap between current benchmarking practices and real-world deployment.

The "Answerable" Bias: Existing datasets (e.g., CUB-200, iNaturalist) consist almost exclusively of "answerable" images where the species can be determined visually. This forces models to guess even when visual cues are insufficient.
Real-World Limitations: In the wild, identification often fails due to non-visual factors (e.g., need for vocalization), occlusion, poor camera angles, or low image resolution.
The Hallucination Risk: Current Multimodal Large Language Models (MLLMs) are trained to always provide an answer. When forced to classify unanswerable images, they tend to hallucinate confident but incorrect predictions rather than abstaining.
The Gap: There is no standard benchmark that evaluates a model's ability to not only identify species but also principled abstention—recognizing when an image is unanswerable and providing a concrete, evidence-based rationale (e.g., "requires vocalization," "view obstructed").

2. Methodology: The RealBirdID Benchmark

Dataset Construction

The authors constructed RealBirdID using data from iNaturalist, focusing on observations where the community could not reach a consensus at the species level.

Unanswerable (UA) Set:
- Source: Extracted from iNaturalist observations with community disagreement at the species level (resolved only to genus or higher).
- Filtering: Used YOLOv3 for bird presence and MANIQA for image quality.
- Annotation: Parsed community comments and identification histories using heuristics and regular expressions to label specific reasons for unanswerability.
- Categories: Reasons include Sound Needed (vocalization required), Angle/Occlusion, Low Quality, Life-stage/Sex dimorphism, and Taxonomic uncertainty.
- Scale: 3,253 unanswerable images across 248 genera.
Answerable (A) Set:
- Source: Curated "Research Grade" images for the same genera found in the UA set.
- Scale: 31,885 images covering 3,442 species.
Structure: The dataset is organized hierarchically (Genus $\to$ Species) to allow evaluation at different taxonomic depths.

Evaluation Metrics

To move beyond simple accuracy, the paper proposes three specific metrics:

Abstention Tradeoff (UA/A): Measures the model's ability to distinguish between answerable and unanswerable examples. It plots the fraction of unanswerable images abstained against the fraction of answerable images abstained (analogous to an ROC curve). The Area Under the Curve (AUC) serves as the score.
Classification Performance (Information Gain - IG): Evaluates accuracy while accounting for partial correctness. If a model cannot identify the species but correctly identifies the genus, it receives partial credit. This metric aggregates accuracy across the taxonomic hierarchy.
Model Calibration (AUC): Measures how well the model's confidence correlates with its accuracy on the answerable set.

Experimental Setup

Models Tested:
- Encoders: CLIP, MetaCLIP, WildCLIP, SigLIP, BioCLIP.
- MLLMs: Qwen2.5-VL, Llama-3.2V, InternVL3, Gemma-3, Gemini-2.5 Pro, GPT-5.
Prompting: Standard zero-shot prompts ("What is the species of this bird?") were used. For MLLMs, the system extracts the final species prediction and the reasoning for abstention using constrained decoding (nlg2choice).
Range Map Augmentation: Experiments included providing geographic range maps to restrict the candidate species list, simulating a realistic tool used by birders.

3. Key Contributions

RealBirdID Dataset: The first benchmark specifically designed to test abstention with reasoning in fine-grained visual recognition. It includes curated unanswerable examples with labeled rationales.
New Evaluation Framework: Introduces metrics (UA/A tradeoff and IG) that jointly evaluate classification capability and the ability to refuse uncertain predictions.
Comprehensive Benchmarking: Provides the first large-scale evaluation of state-of-the-art MLLMs (including GPT-5 and Gemini-2.5 Pro) on fine-grained bird identification and their failure modes regarding abstention.

4. Key Results

Performance on Classification

Low Accuracy: Even on the Answerable set, MLLMs struggle with fine-grained identification. Top models (GPT-5, Gemini-2.5 Pro) achieved only ~13% accuracy at the species level.
Encoder Superiority: Specialized vision encoders (specifically BioCLIP) significantly outperformed MLLMs in classification (BioCLIP achieved ~17% species accuracy vs. ~13% for MLLMs).
Impact of Range Maps: Providing geographic range information drastically improved classification accuracy (IG increased from ~57 to ~88) by reducing the candidate pool, but this did not translate to better abstention capabilities.

Performance on Abstention

Failure to Abstain: Models generally fail to separate answerable from unanswerable examples. The best models achieved an Abstention Tradeoff (UA/A) AUC of only ~0.53, barely better than random guessing (0.5).
No Correlation: There is no strong correlation between a model's classification ability and its ability to abstain. High-performing classifiers (like BioCLIP) are not necessarily better at knowing when not to answer.
Brittleness: MLLM abstention rates are highly sensitive to prompt wording. Changing the instruction from "What species is this?" to "You may refuse if unsure" caused abstention rates to swing wildly (e.g., from 3% to 23% for the same model).

Analysis of Reasoning

Incorrect Rationales: When models did abstain, their reasons were often wrong.
Visual Bias: Models overwhelmingly attributed failures to Image Quality or Angle/Occlusion, even when the ground truth was "Sound Needed" (vocalization).
Audio Blindness: MLLMs almost never identified the need for audio cues as a reason for abstention, reflecting a strong visual bias in their training data and a lack of understanding of non-visual biological constraints.

5. Significance and Conclusion

Safety and Reliability: The paper highlights that current MLLMs are unsafe for deployment in critical domains (like conservation or citizen science) because they hallucinate answers for unanswerable inputs rather than admitting uncertainty.
Beyond Accuracy: The study proves that improving classification accuracy alone is insufficient for robust AI. Systems require explicit abstention-aware objectives or downstream selective classification mechanisms.
Future Directions: The authors call for:
- Training models to recognize non-visual constraints (e.g., audio needs).
- Developing metrics that penalize confident hallucinations.
- Creating benchmarks that simulate out-of-distribution and out-of-schema scenarios.

In summary, RealBirdID exposes a fundamental weakness in current vision-language models: they are confident guessers rather than principled reasoners. The benchmark provides a necessary tool to drive the development of AI systems that know what they do not know.