INQUIRE-Search: Interactive Discovery in Large-Scale Biodiversity Databases

The paper introduces INQUIRE-Search, an open-source, natural language-driven system that enables scientists to efficiently discover, verify, and analyze complex ecological phenomena within large-scale biodiversity image databases like iNaturalist, thereby transforming manual inspection workflows into a scalable paradigm for interactive scientific discovery.

The Gist

Imagine you have a massive library containing 300 million photos of nature. These photos were taken by millions of regular people (birdwatchers, hikers, gardeners) and uploaded to a site called iNaturalist.

Right now, if you wanted to find a photo of a specific thing—like "a bird eating a worm" or "a tree growing back after a fire"—you'd have to act like a librarian with a broken catalog. You could only search by what the animal is, where it was, or when it was taken. You couldn't search by what it was doing. It's like trying to find a book in a library by only knowing the author's name, but not the title or the story inside.

To find that one specific photo, you might have to manually scroll through thousands of images, which is impossible to do at this scale.

Enter: INQUIRE-Search

The authors of this paper built a new tool called INQUIRE-Search. Think of it as a "Magic Search Engine" for nature photos.

Instead of forcing you to use strict categories, you can just type a sentence in plain English, like a normal conversation.

• You type: "Show me pictures of a hermit crab using plastic trash as a shell."
• The Magic: The computer understands the meaning of your sentence. It doesn't just look for the words "plastic" or "shell"; it looks at the picture and understands the concept. It instantly finds the photos that match your idea, even if no one ever tagged them with those words.

How It Works (The Analogy)

Imagine the computer has a giant, invisible map of "ideas."

1. The Map: Every photo in the database is placed on this map based on what it shows. A photo of a "dog chasing a ball" sits right next to a photo of a "puppy playing fetch," even if they are different dogs.
2. The Query: When you type your question, the computer turns your words into a dot on that same map.
3. The Match: It instantly draws a line to the photos closest to your dot. It's like asking a friend, "Do you know anyone who likes pizza?" and they immediately point to the people sitting at the pizza table, rather than checking a list of everyone's names.

What Did They Find? (The Treasure Hunt)

The team tested this tool with five different "treasure hunts" to see if it could find things that were previously hidden:

1. Bird Diets: They asked, "Show me birds eating seeds." The tool found hundreds of photos showing exactly what birds eat in summer vs. winter, matching what scientists already knew but finding it 3 to 25 times faster than looking manually.
2. Forest Recovery: After a massive wildfire, they asked, "Show me young trees growing in burned forests." The tool found tiny saplings that satellites missed and field researchers couldn't easily spot, helping scientists understand how forests heal.
3. Dead Birds: They searched for "dead birds" in cities vs. the countryside. The tool found hundreds of cases, revealing that bird deaths spike during migration seasons, especially in cities where they hit windows.
4. Plant Life Cycles: They tracked a plant called Milkweed through its entire life: sprouting, flowering, making seeds, and dying. The tool found photos for every stage, helping scientists see how climate change affects plant timing.
5. Whale ID: They looked for photos of whale tails (flukes) to identify individual whales. The tool found high-quality tail photos that could be matched to a database of known whales, helping track their movements across the ocean.

Why This Matters

Before this, finding these specific "clues" in nature required:

• Years of manual work (scrolling through photos one by one).
• Expensive field trips to specific locations.
• Specialized AI that only experts could build.

INQUIRE-Search changes the game:

• It's Open Source: Anyone can use it.
• It's Fast: It turns a months-long search into a few hours.
• It's Smart: It understands the story in the photo, not just the tags.

The Catch (The Fine Print)

The tool is amazing, but it's not magic. It relies on the photos people already took.

• If no one took a photo of a rare animal doing a specific thing, the tool can't find it.
• The photos might be biased (people take more photos of cute animals than boring ones).
• Scientists still need to double-check the photos to make sure the computer didn't get confused.

The Bottom Line

This paper introduces a new way to do science. Instead of going out to collect new data, scientists can now dig through the mountains of data we already have and pull out the specific clues they need to answer big questions about how nature works. It turns a passive photo archive into an active, interactive laboratory.

Technical Summary

1. Problem Statement

Ecological research increasingly relies on understanding complex phenomena beyond simple species occurrence, such as species interactions, behaviors, phenology, and responses to disturbance. While community science platforms like iNaturalist host hundreds of millions of biodiversity images containing this "secondary data," extracting it at scale is currently infeasible.

• Limitations of Current Methods: Existing workflows rely heavily on manual inspection or basic metadata filtering (taxonomy, location, time). Rich ecological descriptors (e.g., "eating a worm," "burned forest") are rarely annotated consistently; fewer than 25% of iNaturalist records contain additional annotations.
• The Bottleneck: Finding specific, rare, or context-dependent events (e.g., predation events or early forest regrowth) requires reviewing thousands of irrelevant images manually. For example, a recent study required reviewing ~48,000 observations to find 159 predation events.
• Supervised AI Limitations: Training supervised deep learning models for specific secondary tasks requires exhaustive, labeled datasets and significant computational resources, making them impractical for open-ended discovery across diverse, unstructured data.

2. Methodology: INQUIRE-Search System

The authors introduce INQUIRE-Search, an open-source, interactive system that leverages Vision-Language Models (VLMs) to enable natural language querying of massive ecological image databases.

System Architecture

• Core Model: The system uses SigLIP-So400m-384-14, a state-of-the-art VLM trained to map images and text into a shared high-dimensional semantic embedding space.
• Indexing:
  • All ~300 million iNaturalist images are pre-processed and embedded using the SigLIP image encoder.
  • These embeddings are stored in a FAISS (Facebook AI Similarity Search) vector index, enabling sub-second similarity search.
• Search Process:
  1. Query: A user inputs a natural language query (e.g., "American Robin eating a worm").
  2. Embedding: The text is embedded by the VLM's text encoder.
  3. Retrieval: The system performs a cosine similarity search against the FAISS index to rank images by semantic relevance.
  4. Filtering: Users can apply metadata filters (taxonomy, geography, date) to narrow the search scope.
• Human-in-the-Loop Workflow:
  1. Query & Prioritize: Scientists formulate natural language prompts.
  2. Retrieve & Verify: Experts inspect the top-ranked images sequentially. Images are labeled "informative" only if the target phenomenon is visually unambiguous. Inspection budgets are capped (e.g., 200–500 images) to ensure comparable effort.
  3. Export & Analyze: Verified images and their metadata are exported as CSVs for downstream statistical analysis.

Evaluation Metrics

Since ground truth labels for rare ecological events are often unavailable, the authors evaluate Retrieval Efficiency rather than traditional accuracy:

• Screening Yield ($Y$): The ratio of verified informative images ($N_{ret}$) to inspected images ($N_{insp}$).
• Yield Ratio ($Y_{ratio}$): Compares the efficiency of INQUIRE-Search against a baseline metadata/keyword search ($Y_{INQUIRE} / Y_{baseline}$).

3. Key Contributions

1. Scalable Retrieval Framework: A system capable of surfacing complex ecological scenarios (interactions, behaviors, life-history events) from 300M+ images using open-ended text, bypassing the need for predefined labels.
2. Interactive Human-in-the-Loop Workflow: A novel paradigm treating ecological image analysis as an information retrieval task, where scientists iteratively refine queries and validate results, shifting the bottleneck from data collection to expert verification.
3. Ecological Utility Demonstration: Validation across five diverse case studies proving the system's ability to extract actionable scientific data from unstructured repositories.

4. Results: Five Case Studies

The system was tested on five distinct ecological questions, demonstrating significant efficiency gains over traditional metadata/keyword searches.

| Case Study | Goal | INQUIRE-Search Performance | Baseline (Metadata/Keyword) | Efficiency Gain ($Y_{ratio}$) |
| :--- | :--- | :--- | :--- :--- |
| 1. Bird Diets | Identify seasonal feeding events for 5 species. | Retrieved 669 informative images for American Robin; reproduced known seasonal diet trends. | Only 105 for Robin; zero for others. | 3–25× more efficient. |
| 2. Post-Fire Regrowth | Find young trees in the 2012 High Park Fire perimeter. | Verified 123 regeneration events (78 deciduous, 45 coniferous) with high yield (0.45–0.70). | Only 25 events found (Yield 0.03–0.09). | **5–7×** more efficient. |
| 3. Wildlife Mortality | Detect dead birds in Urban (Boston) vs. Rural areas. | Verified 624 mortality events (543 urban, 79 rural). | Only 337 events found via "dead/alive" tags or keywords. | ~1.8× more efficient. |
| 4. Plant Phenology | Identify 4 stages of Milkweed (emergence, flowering, seeding, senescence). | Recovered 461 stage-specific images. Crucially, found 45 emergence and 52 senescence images (stages often missed by metadata). | Baseline found only 5 emergence and 17 senescence images. | 7.7× (Emergence) and 2.9× (Senescence) more efficient. |
| 5. Whale Re-ID | Find identifiable humpback whale flukes. | Found 153 high-quality fluke images (Yield 0.77); matched 34 unique whales to HappyWhale database. | Only 42 usable images found via keyword "tail" (Yield 0.21). | 3.67× more efficient. |

Key Findings:

• INQUIRE-Search consistently concentrates relevant observations 3–25 times more efficiently than manual inspection budgets.
• It successfully retrieves "secondary data" (behaviors, context) that is invisible to structured metadata filters.
• The system can reproduce known ecological patterns (e.g., seasonal diet shifts, fire severity vs. regeneration) using only opportunistic data.

5. Significance and Implications

• New Scientific Paradigm: INQUIRE-Search shifts ecological research from "data collection" to "data discovery." It allows scientists to generate hypotheses and design experiments by querying existing massive datasets, rather than waiting for new field surveys.
• Cost and Energy Efficiency: Unlike generative AI or repeated inference models, INQUIRE-Search performs a single text embedding and a fast vector search. The marginal energy cost per query is low, making it scalable.
• Addressing Bias and Uncertainty: The authors emphasize that while the tool is powerful, it inherits the biases of community science (e.g., observer preference for charismatic species) and VLM limitations (e.g., difficulty with complex relational concepts). The workflow requires experts to validate results and account for these biases in downstream analysis.
• Generalizability: While demonstrated on iNaturalist, the open-source architecture is adaptable to other large image repositories, including camera trap datasets, enabling NGOs and local communities to perform scalable ecological monitoring.

In conclusion, INQUIRE-Search unlocks the "secondary data" value trapped in hundreds of millions of biodiversity images, providing a scalable, efficient, and interactive tool for modern ecological discovery.