Conversational AI-Enhanced Exploration System to Query Large-Scale Digitised Collections of Natural History Museums

This paper presents a human-centred system design that leverages conversational AI and function-calling capabilities to enable natural language querying and visual-spatial exploration of nearly 1.7 million digitised natural history specimen records at the Australian Museum, overcoming the limitations of traditional keyword-based search tools.

The Gist

Imagine a massive library, but instead of books, it's filled with 21 million real-life treasures: stuffed birds, ancient shells, preserved insects, and fossils. This is the Australian Museum. For years, most of these treasures have been locked away in the "back room," visible only to scientists with special keys (databases) and the ability to speak "computer code."

The average person couldn't easily peek inside. They had to rely on a few items on display or ask a librarian (a museum staff member) to hunt for specific facts, which took a long time.

This paper describes a new digital magic wand created by researchers to solve this problem. They built a system called the Australian Museum Collection Explorer that lets anyone chat with the museum's entire collection using plain English.

Here is how it works, broken down into simple concepts:

1. The Problem: The "Black Box" of Data

Think of the museum's digitized records as a giant, messy warehouse.

• The Old Way: To find something, you had to know the exact aisle number, the box label, and the filing system. If you didn't know the technical terms, you were stuck.
• The Result: Most people never saw the vast majority of the museum's treasures.

2. The Solution: A Conversational Guide

The team built a system with two main parts, like a tour guide who is also a super-powered map:

• The Interactive Map (The Visual Eye): Imagine a giant digital globe. Instead of just pins, it shows nearly 1.7 million dots, each representing a specimen. You can zoom in on your own neighborhood and see, "Oh, there's a bird specimen collected right here in 1920!" It turns dry data into a visual story of where life exists.
• The Chatbot (The Talking Brain): This is the star of the show. Instead of typing complex search codes, you can just ask questions like:
  • "Show me all the kangaroos found in New South Wales in the 1980s."
  • "How many beetles do you have from Christmas Island?"
  • "Upload a photo of this bird I saw in my garden and tell me what it is."

3. The Secret Sauce: How the AI "Thinks"

You might worry, "Won't the AI just make things up?" (This is a common problem with AI called "hallucinating").

The researchers solved this with a clever trick called Function Calling.

• The Analogy: Imagine the AI is a very smart but slightly forgetful tour guide. If you ask a question, instead of guessing the answer from its memory (which might be wrong), it has a magic phone connected directly to the museum's official database.
• How it works: When you ask, "How many beetles?", the AI doesn't guess. It picks up the phone, asks the database, "Hey, how many beetles are in the system?", gets the exact number, and then tells you the truth. It never guesses; it always checks the source.

4. The Journey: How They Built It

The team didn't just build it in a lab and hope for the best. They used a Human-Centered Design approach, which is like cooking a meal by tasting it as you go.

• Step 1: They talked to museum scientists and public engagement staff. The scientists said, "We need to show the scale of our collections," and the public team said, "People need to see pictures, not just text."
• Step 2: They built a rough prototype (a "beta" version) and tested it with volunteers.
• Step 3: The volunteers gave feedback. They said, "The map is great, but I want to see photos of the animals," and "The AI sounded too robotic; make it sound like a scientist."
• Step 4: They refined the system based on this feedback, creating the final version you see today.

5. Why This Matters: From "Access" to "Agency"

The paper argues that we are moving from Ubiquitous Access (just being able to get to the museum online) to Ubiquitous Agency (having the power to control your own exploration).

• Before: The museum told you what to look at.
• Now: You can follow your own curiosity. If you are curious about a specific bug in your backyard, you can ask the system about it, see where similar bugs were found nearby, and learn about them instantly.

The Big Picture

This project is like turning a static, dusty archive into a living, breathing conversation. It proves that we don't need to be computer experts to explore the wonders of nature. By combining a visual map with a truthful AI chatbot, the Australian Museum is opening its back room to the world, allowing anyone to become a temporary scientist, explorer, or storyteller.

It's not just about showing data; it's about weaving the museum's history into our daily lives, right from our phones, making the natural world feel a little closer and a lot more understandable.

Technical Summary

Here is a detailed technical summary of the paper "Conversational AI-Enhanced Exploration System to Query Large-Scale Digitised Collections of Natural History Museums."

1. Problem Statement

Natural history museums possess vast collections (e.g., the Australian Museum holds over 21 million specimens, with 4 million digitised) that are largely inaccessible to the general public.

• Accessibility Barriers: Traditional access is restricted to researchers via loans or limited educational tours. Public access is hindered by the complexity of database schemas and the need for specialized scientific knowledge.
• Limitations of Current Tools: Conventional keyword searches and database interfaces require technical expertise. While conversational AI (Chatbots) has been explored, Large Language Models (LLMs) often suffer from "hallucinations" (generating false facts) when queried about specific, non-public training data, making them unreliable for scientific record retrieval.
• The Gap: There is a lack of systems that combine natural language querying with grounded data retrieval (ensuring answers are based on actual database records) for large-scale, dynamic biodiversity datasets.

2. Methodology

The project followed a Human-Centred Design (HCD) process involving three phases:

A. Stakeholder Requirements Gathering

• Focus Groups: Conducted two workshops with 30 Australian Museum staff (scientists, collection managers, and public engagement teams).
• Key Insights: Identified four primary design goals:
  1. Comprehensibility of Scale: Visualize the vastness of collections.
  2. Query Accessibility: Allow non-experts to query data using natural language.
  3. Response Reliability: Ensure answers are accurate and grounded in collection data.
  4. Ubiquitous Access: Enable remote, 24/7 exploration.

B. Iterative Design & Prototyping

• Iteration 1: Built a minimal prototype using Llama 3 and RAG (Retrieval-Augmented Generation) with a text-based summary.
  • Feedback: Users preferred map-based visualization over text lists; external images (Wikipedia) were insufficient.
• Iteration 2 (Proof-of-Concept): Developed a web app using React, Flask, and PostgreSQL with 59k bird records. Used GPT-4-nano with Text2SQL (generating SQL queries to the database).
  • Feedback: Users valued the map but found the AI unreliable when it failed to generate correct SQL, leading to fallback to general knowledge (hallucinations). The multiple "persona" modes (Scientist, Friend, Robot) caused confusion.

C. Final System Implementation

Based on feedback, the final system was redesigned to address reliability and scalability:

• Architecture:
  • Frontend: React-based interactive map (Leaflet.js) with viewport-based dynamic loading.
  • Backend: Python (Flask) orchestrating requests.
  • Data Source: Connected to the Atlas of Living Australia (ALA) Biocache API, accessing 1.7 million live specimen records (no local database storage of the full dataset).
  • LLM: OpenAI's gpt-5-mini (Note: The paper mentions gpt-5-mini in the text, though this may be a placeholder for a specific model version like gpt-4o-mini given the 2026 date context; the mechanism is the focus).
• Core Technical Innovation: Instead of generating SQL queries, the system uses Function Calling.
  • The LLM is configured with JSON schemas defining specific tools (search_specimens, get_specimen_statistics, get_specimen_by_id).
  • The LLM outputs structured function calls with parameters (e.g., taxonomy, location, date range) which the backend executes against the ALA API.
  • Resolution Layers: Includes taxonomic resolution (converting common names to scientific names via ALA BIE API) and geocoding (converting place names to coordinates via Google Geocoding API).

3. Key Contributions

1. Function-Calling for Grounded Retrieval: The paper demonstrates a robust architecture where LLMs act as an interface to external APIs via function calling, rather than relying on RAG or Text2SQL. This ensures responses are strictly grounded in real-time, authoritative museum data, minimizing hallucinations.
2. Hybrid Visual-Spatial Interface: The system combines an interactive map for spatial exploration with a conversational agent, allowing users to switch between visual browsing and natural language querying seamlessly.
3. Human-Centred Design Framework: Provides a reproducible methodology for museums to integrate AI, moving from stakeholder workshops to iterative prototyping and final deployment.
4. Multimodal Interaction: Supports both text-to-data queries and image-to-data interactions (e.g., users uploading a photo of a bird to identify it and find museum records).

4. Results

• User Study (Design Iteration II): 12 Digital Volunteers tested the prototype.
  • Visual-Spatial Value: 10/12 participants found the map essential for understanding distribution.
  • Image Need: 9/12 requested specimen images directly in the interface.
  • Reliability: 5/12 noted that ungrounded AI responses were untrustworthy; this drove the shift to function calling.
  • Persona Complexity: 4/12 found multiple AI personas confusing; the final system adopted a single "Scientist" persona.
• System Performance: The final system successfully handles nearly 1.7 million records dynamically. It supports complex queries (e.g., "Show me kangaroo specimens from NSW in the 1980s") by translating natural language into API calls, returning accurate counts, locations, and specimen images with direct links to the source data (ALA).

5. Significance

• Paradigm Shift in Museum Engagement: Moves beyond "ubiquitous access" (just viewing data) to "ubiquitous agency," where users actively govern their exploration through open-ended, curiosity-driven queries.
• Scalability and Maintenance: By connecting directly to national data infrastructure (ALA) via APIs rather than maintaining a local copy of the dataset, the system remains up-to-date with minimal maintenance and handles massive scale without local storage constraints.
• Trust and Transparency: The system includes direct links to the source data (ALA) in every response, fostering trust and allowing users to verify information.
• Broader Applicability: The architecture is adaptable to other digitized collections (cultural artifacts, historical archives), offering a blueprint for scientific AI agents in heritage institutions.

In conclusion, this paper presents a successful integration of conversational AI and large-scale data infrastructure, solving the "truthfulness" problem in museum chatbots through function calling and providing a new model for public engagement with scientific data.