Assessing Task-based Chatbots: Snapshot and Curated Datasets for Dialogflow
This paper introduces TOFU-D and COD, two datasets comprising 1,788 and 185 Dialogflow chatbots respectively, to address the scarcity of curated resources for empirical research, while preliminary analysis reveals significant gaps in test coverage and frequent security vulnerabilities that underscore the need for systematic multi-platform studies on chatbot quality and security.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine the world of chatbots as a massive, bustling city. For a long time, researchers studying how to build better, safer, and more reliable chatbots have been trying to navigate this city, but they've been working with a very incomplete map. They've mostly been looking at a few small, open-source neighborhoods (like the "Rasa" district), but they haven't really explored the big, commercial downtown area where the heavy-duty, business-grade chatbots live.
This paper is like a team of urban planners who finally decided to take a snapshot of that entire commercial downtown. Here is what they did, explained simply:
1. The Big Snapshot (TOFU-D)
The researchers went to GitHub (a giant online library where people store their code) and looked for anything labeled "Dialogflow." Dialogflow is a popular, commercial tool made by Google that companies use to build chatbots.
- The Hunt: They started with over 12,000 potential "buildings" (repositories).
- The Filter: They realized many of these were just empty lots or construction sites, not actual chatbots. They filtered out the noise and found 1,788 actual, working chatbot "buildings."
- The Result: They created a dataset called TOFU-D. Think of this as a massive, unedited photo album of every chatbot they could find in that specific district on a specific day. It shows the raw reality: some are tiny, some are huge, some are messy, and some are brilliant.
2. The Curated Tour (COD)
Looking at 1,788 chatbots is overwhelming, and many of them are just "toy" examples or broken prototypes. To make this useful for serious research, the team created a second, smaller dataset called COD.
- The Selection: They acted like a museum curator. They didn't just pick random chatbots; they picked the ones that were actually doing something interesting. They looked for chatbots that:
- Had a real conversation (not just a list of pre-written answers).
- Connected to outside services (like a restaurant bot actually checking a menu database).
- Were written in English (so researchers could understand them).
- Were popular enough to have a "star" on their GitHub page.
- The Result: They ended up with 185 high-quality chatbots. This is their "Hall of Fame" collection, designed to be the perfect test subjects for studying how these bots work and where they fail.
3. What They Found in the City
When the researchers walked through these chatbots, they noticed some interesting things about the "architecture" of this city:
- Language Mix: While most chatbots speak English, the city is surprisingly diverse. The code behind them is written in many different languages (JavaScript, Python, Java, etc.), unlike the open-source neighborhood where everyone mostly spoke Python.
- Complexity: The curated chatbots (COD) were much more complex. They weren't just saying "Hello" and "Goodbye"; they were actually performing tasks, connecting to cloud services, and handling real user data.
- The "Google" Connection: Many of these chatbots were built to work seamlessly with Google's own tools, like the Google Assistant, which is a feature you don't see as often in the open-source district.
4. The Safety Inspection (The "Bandit" and "Botium" Checks)
To see if these chatbots were safe and reliable, the researchers ran two quick tests, like a building inspector and a quality control manager:
- The Quality Check (Botium): They tried to automatically generate test cases (like a robot trying to talk to the chatbot to see if it breaks). They found that the tests were often missing. The chatbots were great at answering simple questions but often failed when users said "Hello," tried to say something unexpected, or needed to remember what was said earlier. It was like finding a car that drives great on a straight road but stalls at the first stop sign.
- The Security Check (Bandit): They looked at the code for security holes. They found that many chatbots had "open doors." For example:
- Some were listening for connections from any network, not just the ones they should.
- Some were allowing code to be run that shouldn't be run.
- They also found common mistakes, like forgetting to set time limits on requests (which can freeze the system) or using weak random number generators.
The Bottom Line
The main point of this paper is that researchers can no longer just study a few simple chatbots. The world of chatbots is huge and varied. By providing these two new datasets—the massive snapshot (TOFU-D) and the high-quality curated list (COD)—the authors are giving the research community a much better map and a better set of test subjects.
They aren't saying these chatbots are perfect; in fact, their tests show they have significant flaws in testing and security. But by having a clear, large, and diverse collection of real-world examples, researchers can finally start fixing those flaws systematically, rather than guessing based on a few small examples.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.