What Questions Should Robots Be Able to Answer? A Dataset of User Questions for Explainable Robotics
This paper introduces a dataset of 1,893 user questions for household robots, collected from 100 participants across 12 categories, to guide the development of explainable robotics by revealing that while users frequently ask about task execution, they prioritize understanding how robots handle difficult scenarios, with question patterns varying significantly between novice and experienced users.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you just bought a brand-new, super-smart robot to help you with chores around the house. You watch it vacuum the living room, wash the dishes, and fold your laundry. But as you watch, your brain starts buzzing with questions.
- "Why did you skip the kitchen?"
- "Can you also clean the windows?"
- "What if you drop a plate?"
- "How do you know the food is cooked?"
This paper is essentially a massive inventory of those questions. The researchers wanted to know: Exactly what are humans curious about when they see a robot working? They didn't just guess; they asked 100 real people to watch videos and read stories about robots doing household tasks and write down every question they would want to ask.
Here is the breakdown of their findings, using some simple analogies:
1. The "Question Library" (The Dataset)
The researchers collected 1,893 questions and organized them into a giant library with 12 main sections and 70 smaller shelves.
- The Most Popular Shelf (Execution Details): About 21% of the questions were like checking a receipt. People wanted to know the nitty-gritty facts: What did you do? What objects did you touch? How did you do it?
- Analogy: It's like a parent asking a child, "Did you really take out the trash, and did you tie the bag?"
- The "Can You Do This?" Shelf (Capabilities): The second most common group (12.6%) was people asking about the robot's skills. Can you wash dishes? Can you do it faster?
- The "Did You Do It Right?" Shelf (Self-Assessment): People wanted to know if the robot thought it did a good job. Was that difficult? Did you make any mistakes?
2. The Big Surprise: What People Actually Care About
Here is the twist. In the world of AI research, scientists often think people care most about "Why" questions (e.g., "Why did you choose the red cup?"). They assume we want to understand the robot's logic.
The paper found the opposite.
While "Why" questions were common, they were rated as less important by the users.
Instead, the questions people rated as most critical (the "must-haves") were about Safety and Potential Problems.
- Analogy: If you hire a babysitter, you don't care as much about why they chose a specific toy to play with. You care deeply about: "What would you do if the baby fell?" or "How do you make sure they don't eat something dangerous?"
- Users wanted to know: What happens if you drop something? How do you ensure you don't break the dishes?
3. The "Newbie" vs. The "Expert"
The researchers noticed that the questions changed depending on who was asking, similar to how a new driver asks different questions than a race car driver.
- The Novices (New to Robots): They asked simple, factual questions. What did you do? Where is the cat? They were focused on the immediate facts of the situation.
- The Experts (People who know robots): They asked deeper, more complex questions. How would you handle a difficult situation? How did you decide to prioritize that task? They were less interested in the "what" and more interested in the "how" and "what if."
4. Why This Matters for Robot Builders
The paper argues that for robots to be truly helpful and trustworthy, they need to be built with a specific "memory" and "explanation kit."
- The "Black Box" Problem: If a robot uses a fancy AI (like a Large Language Model) to talk to you, but that AI doesn't have access to the robot's actual logs (like a flight recorder), the AI might just make things up (hallucinate).
- The Solution: Robot designers need to know what to record. If users care most about safety and "what-if" scenarios, the robot needs to log data about potential errors and decision-making processes, not just a simple list of actions.
Summary
Think of this paper as a user manual for building a robot's brain. It tells engineers: "Don't just build a robot that can explain why it moved a chair. Build a robot that can confidently tell you how it would handle a broken plate, because that's what actually makes people feel safe and trust the machine."
The researchers also noted that their study used videos and text stories (like a movie script) rather than real-life interactions. While this isn't a perfect simulation of real life, it allowed them to test a huge variety of scenarios quickly, giving them a clear map of what people want to know before the robots even enter our homes.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.