DRACULA: Hunting for the Actions Users Want Deep Research Agents to Execute
This paper introduces DRACULA, the first dataset capturing expert user feedback on intermediate actions for Scientific Deep Research agents, revealing that leveraging a user's selection history significantly improves the prediction of preferred actions and highlighting the critical challenge of deciding which actions to execute rather than just how to execute them.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you have a super-smart research assistant (an AI agent) that can read thousands of scientific papers and write a long, detailed report for you. In the past, if you didn't like the final report, you could only say, "This is bad," or "This is good." But that's like telling a chef, "The soup tastes wrong," without telling them whether they used too much salt, forgot the onions, or burned the garlic. You don't know which step went wrong, so the chef can't really learn how to fix it next time.
This paper, DRACULA, introduces a new way to talk to these research assistants. Instead of just judging the final soup, the researchers asked users to taste-test the ingredients and steps before the soup was even made.
Here is the breakdown of what they did and found, using simple analogies:
1. The Problem: The "Black Box" Report
Usually, when an AI writes a research report, it makes hundreds of tiny decisions along the way: Which papers should I read? Should I add a chart? Should I explain this concept with an analogy?
Previously, users only saw the final result. If the report was boring, the AI didn't know if it was because it picked the wrong papers or because the writing style was too stiff. It was a "black box."
2. The Solution: The "Recipe Card" Approach
The researchers built a system where, before the AI writes the report, it presents the user with a menu of possible actions (like a recipe card).
- The Menu: "I could add a section on datasets," "I could explain this with a real-world example," or "I could focus only on the last five years of research."
- The User's Job: The user picks the actions they want (like checking boxes on a recipe) and explains why.
- The Result: The AI then writes the report using only those selected actions. Finally, the user judges: "Did you actually do what I asked?"
They collected over 8,000 of these "action choices" and 5,000 "execution checks" from 19 expert computer science researchers.
3. The Big Discovery: "Knowing What You Want" is Harder Than "Doing It"
The researchers found two surprising things:
- The AI is actually pretty good at following orders. Once a user says, "Add a chart," the AI can usually draw a good chart.
- The AI is terrible at guessing what you want in the first place. The AI often suggests the wrong actions. It might suggest adding a chart when the user just wanted a simple summary.
The Analogy: Imagine a waiter who is amazing at cooking the steak exactly how you ordered it (medium-rare, with garlic butter). But the waiter is terrible at guessing that you actually wanted a salad instead. The problem isn't the cooking; it's the guessing.
4. The "Mind-Reading" Experiment
The team asked: Can we train a computer to guess what a human will pick?
They tried to teach AI models to predict a user's choices by looking at different clues:
- Clue A: What papers has the user read before? (Not very helpful).
- Clue B: What did the user say they like? (e.g., "I like short reports"). (Not very helpful).
- Clue C: What actions did the user actually pick in the past? (This was the winner).
The Lesson: People's actions speak louder than their words. If you tell a friend, "I love spicy food," but you always order mild soup, your friend should listen to what you order, not what you say. The AI learned that looking at a user's past "checkmarks" was the best way to guess what they would pick next.
5. The "Changing Mind" Problem
The researchers also found that human preferences are tricky. Sometimes, a user picks an action today, but if you ask them again in a few months, they might pick something different because their goals changed.
- Example: A user might want "detailed technical steps" today because they are building a project. Next month, they might want "high-level summaries" because they are just trying to understand the basics.
- The Takeaway: You can't perfectly predict what a human wants forever. Sometimes, you just have to ask them (like the "Recipe Card" menu) because their goals shift like the wind.
6. The Final Result: A Better "Recipe"
Because they learned that looking at past actions helps, they built a system that uses a user's history to suggest better "menu items" automatically.
- When a user with a history of picking "comparison tables" asks a new question, the system is more likely to suggest, "Should I add a comparison table?"
- This made users much happier with the suggestions, even if the system didn't know everything about them perfectly.
Summary
DRACULA is a dataset and a study showing that for AI research agents, the hardest part isn't writing the report; it's figuring out which steps to take to make the report useful. By letting users pick the steps (actions) before the work begins, and by studying those choices, we can build AI that understands us better.
The paper concludes that while AI is getting better at doing the work, the real challenge is deciding what work to do in the first place. The best way to solve this is to listen to what users actually do, not just what they say they want.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.