$\tau$-Rec-Books: A Disclosure-Controlled Benchmark for Requirement Elicitation in Agentic Recommender Systems

Imagine walking into a library where the librarian knows every book in the building, but you cannot simply hand them a list of what you want. Instead, you must have a conversation. You might start by saying you want a science fiction novel, but as the librarian suggests titles, you realize you also need a story set before the year 2000, or perhaps one written in a specific language. The challenge for an artificial intelligence acting as this librarian is not just finding books that match a list of facts, but figuring out what those facts are in the first place. Real people rarely state every single requirement at the very beginning of a conversation. They often reveal their true needs only after seeing a suggestion, or they might hesitate to say exactly what they are looking for until the right moment. If the AI guesses too early, it might recommend a book that misses a hidden requirement, frustrating the user. If it waits too long or fails to ask the right questions, it wastes time. The core question researchers are trying to answer is how well these digital assistants can navigate this delicate process of discovery, especially when the user holds back information.

To investigate this, a team of researchers created a controlled experiment called a benchmark, which is essentially a standardized test designed to measure performance under specific conditions. They built a digital catalog containing 20,000 real books from a public database, complete with details like genre, language, country of origin, and publication year. They then designed 450 specific tasks, each representing a unique request for a book. The clever part of their design was that they kept the actual request for every task exactly the same, but they changed the rules for when the user was allowed to reveal the details. In one scenario, called "volunteer," the user told the AI everything they wanted right at the start. In a second scenario, called "mixed," the user stated one detail but kept others hidden, revealing them only if the AI asked the right question. In the third and most difficult scenario, called "hidden," the user refused to state certain requirements entirely, forcing the AI to guess by proposing books and watching whether the user accepted or rejected them. The researchers used a simulated user, a computer program following strict rules, to interact with four different AI agents to see how they performed.

The results revealed a stark reality about how these systems handle information. When the user volunteered all the details immediately, the AI agents were remarkably successful, finding the right book in nearly every single attempt. However, as soon as the researchers introduced the need to ask questions, the success rate dropped significantly. When the user revealed some details only upon being asked, the agents succeeded about three-quarters of the time. The situation became much more dire when the user hid some requirements completely. In these cases, the AI agents succeeded less than one-third of the time. The study showed that the difficulty was not in the size of the book collection or the specific tools the AI used to search, but in the timing of the information. The agents struggled most when they failed to ask questions before making a recommendation. When the researchers explicitly instructed the agents to ask questions, their performance on the "mixed" tasks improved dramatically, reaching the same high levels as when all information was given upfront.

The researchers also discovered that the biggest failures in the "hidden" scenarios came from a specific mistake: the agents would recommend a book they had never actually proposed to the user for a verdict first. Instead of testing their ideas, they would commit to a final answer too quickly. When the agents were forced to propose a book and wait for a yes or no before making a final recommendation, they performed better, though they still fell short of a perfect score. The study also compared two ways the AI could search the catalog. One method was like typing keywords into a search bar, while the other was a structured filter that allowed the AI to narrow down results by specific categories like year or country. The structured search method helped the agents find the right book more often, but it could not overcome the fundamental problem of not knowing what the user wanted.

Ultimately, this work demonstrates that the ability of an AI to have a successful conversation is not just about having access to a large database of information. It is about the strategy of discovery. The study proves that when requirements are hidden, even advanced AI agents struggle to find the right answer without a clear process for asking and verifying. The researchers found that if an agent is told to ask questions, it can solve almost all the problems where the user is willing to answer. However, when the user refuses to state a requirement and only judges the suggestions, the agents still make significant errors, often recommending books that fail the user's unspoken rules. The team released all their data, the book catalog, and the code they used, allowing other scientists to repeat the experiment and build upon these findings. Their work suggests that for AI to become a truly effective conversational partner in recommendation, it must learn to treat a user's silence or refusal to answer not as a dead end, but as a clue that a hidden requirement exists, requiring a more careful and systematic approach to elimination and discovery.