🤖 machine learning

Ask to Be Sure: Informative Interactions for Confident Multi-Turn LLM Recommendation

This paper proposes a novel method for training conversational recommender systems that fine-tunes large language models to strategically ask questions by using entropy reduction as a reward signal, thereby improving recommendation accuracy and interaction efficiency without relying on unavailable ground-truth data.

Original authors: Cedar Site Bai, Duanshun Li, Zhenyu Liao, Sheikh Sarwar, Huiyuan Chen, Yuan Chen, Changhe Yuan, Haiyang Zhang, Qilin Qi

Published 2026-08-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Cedar Site Bai, Duanshun Li, Zhenyu Liao, Sheikh Sarwar, Huiyuan Chen, Yuan Chen, Changhe Yuan, Haiyang Zhang, Qilin Qi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine sitting down with a friend who knows everything about movies but has never met you. They might start by listing their own favorites, or perhaps they guess what you like based on a single word you say. If they get it wrong, the conversation stalls. This is the core challenge facing modern artificial intelligence when it tries to act as a personal guide. In the field of conversational recommendation, the goal is to build systems that do not just wait for a user to say exactly what they want, but instead engage in a back-and-forth dialogue to uncover hidden preferences. The system must ask the right questions at the right time to narrow down millions of possibilities into a few perfect choices. The difficulty lies in knowing which question will actually teach the computer something new, rather than just filling up the conversation with polite but useless chatter.

A team of researchers at Amazon set out to solve this problem by teaching artificial intelligence to measure its own confusion. In their work, they observed that when a computer is unsure about what to recommend, its suggestions are scattered and inconsistent. If you ask the same computer to list the best movies for a user at the start of a conversation, it might guess wildly. But as the user shares more details—mentioning they love sci-fi but hate horror, or that they enjoyed a specific actor's performance—the computer's guesses become more focused and consistent. The researchers realized that this shift from scattered guesses to focused ones is a measurable signal of learning. They decided to treat this reduction in confusion as a reward, a way to tell the artificial intelligence that it had successfully gathered useful information.

To test this idea, the researchers created a system where the artificial intelligence plays a game of discovery. They started with a conversation where the user's preferences were unknown. The computer would generate a list of movie recommendations based on what it knew at that moment. Then, they asked the computer to generate that same list many times to see how much the suggestions varied. If the suggestions were all over the place, the computer was highly uncertain. Next, they simulated a user responding to the computer's questions, revealing a preference. The computer then generated a new list of recommendations based on this new information. The researchers measured how much the variety in the suggestions had decreased. A large drop in variety meant the computer had learned something valuable; a small drop meant the question had been unhelpful. They used this measurement of reduced confusion as a score to train the artificial intelligence, rewarding it for asking questions that led to clearer, more confident recommendations.

The researchers tested this approach on two large collections of human-written conversations about movies. They compared their new method against other ways of training artificial intelligence, including systems that relied on human judges to decide if a conversation was "interactive" or systems that tried to guess the correct answer directly. In their simulations, the artificial intelligence trained with the confusion-reduction score learned to be much more efficient. It required fewer turns of conversation to reach a recommendation that the simulated user would accept. For example, in one set of tests, the system using this new method reached a successful recommendation in about three turns, whereas other methods often took longer or failed to find a good match as often. The system did not just ask more questions; it asked better questions that moved the conversation toward a solution.

One of the most significant aspects of this work is that it does not require the researchers to know the "correct" answer in advance. In many real-world situations, there is no perfect movie that a user is secretly waiting for; preferences are fluid and subjective. Traditional methods often struggle here because they rely on comparing the computer's guess to a known ground truth. This new approach, however, works by measuring the internal state of the computer itself. If the computer becomes more certain after a user's response, it knows it has made progress, regardless of whether the final recommendation is perfect. This makes the method practical for real-world applications where the ideal answer is often unknown.

The study also highlights a limitation in how we currently evaluate these systems. Many existing methods judge a conversation by how "engaging" it sounds or by whether it follows a rigid format, such as asking yes-or-no questions. The researchers found that a conversation can sound very interactive and friendly while still revealing very little about what the user actually wants. By focusing on the reduction of uncertainty, their method bypasses the need for a human or another computer to judge the quality of the chat. Instead, the system learns to value the information gain directly. The results showed that when the artificial intelligence was trained to prioritize this information gain, it became a more strategic partner, guiding the conversation toward a useful conclusion with fewer wasted words.

In the end, this research offers a new way to think about how machines learn to talk to us. It suggests that the best conversations are not necessarily the longest or the most entertaining, but the ones that most effectively reduce the gap between what the machine knows and what the user wants. By teaching artificial intelligence to recognize when it has learned something new, the researchers have created a path toward systems that are not just responsive, but genuinely insightful. The work demonstrates that with the right feedback loop, machines can learn to ask the questions that matter, turning a simple exchange of words into a powerful tool for discovery.

Technical Summary: Ask to Be Sure

Problem Statement

Conversational Recommender Systems (CRS) aim to transform recommendation from a one-shot prediction task into an interactive preference discovery process. While Large Language Models (LLMs) have demonstrated strong capabilities in natural dialogue and recommendation accuracy, a critical challenge remains: how to guide multi-turn interactions to effectively elicit user preferences.

Existing approaches face specific limitations:

Separate RL Agents: Many methods train a separate reinforcement learning (RL) agent to decide when to ask or recommend, restricting interactions to rigid formats (e.g., multiple-choice, Yes/No) and decoupling the dialogue generation from the recommendation logic.
LLM-Judge Rewards: Other approaches, such as CollabLLM, optimize for interactivity using an LLM-judge. However, these methods do not measure the actual useful information gained regarding user preferences. An interaction may appear proactive yet reveal little about what the user actually wants.
Ground-Truth Dependency: Many optimization strategies rely on ground-truth recommendations (e.g., whether a specific movie was clicked), which are often unavailable in real-world deployment scenarios.

The paper focuses on designing interactions that strategically solicit user preferences to reduce the assistant's uncertainty about what to recommend, without relying on ground-truth labels during the training phase.

Methodology

The authors propose a framework that quantifies the effectiveness of an assistant's interaction based on the reduction in recommendation uncertainty. This uncertainty is measured via entropy over sampled recommendation lists.

1. Uncertainty Measurement

The system defines uncertainty as the Shannon entropy over the distribution of recommendations induced by repeated sampling of a fixed prompt given a conversation history $C$ .

Sampling: For a given conversation state, the assistant generates $n$ lists of top- $m$ recommendations.
Distribution Estimation: The empirical frequency of each item across these lists is calculated to form a recommendation distribution $p(r)$ .
Entropy Calculation: The entropy $H(C)$ is computed. To account for ranking position, a weighted entropy $H_w(C)$ is used, applying logarithmic decay weights ( $w_j = 1/\log_2(j+1)$ ) to items based on their rank in the list.

2. Information Gain as Reward

The core innovation is using the reduction in entropy as a reward signal.

Turn-Level Reduction ( $I_T$ ): Measures the entropy decrease after a single assistant turn ( $A_1$ ) and the subsequent user response ( $U_2$ ).
$I_T(A_1) = H_w(C_1) - H_w(C_2)$
Conversation-Level Reduction ( $I_C$ ): Measures the total entropy reduction from the start of the conversation to the terminal state, attributed to an initial turn.
$I_C(A_1) = H_w(C_1) - H_w(C)$

This entropy reduction serves as a reward for fine-tuning the LLM. Crucially, this metric does not require ground-truth recommendations, making it applicable to real-world scenarios where such labels are missing.

3. Fine-Tuning Strategies

The proposed entropy-reduction score is integrated into two standard optimization paradigms:

Supervised Fine-Tuning (SFT): Training examples are selected from generated data based on high entropy-reduction scores.
Direct Preference Optimization (DPO): Preference pairs are constructed where the assistant response leading to higher entropy reduction (and thus greater information gain) is preferred over the alternative.

Key Contributions

Uncertainty-Reduction Reward: The introduction of a reward mechanism that improves conversation strategy and recommendation accuracy by explicitly optimizing for information gain (uncertainty reduction) rather than generic interactivity.
Ground-Truth Independence: A practical solution for training conversational recommenders in scenarios where ground-truth recommendations (e.g., click-through data) are unavailable.
Datasets: The contribution of two turn-level completion-pair datasets derived from INSPIRED and ReDial, specifically labeled with entropy-reduction rewards for DPO fine-tuning.

Experimental Results

The method was evaluated on the INSPIRED and ReDial datasets using Llama-3.2-1B-Instruct. The models were compared against:

Vanilla: Zero-shot pre-trained LLM.
SFT (Raw): Fine-tuned on raw dataset conversations.
CollabLLM: A baseline using a reward combining ground-truth hits, LLM-judge interactivity, and token efficiency.

Key Findings:

Recommendation Quality: Models fine-tuned with entropy-reduction rewards (specifically DPO (Turn Entropy) and SFT (Conv Entropy)) achieved the highest Hit@1 and Hit@5 scores on both datasets. For instance, on INSPIRED, DPO (Turn Entropy) achieved a Hit@1 of 3.32%, outperforming the Vanilla baseline (1.60%) and the CollabLLM baseline (3.15%).
Conversational Efficiency: The proposed methods significantly reduced the number of turns required to reach the ground-truth recommendation. On the ReDial dataset, the DPO (Conv Entropy) model required only 2.74 turns on average to reach the ground truth, compared to 2.94 for the Vanilla baseline and 2.85 for SFT (Raw).
Simulated Conversation Hit Rate: The entropy-reduction variants consistently outperformed the CollabLLM baseline in simulated conversation hit rates, even without access to ground-truth labels during the reward calculation phase.

Significance and Claims

The paper claims that by aligning interaction design with information gain, the proposed method enables LLMs to engage in more strategic, proactive, and efficient conversations.

Strategic Elicitation: Unlike methods that rely on subjective LLM-judges or rigid templates, the entropy-reduction reward allows the model to learn which types of questions (whether open-ended or specific) effectively reduce uncertainty in a data-driven manner.
Practicality: The ability to fine-tune CRS without ground-truth recommendations addresses a major bottleneck in deploying conversational agents in real-world environments.
Efficiency: The results demonstrate that reducing uncertainty leads to fewer turns to reach a satisfactory recommendation, validating the hypothesis that "asking to be sure" is a viable strategy for confident multi-turn recommendation.

The authors note limitations, including the reliance on automatic metrics and LLM-based user simulators rather than human evaluation, and the assumption that the base LLM can name candidate items without external retrieval (though they suggest this can be extended to retrieval-based settings). Future work includes scaling to larger models and integrating the reward into on-policy RLHF methods like PPO.