Technical Summary: Uncertainty-Aware Generation and Decision-Making Under Ambiguity

Problem Statement

Large Language Models (LLMs) are increasingly deployed in high-stakes, subjective domains such as automated tutoring and peer review. In these settings, tasks often involve multiple decision steps (e.g., identifying a student's error, selecting a tutoring strategy, and generating a response) and exhibit high degrees of ambiguity. Human experts frequently disagree on the "correct" strategy or score for a given input, leading to significant uncertainty.

Current decision-making algorithms for LLMs often fail to explicitly model this ambiguity or provide statistical guarantees. Standard approaches, such as simple sampling or Minimum Bayes Risk (MBR) decoding that compares output strings directly, do not account for uncertainty over intermediate states (e.g., the specific tutoring strategy or review score). Consequently, these methods may propagate errors, fail to narrow down plausible decision ranges, and lack the statistical reliability required for trustworthy deployment.

Methodology

The authors propose a framework that integrates Bayesian decision theory and conformal prediction to create uncertainty-aware generation and decision-making algorithms. The core decision problem involves selecting an output $y^*$ (e.g., a review or tutor response) from a decision space $V^*$, given a context $x$ and a discrete set of states $\Omega$ (e.g., tutoring strategies or review scores).

1. Bayesian Decision Rule

The primary approach maximizes the expected utility over the distribution of states $p(\omega | x)$. The optimal output $y^*$ is defined as:
$$y^* = \arg \max_{y \in V^*} \sum_{\omega \in \Omega} u(y, \omega) \cdot p(\omega | x)$$
where $u(y, \omega)$ is a utility function encoding quality and consistency criteria. Unlike standard MBR which compares output strings, this method marginalizes over the state space $\Omega$. To handle the infinite space of $V^*$, the authors approximate the maximization by sampling a set $H$ of candidate outputs and selecting the one that maximizes the expected utility over the state distribution.

2. Conformal Prediction for State Pruning

To address the computational cost of summing over all states and to provide statistical guarantees, the authors employ conformal prediction. This technique constructs a prediction set $C(x) \subseteq \Omega$ that covers the true state with probability at least $1-\alpha$.

• Ambiguity Handling: Recognizing that standard conformal prediction assumes a single ground truth (violated in ambiguous tasks), the authors utilize Monte-Carlo conformal prediction. This allows for multiple plausible labels (strategies or scores) to be considered, widening the theoretical coverage to $1-2\alpha$ but empirically often achieving tighter bounds.
• Pruning: The prediction set $C(x)$ is used to prune the state space in the Bayesian objective, reducing the sum to only plausible states:
  $$y^* = \arg \max_{y \in V^*} \sum_{\omega \in C(x)} u(y, \omega) \cdot p(\omega | x)$$

3. Risk-Averse Decision Making

The authors also evaluate a risk-averse strategy (minimax) that maximizes the utility of the worst-case state within the conformal set:
$$y^* = \arg \max_{y \in V^*} \min_{\omega \in C(x)} u(y, \omega)$$
This approach aims to prevent the generation of low-quality outputs by optimizing for the most adverse plausible scenario.

Experimental Setup

The methods were evaluated on two tasks:

• Peer Review: Generating reviews and scores for papers from the Review-5K and NLPEER datasets. Models included Gemma-3-27B, Qwen3-30B, and Mistral-3.2-24B. A fine-tuned Llama-3.2-3B was used to predict the score distribution $p(\omega | x)$.
• Tutoring: Generating tutor responses for the MathDial dataset using the TutorRL-7B model. Strategies (Focus, Probing, Telling, Generic) were predicted by a fine-tuned Llama-3.2-3B.

Utility was estimated using LLM-as-a-judge approaches (Qwen3-14B for reviews, a reward model for tutoring) with 5-point Likert scales.

Key Results

• Bayesian Improvements: The Bayesian decision rule consistently improved the utility of generated outputs compared to standard ancestral sampling across both domains. In peer review, it notably improved "actionability." In tutoring, it improved the scalar reward metric.
• Conformal Pruning: Using conformal prediction to prune the state space retained performance while significantly reducing the number of utility estimations required (up to ~50% compression in review tasks).
• Risk-Averse Limitations: The minmax (risk-averse) strategy showed mixed results. While it performed well in some review criteria, it often degraded performance in tutoring. The authors attribute this to the strategy optimizing for "generic" or "telling" strategies when these appeared frequently in the conformal prediction sets, effectively lowering the quality of the output to satisfy the worst-case constraint.
• Scaling Behavior: Increasing the sample size $|H|$ consistently improved the performance of the Bayesian rule. Conversely, the minmax rule's performance degraded as $|H|$ increased, likely due to interactions with wider prediction sets in ambiguous tasks.
• Model Sensitivity: Qwen3 showed different performance characteristics compared to Gemma and Mistral, suggesting that post-training objectives influence how well these models align with specific utility criteria.

Significance and Claims

The paper claims to outline a method for improving LLM-based decision-making in ambiguous, high-stakes domains by explicitly modeling uncertainty over intermediate states. The significance of the work lies in:

1. Bridging Theory and Practice: It demonstrates how combining Bayesian decision theory with conformal prediction can achieve real-world goals like statistical guarantees and high utility, even when the underlying state distribution is ambiguous.
2. Handling Ambiguity: It highlights that standard decision rules fail when human label variation is high and proposes specific techniques (Monte-Carlo conformal prediction) to handle multiple plausible ground truths.
3. Community Call to Action: The authors emphasize that current LLMs often struggle with categorical prediction tasks (like score prediction) and that existing datasets lack multiple annotations for the same input. They call for the community to focus on collecting multi-label data, developing better LLM judges, and creating models that reflect plausible human diversity in decision-making.

The authors maintain a modest stance, noting that their methods are designed to support human reviewers and teachers rather than automate the peer review process or replace human judgment. They acknowledge limitations regarding data leakage (models may have seen training data) and the lack of real-world field studies to measure long-term impact.